A summary of Protocol Analysis: Verbal Reports as Data

A while ago I read Protocol Analysis which is a summary of the research about using verbal reports as data.  This should be relevant for running user tests and in general understanding what happens when people try to tell you what they're thinking.

Giving verbal reports affects the subject's cognitive processes, that is, talking about something can change how we think about it.

In order to give a verbal report, the subject must:

1. Understand what you are asking
2. Turn this understanding into retrieval cues
3. Use those retrieval cues to select the relevant information from long-term memory
4. Put the retrieved information into a sequential form, so that s/he can...
5. Generate a coherent series of verbalisations

When thinking aloud, ask people to only verbalise the information they attend to, not describe or explain what they are doing.  For example, "4 times 6", "24", etc. not "I am trying to multiply 24 by 36".

If you also ask people to explain what they are doing, you will change the sequence of their thoughts.

There are several design tactics to improve the quality of verbal reports:

• Make it clear that social interaction is not intended by sitting behind the subject or otherwise being invisible
• Explicitly warn the subject against explanation and description
• Provide practise problems to practise concurrent verbalisation and just verbalising without explaining or describing
• Minimise social interaction.  Say "Keep talking", not "Tell me what you are thinking".
• Always tell the subject to focus on completing the task with thinking aloud as a secondary instruction.  Only if the subject is completely focused on the task can we expect the verbalisation to reflect the same thought process as the silent condition.

Concurrent verbalisation, unlike normal social interaction, often lacks coherence and is disjointed without explicit relationships.  You have to resist the urge to therefore ask for explanation and description.  Instead, we need to work out how to infer underlying cognitive processes by encoding and analysing the verbal sequences.

For tasks of very short duration (that is, less than a few seconds) or with severe real-time constraints (e.g., juggling) retrospective verbal reports may be preferred.  Whenever appropriate, collect both concurrent and retrospective reports.

There are several design tactics to improve the quality of retrospective verbal reports:

• Start with "I first thought of..."  This helps subjects focus on the task of recalling distinct thought episodes rather than elaborating, rationalising, and justifying.
• Do warm-up tasks by retrospectively reporting on relatively fast processes (less than 2 seconds) (e.g., single and two digit addition, judge whether two letters are in alphabetical order)
• Instruct and remind subjects to remain focused on the task and retrieve info for the retrospective report only after completion.

Teaching aloud, for example, being asked to instruct an unseen partner, or being asked to explain reasoning, tends to improve performance.  So even if teaching aloud and describing reasoning is not as useful for user testing, it's actually quite useful for instruction and improving performance.  See Think Like an Expert.