Think-Aloud Protocol

Think Aloud Protocols (Methods)

What is a Think-Aloud Protocol?
•Participants, report (verbally) all task-relevant thoughts as they complete the task
–I prefer concurrent report
–Interviews: long-term version of retrospective protocols
–If you must use retrospective, use small time intervals! - You don't want them to have to remember what they thought about things from the very beginning of the task.
•Experimenters prompt users to produce (continuous) verbalizations, in a neutral manner
–“Don’t forget to keep talking”
–“What are you thinking?”
- But don't try to get them to say what you want them to say. Be neutral.

Getting Users Talking
•Tell them what they are going to be asked to do, and why
–We’re going to be asking you to “think-aloud,” which just means that we want you to say anything that’s in your head as you’re doing the task. This includes ideas, questions, frustrations, confusions, or comments as you work. Basically, you’ll be giving us your “stream of consciousness” thoughts as you work on each task.
Some people find this easy, but a lot of people find it weird –especially at first. But it’s the best way for us to get really good data on what is helping you and what is causing you problems as you work with the system. We’ll practice a little to help you get started.
•Always model the behavior FIRST.
–Users feel ridiculous
–Users need to hear what good thinking-aloud sounds like
–Need to model the level of detail required
•Practice! Need 2 practices (MINIMUM).
–Can use 1 practice, and provide feedback in the first few “throw-away” screens if you have them. Do not model or practice on your actual task. Use something else.
Model at the most detailed level possible. People will gravitate to less detail on their own, so model with a lot of detail. Give them feedback when they practice encouraging them to give you plenty of detail.

Practice Tasks: Option 1
•Mental Arithmetic
•Describe your thinking as you mentally solve an addition (or subtraction*) problem.
*Some users will find you unusually cruel and heartless, especially if the problems are too difficult.
49+56=

Practice Tasks: Option 2
•The Windows Walk
•Prompt: Imagine walking through your house or apartment. Go through each room, describing and counting the number of windows that you find.

Setting Up for Think-Aloud
•Position yourself behind and to the side of the user
–Peripherally visible, but not “in the action”
–Users are supposed to generate a verbal stream of data, not communicate with you
–If users ask you questions, praise them and encourage them to keep voicing those questions (even though you can’t answer them)
•Have them read their task scenario first
–E.g., “You are a 4thgrade teacher who is trying to set up the initial gradebook for your class. You want to …”
–Make sure the task is available on paper, for constant reference
•Remind the user of the instructions:
–Now we’ll start working with the system. Remember, just say whatever comes into your head, no matter how silly it seems to you. All that data is really useful in helping to improve the system.

Prompting the User
•Try to stay neutral
–Don’t ask ‘why’
–Don’t react to errors or successes
–Try to get them to forget about you!
•Prompt as needed (but keep it easy and breezy)
–“Don’t forget to keep talking”
–“What are you thinking?”
–“Can you say more about that?”
•Especially when they start to say something interesting but stop! This happens a lot…
•You can also just repeat the last bit of what they said as a question. Participant “So…. [trails off]” Experimenter: “So?”

Kids vs. Adults
•Both
–Concerned about looking “stupid”
–Vary widely in how naturally they keep talking
•Kids
–Generally more reluctant to talk aloud (prompt more)
–Mumble
–May need more reassurance

*If a user really can't complete the task, set a time limit, then just have them stop.
You can stop the entire thing and be done, but if the parts of your system are separate enough, get them to the next task and start there.

Praise, Praise, Praise
•Uncomfortable users = QUIET users
•Use praise for process liberally during practice and early in the task/study
•Sneak it in while system loads, or there are natural pauses
–“You’re doing a great job of thinking-aloud! Keep up the good work!”
–“You’re a natural at this” or “That’s great! Just what we need. I’ll keep prompting you to help.”

Example Task Scenario
Jack is a 10th grade science teacher who has volunteered to fill in for Jan, a 7th grade science teacher, while she’s out sick. Jan was supposed to teach her class about changes in the Earth’s surface this week. She suggests Jack come up with a classroom activity based on changes in the Earth’s surface.
One of the topics that Jack teaches in his 10th grade class relates to earthquakes. He wants to teach the 7th graders something related to this topic. Jack often uses DLESE in order to find activities and detailed text on material he teaches in his 10th grade class. He decides to check out what DLESE has to offer. He wants to find out which concepts he needs to teach the 7th graders, in addition to a classroom activity that support these concepts.
(Have www.dlese.orgopen to begin the task).

*If there is a site where there is going to be a lot of text, prompt them to read aloud so you can follow where they are on the site.

Other Tips
•Have a bottle of water for your participant
•If you have a long session planned, give them breaks
•Keep things upbeat and friendly
–If the user gets down, tell them “We are learning so much from your data! You are doing a great job for us in this study!”

Record the Session
•I like CamtasiaStudio
–Screen-capture + Voice recording (synched)
–Free trial for 30 days at:
http://www.techsmith.com/camtasia.asp
–1 year license through U of U Software Licensing
•$25 download, $30 for CD
https://software.utah.edu/osl/detail.shop?productId=1308

Create Videos
•Anonymity largely protected (voice is only identifying info)
•Powerful in highlighting problems, processes