Think-Aloud Protocol II

Class Objectives
By the end of this class you should be able to:
•Articulate the difference between think-aloud and self-explanation protocols
•Describe basic categories of analysis for educational technology protocols
•Classify participant utterances into prescribed categories

Think-Aloud Protocol vs. Self Explanation
•Think Aloud
–Purported not to change user’s cognition
–Description (not explanation) is the target
–User’s process is the target
•Self Explanation
–Improves learning and understanding
–Explanation and reflection is the target
–User’s understanding is the target

Think Aloud vs. Self Explanation
•Purely HCI issues? Think-aloud
•Educational effectiveness? Self-Explanation
•Use a combination to evaluate your system’s educational content as well as the design
–Why students learn differently
–How students learn with your system (redesign)

Protocol Analysis for Experimental Studies
•Theory guides initial ideas for categories
•Comparison will be across groups
–What utterances will show learning differences?
•Errors
•Paraphrases vs. inferences
–What utterances will show processing differences?
•Shallow approaches (what kinds of statements?)
•Deep approaches (what kinds of statements?)

HCI Studies: Ed Tech
Why do we need to go beyond the think-aloud to self-explanations?
•Where do learners go wrong and why?
•Do learners understand the content?
•How do they work with the system and the content?
•Goals:
–Improve depth of processing
–Support metacognition
–Minimize errors

Useful Basic Categories
•Metacognitive Monitoring (thinking about your own thinking)
–Occurs when the learner/user is monitoring the accuracy of their thinking
–Positive: Identifies thinking as correct
•“Yeah, that’s what I thought.”
–Negative: Identified thinking as incorrect or confused
•“Um, I totally don’t get this.”
•“I thought I should click on Assignments, but looks like not.”
•Paraphrasing
–Occurs when learner reads, paraphrases, or summarizes system text
–Shows engagement, but not deep processing
–Repeated rereading or paraphrasing may indicate confusing instructions/text, especially if intermixed with errors
•Goal-Directed Inferences
–Content: Learner uses goal to make an inference about the educational content.
•“I’m trying to find information on reading strategies, so I’ll watch this video to see what it tells me.”
•“So … if matter is neither lost nor gained … it must mean that the water just changed states here.”
–System Behaviors: Learner uses goal to make an inference about what the system will do.
•“I think this link will open the video for me.”
–Navigation: Learner uses goal to make an inference about where to go.
•“I’m looking for homework, so I think I need the calendar.”
•Errors
–Inferences can be correct or incorrect
–Often occur in context
•Not an error in system design (e.g., a broken or misdirected link)
•Error or problem is tied to learner’s intended goal, or their interpretation of system behavior or educational content.
–“I can’t find the homework link.”
–“The video didn’t start. It must be broken.”
–“So, matter usuallyis conserved but sometimes it can be lost.”

Other Useful Categories
•Negative vs. Positive Affect
“Ugh. I hate that picture!”
“Wow. This is such a cool activity”
•Help use
–Do users seek help? Is it available when they do?
•Strategies for use
–What will they try to do with the system?
–Is it what you intended?

Segmenting
•Look for fairly broad statements
–Idea units. An utterance that expresses one thought, idea, or confusion.
•May consist of a partial sentence
•May consist of several sentences
•Fits into a single category
–Look for pauses–often indicate shift
–Turn-taking often results in new idea units

Example Transcript
Participant:
“When assembling the door, Nichols accidentally nails the transversal plank at a 36º angle. The measure of angle ARNequals 36º. What is the measure of angle BAR?”
Reading

ARNis 36º because that’s given. And BARis going to be 36º also, because of alternate interior angles. That’s because of measure angle ARN.
Content Inferences

“In the diagram below line ABis parallel to DC. If the measure of angle ACDis equal to 66.1º and the measure of angle ACBequals 44º, find the measure of angle ABC.”
Reading

Experimenter:
You seem a little unsure. What were you thinking?
Prompt

Participant:
I couldn’t find the question mark for where ACD was.
Error?

Tips for Coding
•Video/Audio
–Listen at LEAST 2 times before coding.
•Text transcripts
–Listen to audio while reading the transcript - this will allow you to hear the inflection of phrases that could mean a variety of things
•Context and intonation is important
–Then read at LEAST 2 times before coding

Videos to Critique
Usability Test: Would benefit from more verbalization
http://www.youtube.com/watch?v=SFwU_rvMBaE
Usability Test: Watch out for creating a conversation!
http://www.youtube.com/watch?v=Y_rKE0O7tek
Good information (but don’t get user to design!!)
http://www.youtube.com/watch?v=Jpt3qz1gtXI