2.3 Verbal Protocol Analysis

Presentation on theme: "2.3 Verbal Protocol Analysis"— Presentation transcript:

1 2.3 Verbal Protocol Analysis
What is wanted Analysis Design Evaluation

2 Mental Model What is it? Does it matter? Not limited to STM Dynamic

3 Protocol Analysis Assumption on working memory Verbal protocol
Think aloud Assumption on working memory

4 Types of Verbal Protocol Generation
Concurrent protocol analysis Observation-based Generated during task a.k.a. think aloud Retrospective protocol analysis Generated after task; replay (-) Forget (-) Post-task explanation (intervene) (-) Post-task explanation (self-presentation) (+) Some cannot talk aloud (+) More time for preparing interview (+) Minimize interruptions (benchmark)

5 Title Source The validity of verbal protocols
Memory & Cognition, 1989, 17 (6),

6 Research Questions Research questions Two independent variables
Verbalization  primary tasks (processes)? Concurrent vs. retrospective? Two independent variables X1: protocol types X2: task types One dependent variable (Y): accuracy There’s still another Y: response time

7 Task Types Addition Anagram

8 Task Types Evaluate profits of 2 plans Plan Cost: $6.9 P(Win): 32%
Gambling Raven’s Matrice Evaluate profits of 2 plans Plan Cost: $6.9 P(Win): 32% Gain: $11.5

9 Design Subjects (N=24) 所有受試者經歷所有任務型態 每種任務型態搭配一種原案產生方式
Five 2-h sessions 一個練習+四個任務 Paid $9.00 plus $0.05 for every problem solved correctly For each first hint (on an anagram or a Raven's problem) $0.02 was subtracted, with $0.01 subtracted for every subsequent hint 所有受試者經歷所有任務型態 每種任務型態搭配一種原案產生方式

10 Protocol Types (Retrospective)
Prompting was rarely needed; it was used roughly once every 50 trials Stimulus-cued Response-cued prompted Concurrent Solve problem silently Please tell me what you are thinking Original problems offered Solutions offered Problems + eye-tracking offered Immediately afterwards “tell what you were thinking as you solved the problem” "explain why you looked where you looked and what you were thinking when you looked there."

11 Accuracy (24 participants)
Concurrent vs. Control Four Experimental Conditions

12 Compare Statement Types in Verbal Protocols
Other types Ignored here

13 Discussion 不應將放聲思考視為絕對有效 依據任務型態預測干擾，存在相當的困難
干擾：語言過程干擾解題過程 Gamble’s task: 不應干擾(但實際上受干擾) Raven’s task: 應干擾(但實際上未干擾) 回溯思考組可能發生偽造(fabrication)或遺忘的 現象

14 課堂練習：Think Aloud Method
步驟 說明 時間長度 課堂分組 每組人數(N=5~6)；角色確立 5 min 確認測試目的 挑選一個軟體或網站(手機App亦可) 10 min 挑選幾個有意義(如困難)的任務(N  3) *任務的意義取決於某種「反差」的呈現 招募受試者 挑選(生手)使用者(N  2) 原案資料蒐集 主測者(N=1)：任務說明與實驗主持 記錄員(N=3)：觀察並記錄原案資料 20 min 原案資料分析 了解受試者對於任務的理解與執行 15 min 分享與討論 呈報資料分析的發現

15 Stages in Verbal Protocol Analysis
Determine research objectives Collect data Process data

16 Task-Subject Combination NOT for Think-Aloud
Someone cannot talk well Too Quick to Be Told Automated response

17 Data Collection Informed consent Multiple recorders
Videotaping and recording Access privilege Privacy protection Multiple recorders From different angles At a closer distance # of participants: 1, [5, 10] for novices Orientation Practice think-aloud with no guidance 如何教導 受試者think aloud

18 Data Collection Remind participants while they stop
Please keep talking What are you thinking of? Types of difficulties Type 1: no talk until reminded Type 2: too much explanation Possible remedies No language restriction; abandon Relax & remind Which is better to ask 施測者：我注意到你比一般更容易被電擊，你覺得這是因為什麼？ 受試者： Gee, I don't really know… Well, I used to build radios and stuff when I was 13 or 14, and maybe I got used to electric shock.

19 Data Collection 問題討論 哪個位置最適合作為Think-aloud法的實驗場所？ 關於探索受試者行為，下列描述是否正確？
(A) 實驗室；(B) 教室；(C) 走廊；(D) 系辦 關於探索受試者行為，下列描述是否正確？ 應定期提醒受試者專注於實驗任務 當受試者面露疑惑時，應盡速詢問其疑惑處 應盡可能維持受試者的專注狀態

20 Processing Data Transcribing verbal protocols Segmentation
Two initial steps 問題討論 Transcribing verbal protocols By assistants aware of terminology Segmentation Complete thought S + V Research goal Google map 研究 受試者 要用一下路徑規劃 我想…ㄟ…喔…對了…在 這邊 東、右邊、差一點，咦… Amanda往東經過忠孝橋， 再轉市民大道。最後， Amanda在京華城下橋 你如何斷句?

21 問題討論：文稿中是否存在不確定感? 改編自(Trickett & Trafton, 2009)
氣象分析員應用兩個預測模型預測降雨 改編自(Trickett & Trafton, 2009) OK, 讓我們跟ETA模型比比較看…恩…他們不一樣，這有意思 恩，通常不會差太多 一般是降雨時才會不同 讓我們先看看36小時後 OK…不然再看一下42, 48, 跟54 恩…OK…48小時後會有降雨 我想回到GFS模型，看看48小時 看起來他們並不一樣 恩，我想沒錯，這絕對需要更多資訊，預測的困難度肯定不低

22 Processing Data: Coding
Linguistic level 語助詞 Um(恩), er(ㄟ) 規避語 有點(sort of)、可能(maybe)… 明確表達不確定感 我不清楚(I have no idea)… Contextual clues Model disagreement; dithering; frustration

23 Processing Data: Coding

24 編碼練習 研究目的：瞭解新教育措施被採行 「我問這位老師為什麼會採行新方案，她說因為 九年級學生只有七年級的程度，舊課程沒有效果。 Nelson閱讀測驗成績顯示：學生經十個月的學習， 在課業上只有五到六個月的進步」 請為上述文字進行編碼

25 編碼練習 研究目的：瞭解新教育措施被採行 Mr. Walt說：「我不是很清楚SCORE-ON計畫中包括些什麼。 不過，我知道裡面結合了很多東西。」 訪員札記：那天早上資源中心主要是給FACILE專案使用， 這是Mr. Walt比較瞭解的案子 當天下午，Mrs. Hampshire將資源中心調配給SCORE_ON專 案。對此，Mr. Walt說：「這是不同的案子，因此會有不 同的用法。」 請為上述文字進行編碼

26 Inter-rater Reliability
≥ 2 raters Rater training Understand code meaning Partial data or different data set K coefficient of agreement (Cohen’s Kappa) 選擇 類別數目 輸入 兩評分員 歸類結果 計算

27 Analysis Result by Cohen’s Kappa

28 Summarizing Verbal Protocol Analysis
Advantages Simple to do (for some) Useful insights Used in design and in evaluation Disadvantages May alter how users act; 霍桑效應 Unnatural response changes how users act Negatively affect quantitative measurement (e.g., time) Inevitable subjectivity Resources burden 資安行為觀察 Only for Think Aloud