ACT-PRO Action Protocol Tracer A Tool for Analyzing Simple, Rule- based Tasks Wai-Tat Fu & Wayne D....

ACT-PROAction Protocol Tracer

A Tool for Analyzing Simple, Rule-based Tasks

Wai-Tat Fu & Wayne D. Gray

ARCH Lab

George Mason UniversityARCH LABORATORY

Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks 2

Overview

• Problem

• What is ACT-PRO?

• Utility of ACT-PRO

• How it works


*where correct = behavior that the model knows about

Problem• Model tracing correct* behavior

– Successfully developed and used to shadow students as they perform a task– Usually combined with tutors that keep students from straying off the correct

path

Can model tracing techniques be applied after-the-fact to large quantities of action protocol data?

• Model tracing errors, error detection, and error correction– Usually done “by hand” or not at all

Can the tracing of errors, error detection, and error correction be handled as routinely as the tracing of correct behavior is handled by existing tutoring systems?


ACT-PRO• Developed to handle a class of interactive tasks

– Programming VCR, flight computer, ATM, etc

• Requires task analysis of interactive behavior– Format of task analysis is irrelevant; e.g., ACT-R,

GOMS, Soar, or Hierarchical Task Analysis

• Currently implemented in visual basic for applications


The Task -- Overview

• ACT-PRO has been applied to action protocols collected from Ss programming a simulated VCR

• The VCR is representative of a large class of human-computer interaction tasks (e.g., ATMs, flight computers, and so on)

• The ACT-PRO approach should work with this class of interactive tasks


Level of Effort

• How does ACT-PRO compare to alternative methods of classifying action protocols?

Study 1 Study 2

Done By Hand ACT-PRO

Keypresses 1,946 9,545

Goal pushes & pops 2,118 6,750

Time spent ≈ 12 months ≈ 12.5 days

≈ 12 days to program

≈ 12 min to setup and run

≈ 12 hr to verify results


Overview of Rest of Talk

• The task in more detail

• ACT-PRO -- how it works


The Task• A simple rule-based task - programming a simulated

commercial VCR


Device-Task Rule-Hierarchy

SET-

startMode

SET-

DOW

SET-

channel

START SAT 11 END CS ON

7 0 0 8 0 0

SET-

endMode

SET-

csMode

SET-

progRec

SET-start-

hour

SET-start-

10min

SET-start-

min

SET-end-

hour

SET-end-

10min

SET-end-

min

VIDEOTAPE

StarTrek

PROGRAM -StarTrek RECORD

DO-startMode DO-endMode

DO-

startTime

DO-

endTime

Example: Start Time: 7:00; End Time: 8:00; Dow: Sat Channel: 11


Focus of the Study• Focus of the study is on the explanation of human

errors in terms of the model’s control structure

• By comparing the human data with the model’s prediction, we can characterize human behavior as

matching or violating the model’s behavior

• ACT-PRO is needed to correctly categorize the correct behavior and to classify errors, error detection, and error correction


ACT-PRO

ACT-PROAction ProtocolsControl StructureCorrect BehaviorErrors,Error Detection,Error Correction

MatchingViolating


Matching Protocols to Model

HOUR

HOUR

10MIN

10MIN

10MIN

MIN

Push-goal-SET-StartHour

Click On HOUR

Click On HOUR

Pop-goal-SET-StartHour

Push-goal-SET-Start10Min

Click On 10MIN

Click On 10MIN

Click On 10MIN

Pop-goal-SET-Start10Min

Push-goal-SET-StartMin

Click On MIN

Pop-goal-SET-StartMin

Button Clicksfrom Ss

Model


ACT-PRO

• Two levels:– Keypress level

• Classify each button click as matching or violating the model

– Goal level• Parse button clicks into goal pushes and pops

• Classify each goal push and pop as matching or violating the model


Keypress Level

• Button clicks are classified as a type of hit or error*• A transition matrix is used to parse the button clicks

• Within context defined by prior button clicks, clicks are unique to a certain goal

Prior/Current btn 1 btn 2 btn 3 …

btn 1 hit error errorbtn 2 error hit hitbtn 3 error error hit

…

* see Gray, W. D. (in press). The nature and processing of errors in interactive behavior. Cognitive Science.

..Btn 2 Btn 3 - hitBtn 1 - error..


Goal Level

• Parse button clicks into goal pushes and pops• Classified each push and pop as a hit or error

– Hits are labeled according to the ideal model (i.e., ACT-R productions or other task analysis)

– Errors are further classified into:• Push-error: Ss attempted to do something that the model

would not do.

• Pop-error: Ss stopped working on a goal either before or after the model would stop.


Example 1 - Push-error

Show Time

End Time:20:00

Button Display

• END-MODE E:EE

• HOUR 19:00

• HOUR 20:00

• 10MIN 20:10

• 10MIN 20:20

• 10MIN 20:30

• 10MIN 20:40

• 10MIN 20:50

• 10MIN 20:00

• DOW 20:00

ErrorHit

Error

Push-error

Push-error

Error detection and correction


Example 2 - Pop-error

Show Time

Start Time

21:15

Button Display

• START-MODE E:EE

• HOUR 21:34

• 10MIN 21:44

• MIN 21:45

• 10MIN 21:55

• 10MIN 21:05

• 10MIN 21:15

Pop-errorGoal not accomplished yet.

Error detection and correction


Results

Keypress Goal push/popTotal 9545 6750

Correct Behavior 80.1% 94.81%Errors, Error

Detection, or Error Correction 19.9% 5.19%

Correctly Classified by ACT-PRO 98.95% 99.30%

VCR 2.0 -- Condition A: Ubiquitous Information


Conclusions

• ACT-PRO works very well

• Involves several orders of magnitude less effort than hand encoding

• Well suited to modeling behavior from a large class of interactive devices

• Next challenge:– Condition B - where target settings are covered by gray

boxes and Ss need to click on the gray box to access the information

– Condition C - Same as Cond B, but Ss are forced to memorize the target settings before the experiment

ACT-PRO Action Protocol Tracer A Tool for Analyzing Simple, Rule- based Tasks Wai-Tat Fu & Wayne D....

Documents

Transcript of ACT-PRO Action Protocol Tracer A Tool for Analyzing Simple, Rule- based Tasks Wai-Tat Fu & Wayne D....