1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.
-
date post
15-Jan-2016 -
Category
Documents
-
view
220 -
download
0
Transcript of 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.
![Page 1: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/1.jpg)
1
Action Analysis Automated Evaluation
![Page 2: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/2.jpg)
2
Hall of Fame or Hall of Shame?
java.sun.com
![Page 3: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/3.jpg)
3
Hall of Fame
Good branding java logo value prop
Inverse pyramid writing style
Fresh content changing first
read news in sidebar
![Page 4: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/4.jpg)
4
Hall of Fame or Hall of Shame?
Bryce 2 for building 3D
models
![Page 5: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/5.jpg)
5
Hall of Shame!
Icons all look similar what do they do????
How do you exit? Note
nice visuals, but must be usable
What if purely for entertainment?
![Page 6: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/6.jpg)
6
Outline
Action analysis GOMS? What’s that? The G, O, M, & S of GOMS How to do the analysis Announcements Automated evaluation tools
![Page 7: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/7.jpg)
7
Action Analysis Predicts Performance
Cognitive model model some aspect of human understanding, knowledge,
intentions, or processing two types
competence predict behavior sequences
performance predict performance, but limited to routine behavior
Action analysis uses performance model to analyze goals & tasks generally done hierarchically (similar to TA)
![Page 8: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/8.jpg)
8
GOMS – Most Popular Action Analysis
Family of UI modeling techniques based on Model Human Processor
GOMS stands for (?) Goals Operators Methods Selection rules
Input: detailed description of UI/task(s) Output: qualitative & quantitative measures
![Page 9: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/9.jpg)
9
Quick Example
Goal (the big picture) go from hotel to the airport
Methods (or subgoals)? walk, take bus, take taxi, rent car, take train
Operators (or specific actions) locate bus stop; wait for bus; get on the bus;...
Selection rules (choosing among methods)? Example: Walking is cheaper, but tiring and slow Example: Taking a bus is complicated abroad
![Page 10: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/10.jpg)
10
Goals
Something the user wants to achieve Examples?
go to airport delete File create directory
Hierarchical structure may require many subgoals
![Page 11: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/11.jpg)
11
Methods
Sequence of steps to accomplish a goal goal decomposition can include other goals
Assumes method is learned & routine Examples
drag file to trash retrieve from long-term memory command
![Page 12: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/12.jpg)
12
Operators
Specific actions (small scale or atomic) Lowest level of analysis
can associate with times Examples
Locate icon for item on screen Move cursor to item Hold mouse button down Locate destination icon User reads the dialog box
![Page 13: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/13.jpg)
13
Selection Rules
If > 1 method to accomplish a goal, Selection rules pick method to use
Examples IF <condition> THEN accomplish <GOAL> IF <car has automatic transmission> THEN
<select drive> IF <car has manual transmission> THEN <find
car with automatic transmission>
![Page 14: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/14.jpg)
14
GOMS Output
Execution time add up times from operators assumes experts (mastered the tasks) error free behavior very good rank ordering absolute accuracy ~10-20%
Procedure learning time (NGOMSL only) accurate for relative comparison only doesn’t include time for learning domain knowledge
![Page 15: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/15.jpg)
15
GOMS Output
Ensure frequent goals achieved quickly Making hierarchy is often the value
functionality coverage & consistency does UI contain needed functions? consistency: are similar tasks performed similarly?
operator sequence in what order are individual operations done?
![Page 16: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/16.jpg)
16
How to do GOMS Analysis
Generate task description pick high-level user Goal write Method for accomplishing Goal - may invoke
subgoals write Methods for subgoals
this is recursive stops when Operators are reached
Evaluate description of task Apply results to UI Iterate!
![Page 17: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/17.jpg)
17
Comparative Example - DOS
Goal: Delete a File Method for accomplishing goal of deleting a file
retrieve from Long term memory that command verb is “del”
think of directory name & file name and make it the first listed parameter
accomplish goal of entering & executing command return with goal accomplished
![Page 18: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/18.jpg)
18
Comparative Example - Mac
Goal: Delete a File Method for accomplishing goal of deleting a
file find file icon accomplish goal of dragging file to trash Return with goal accomplished
![Page 19: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/19.jpg)
19
Comparative Example - DOS
Goal: Remove a directory Method for accomplishing goal of removing a
directory -
![Page 20: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/20.jpg)
20
Comparative Example - DOS
Goal: Remove a directory Method for accomplishing goal of removing a
directory Accomplish goal of making sure directory is empty Retrieve from long term memory that command verb is
‘RMDIR’ Think of directory name and make it the first listed
parameter Accomplish goal of entering and executing a command Return with goal accomplished
![Page 21: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/21.jpg)
21
Comparative Example - MAC
Goal: Remove a directory Method for accomplishing goal of removing a
directory ????
![Page 22: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/22.jpg)
22
Comparative Example - MAC
Goal: Remove a directory Method for accomplishing goal of removing a
directory Find folder Accomplish goal of dragging folder to trash Return with goal accomplished
![Page 23: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/23.jpg)
23
Applications of GOMS
Compare different UI designs Profiling (time) Building a help system? Why?
modeling makes user tasks & goals explicit can suggest questions users will ask & the
answers
![Page 24: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/24.jpg)
24
What GOMS can model
Task must be goal-directed some activities are more goal-directed
creative activities may not be as goal-directed
Task must be a routine cognitive skill as opposed to problem solving good for things like machine operators
Serial & parallel tasks (CPM-GOMS)
![Page 25: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/25.jpg)
25
Real-world GOMS Applications
Keystroke Level Model (KLM) Mouse-based text editor Mechanical CAD system
NGOMSL TV control system Nuclear power plant operator’s associate
CPM-GOMS Telephone operator workstation
![Page 26: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/26.jpg)
26
Advantages of GOMS
Gives qualitative & quantitative measures Model explains the results Less work than user study – no users! Easy to modify when UI is revised Research: tools to aid modeling process since it
can still be tedious
![Page 27: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/27.jpg)
27
Disadvantages of GOMS
Not as easy as HE, guidelines, etc. Takes lots of time, skill, & effort Only works for goal-directed tasks Assumes tasks performed by experts without
error Does not address several UI issues,
readability, memorizability of icons, commands
![Page 28: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/28.jpg)
28
Rapid Iterative Design is the Best Practice for Creating Good UIs
Design
Prototyping
Evaluation
We have seen how computer-based tools can improvethe Design (e.g., Denim) & Prototyping (e.g., VB) phases
![Page 29: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/29.jpg)
29
Automated GOMS Tools
Can save, modify and re-use the model Automation of goal hierarchy, method,
selection rule creation
![Page 30: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/30.jpg)
30
QGOMS tool
![Page 31: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/31.jpg)
31
CRITIQUEHudson et al (1999)
1. Prototype system in this case with the SubArctic toolkit
2. Demonstrate a procedure (task) record events apply rules
3. Automatically generate KLMs
4. Semi-automatically generate classic GOMS models
![Page 32: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/32.jpg)
32
Automated Web Evaluation
Motivation Approaches
GOMS inspired models remote usability testing
![Page 33: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/33.jpg)
33
Factors Driving Repeat Visits Should Drive Evaluation
High quality content 75% Ease of use 66% Quick to download 58%
(Source: Forrester Research, 1/99)
![Page 34: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/34.jpg)
34
Max – WebCriteria’s GOMS Model
Predicts how long information seeking tasks would take on a particular web site
Automated procedure:seed with start page and goal page
procedurereads page
model predicts how long to find & click proper linkload time, scan time, and mouse movement time
repeat until find goal page
Claim time is directly related to usability
![Page 35: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/35.jpg)
35
Sample of Max’s Reports
![Page 36: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/36.jpg)
36
Advantages of Max-style Model
Inexpensive (no users needed) Fast (robot runs & then computes model) Can run on many sites & compare ->
benchmarks
![Page 37: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/37.jpg)
37
Disadvantages of Max-style Model
Focus on time (much of it download time) only 3rd in important factors driving repeat visits can’t tell you anything about your content doesn’t say anything directly about usability problems
Robots aren’t humans doesn’t make mistakes
remember, GOMS assumes expert behavior! doesn’t account for understanding text only tries the best path – users will use many
Major flaw is the lack of real users in the process
![Page 38: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/38.jpg)
38
The Trouble With Current Site Analysis Tools
Unknowns Who? What? Why? Did they find it? Satisfied?Leave
![Page 39: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/39.jpg)
39
NetRaker Provides User-centric Remote Evaluation Using Key Metrics
NetRaker Index short pop-up survey shown to 1 in n visitors on-going tracking & evaluation data
Market Research & Usability Templates surveys & task testing invitation delivered through email, links, or pop-ups
![Page 40: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/40.jpg)
40 Small number of rotated questions increases response rate
NetRaker Index: On-going customer intelligence gathering
![Page 41: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/41.jpg)
41 Small number of rotated questions increases response rate
NetRaker Index: On-going customer intelligence gathering
![Page 42: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/42.jpg)
42 Small number of rotated questions increases response rate
NetRaker Index: On-going customer intelligence gathering
![Page 43: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/43.jpg)
43 Small number of rotated questions increases response rate
NetRaker Index: On-going customer intelligence gathering
![Page 44: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/44.jpg)
44
NetRaker Index: On-going customer intelligence gathering
Increasing these indices (e.g., retention) moderately (5%) leads to a large increase in revenue growth
![Page 45: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/45.jpg)
45
NetRaker Usability Research:See how customers accomplish real tasks on site
![Page 46: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/46.jpg)
46
NetRaker Usability Research:See how customers accomplish real tasks on site
![Page 47: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/47.jpg)
47
NetRaker Usability Research:See how customers accomplish real tasks on site
![Page 48: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/48.jpg)
48
NetRaker Usability Research:See how customers accomplish real tasks on site
![Page 49: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/49.jpg)
49
WebQuilt: Visual Analysis Goals
link page elements to user actions identify behavior/nav. patterns highlight potential problems areas
Solution interactive graph based on web content
nodes represent web pages edges represent aggregate traffic between pages
designers can indicate expected paths color code common usability interests filtering to show only target particpants use zooming for analyzing data at varying granularity
![Page 50: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/50.jpg)
50
![Page 51: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/51.jpg)
51
![Page 52: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/52.jpg)
52
![Page 53: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/53.jpg)
53
Advantages of NetRaker
Fast can set up research in 3-4 hours get results in 36 hours
More accurate can run with large samples (50-200 users -> stat. sig.) uses real people (customers) performing tasks natural environment (home/work/machine)
Easy-to-use templates make setting up easy
Can compare with competitors indexed to national norms
![Page 54: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/54.jpg)
54
Disadvantages of NetRaker
Miss observational feedback facial expressions verbal feedback (critical incidents)
Need to involve human participants costs some amount of money (typically
$20-$50/person) People often do not like pop-ups
need to be careful when using them
![Page 55: 1 Action Analysis Automated Evaluation. 2 Hall of Fame or Hall of Shame? java.sun.com.](https://reader035.fdocuments.us/reader035/viewer/2022062314/56649d435503460f94a1f9fc/html5/thumbnails/55.jpg)
55
Summary
GOMS provides info about important UI properties doesn’t tell you everything you want to know about a UI
only gives performance for expert behavior hard to create model, but still easier than user testing
changing later is much less work than initial generation Automated usability
faster than traditional techniques can involve more participants -> convincing data easier to do comparisons across sites tradeoff with losing observational data