Robert W. Reeder and Roy A. Maxion - CUPS · Robert W. Reeder and Roy A. Maxion Dependable Systems...
1
Robert W. Reeder and Roy A. Maxion Dependable Systems Lab, Computer Science Dept., Carnegie Mellon University Improve accuracy rates for Windows XP file- permission setting tasks. Find a replicable approach for designing dependable interfaces in other security domains. Understand cognitive concepts behind why the approach works. 1. Goal errors, which are caused by an interface not providing users with the information necessary to know what to do, dominated in XP. 2. External subgoal support (ESS), a procedure for designers to ensure that all necessary information is available to users, was formulated. 3. Salmon, an alternative to XP, was developed in accordance with the ESS procedure. 4. Salmon and XP were compared in a laboratory user study with 24 participants (12 per interface). Salmon vastly outperformed XP in accuracy and in goal errors committed, and modestly outperformed XP in speed. Windows XP – not designed with external subgoal support Salmon – our interface, designed using external subgoal support Accuracy: Salmon users improved up to 300% over XP users in accuracy Many security vulnerabilities are caused by user interfaces that induce people to make errors. What kinds of errors do people make, what causes these errors, and what can be done to prevent them? Percent successful completions 58 25 83 100 0 25 50 75 100 Wesley task Jack task Task Name Percent of Users Who Correctly Completed Tasks XP XP Salmon Salmon Goal Errors: Salmon users committed up to 94% fewer goal errors than XP users did. Goal error counts 5 15 1 1 0 4 8 12 16 Wesley Jack Task name Total goal errors committed Speed: Successful Salmon users completed tasks moderately faster, on average, than their XP counterparts XP XP Salmon Salmon Salmon Average Time-to-Task-Completion 208 208 183 173 0 50 100 150 200 250 Wesley task Jack task Task Name Time (seconds) XP XP Salmon Salmon p = 0.35 p = 0.20 p = 0.03 p = 0.0005 p = 0.09 p < 0.0001 Steps to complete a typical task with XP: Steps to complete the same task with Salmon: 1 2 3 4 5 Check initial permissions Check group memberships Change permissions Check work Focus: Windows XP file-permission setting interface 1. Observe errors in the native XP file permissions interface. 2. Categorize errors – Goal, Plan, Action, or Perception. (after Pocock et al., 2001) 3. Note dominant form of error; identify its cause. 4. Propose UI design solution to eliminate error. 5. Implement design solution in a new interface 6. Evaluate performance of the new interface against the original in a controlled user study. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Check initial permissions Check group memberships Change permissions Check work
Transcript of Robert W. Reeder and Roy A. Maxion - CUPS · Robert W. Reeder and Roy A. Maxion Dependable Systems...
Robert W. Reeder and Roy A. MaxionDependable Systems Lab, Computer Science Dept., Carnegie Mellon University
Improve accuracy rates for Windows XP file-permission setting tasks.
Find a replicable approach for designing dependable interfaces in other security domains.
Understand cognitive concepts behind why the approach works.
1. Goal errors, which are caused by an interface not providing users with the information necessary to know what to do, dominated in XP.
2. External subgoal support (ESS), a procedure for designers to ensure that all necessary information is available to users, was formulated.
3. Salmon, an alternative to XP, was developed in accordance with the ESS procedure.
4. Salmon and XP were compared in a laboratory user study with 24 participants (12 per interface). Salmon vastly outperformed XP in accuracy and in goal errors committed, and modestly outperformed XP in speed.
Windows XP – not designed with external subgoal support
Salmon – our interface, designed using external subgoal support
Accuracy: Salmon users improved up to 300% over XP users in accuracy
Many security vulnerabilities are caused by user interfaces that induce people to make errors. What kinds of errors do people make, what causes these errors, and what can be done to prevent them?
Percent successful completions
58
25
83
100
0
25
50
75
100
Wesley task Jack task
Task Name
Per
cen
t o
f U
sers
Wh
o
Co
rrec
tly C
om
ple
ted
T
asks
XP
XPS
alm
on
Sal
mo
n
Goal Errors: Salmon users committed up to 94% fewer goal errors than XP users did.
Goal error counts
5
15
1 10
4
8
12
16
Wesley Jack
Task name
To
tal g
oal
err
ors
co
mm
itte
d
Speed: Successful Salmon users completed tasks moderately faster, on average, than their XP counterparts
XP
XP
SalmonSalmon Salmon
Average Time-to-Task-Completion
208 208183 173
0
50
100
150
200
250
Wesley task Jack task
Task Name
Tim
e (s
eco
nd
s)
XP
XP
Sal
mo
n
Sal
mo
n
p = 0.35 p = 0.20
p = 0.03 p = 0.0005p = 0.09 p < 0.0001
Steps to complete a typical task with XP:
Steps to complete the same task with Salmon:
1 2 3
45
Check initial permissions Check group memberships
Change permissionsCheck work
Focus: Windows XP file-permission setting interface
1. Observe errors in the native XP file permissions interface.
2. Categorize errors – Goal, Plan, Action, or Perception. (after Pocock et al., 2001)
3. Note dominant form of error; identify its cause.
4. Propose UI design solution to eliminate error.
5. Implement design solution in a new interface
6. Evaluate performance of the new interface against the original in a controlled user study.
1 2 3 4
5 6
7 8
9
10111213141516
Check initial permissions Check group memberships
Change permissionsCheck work
