Component-specific usability testing
Dr Willem-Paul BrinkmanLecturer
Department of Information Systems and Computing
Brunel University([email protected])
Topics Introduction
Whether and how the usability of components can be tested empirically.
- Testing different versions of component- Testing different components
Whether and how the usability of components can be affected by other components.
- Consistency- Memory load
Layered Protocol Theory
(Taylor, 1988)
15 + 23 =
15+23=
01111
10111
Add
100110
38
Processor
Editor
Control results
Control equation
User Calculator
15
15
15 +
15 +
15 + 23
15 + 23
38
38
Usability Testing
Aim to evaluate the usability of a component based on the message exchange between a user and a specific component
Two paradigms
Create
S upport
Reuse
M anage
new components
components from repos ito ry
productsP roduct requirementsand exis ting so ftware
feedback
feedback
Multiple versions testing paradigm
Single version testing paradigm
Manage
Support
Re-use
Create
Test Procedure
Normal procedures of a usability test
User task which requires interaction with components under investigation
Users must complete the task successfully
Component-specific component measures
Number of messages received
The effort users put into the interaction
Objective performance
Perceived ease-of-use
Perceived satisfaction
ComponentControl process
Control loop
Component-specific component measures
Increasing the statistical power
Objective performance
Perceived ease-of-use
Perceived satisfaction
y
y1 = xk+ k
y2 = xm+ m
k = k component + k rest
m = m component + m rest
Assumption
k rest m rest
messages
keys
Component-specific component measures
Objective performance
Perceived ease-of-use
Perceived satisfaction
Component-specific questionnaire increase the statistical power because they help help the users to remember their control experience with a particular interaction component
Component-specific component measures
Objective performance
Perceived ease-of-use
Perceived satisfaction
Perceived Usefulness and Ease-of-use questionnaire (David, 1989), 6 questions, e.g.
Learning to operate [name] would be easy for me.
I would find it easy to get [name] to do what I want it to do.
Unlikely Likely
Component-specific component measures
Objective performance
Perceived ease-of-use
Perceived satisfaction
Post-Study System Usability Questionnaire (Lewis, 1995)
The interface of [name] was pleasant.
I like using the interface of [name].
StronglyStrongly
disagree agree
Experimental validation
80 users8 mobile telephones3 components were manipulated
according to Cognitive Complexity Theory (Kieras & Polson, 1985)
1. Function Selector 2. Keypad3. Short Text Messages
Architecture Mobile telephone
Send Text Message
Send Text Message Function
SelectorFunction Selector
KeypadKeypad
Results
Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80
Number of subjects
Po
wer
1. messages received
2. corrected messagesreceived
3. task duration
4. keystrokes
5. corrected keystrokes
6. comp.-spec. ease-of-use7. comp.-spec. satisfaction
8. overall eas-of-use
9. overall satisfaction
Wilcoxon Matched-Pairs Signed-Ranks Tests between the number of correct classification made by discriminant analyses on overall and component-specific measures
Correctly classified
Dimension Overall Component-Specific N T p
Observed performance 62% 78% 37 3 <0.001
Perceived ease-of-use 61% 60% 62 30 0.907
Perceived satisfaction 58% 61% 61 27 0.308
Results
Topics Introduction
Whether and how the usability of components can be tested empirically.
- Testing different versions of component- Testing different components
Whether and how the usability of components can be affected by other components.
- Consistency- Memory load
Two paradigms
Create
S upport
Reuse
M anage
new components
components from repos ito ry
productsP roduct requirementsand exis ting so ftware
feedback
feedback
Multiple versions testing paradigm
Single version testing paradigm
Manage
Create
Support
Re-use
Testing Different Components
Component specific objective performance
measure:1. Messages received + Weight factor
A common currency
2. Compare with ideal userA common point of reference
Usability of individual components in a single device can be compared with each other and prioritized on potential improvements
Click <right>Click <left on Properties option>
{1}{1}
Click <left on Fill tab>Click <left on on colour red>Click <left on Outline tab>Click <left No Line button>Click <left no Ok button>
{1}{1}{1}{1}{1}
Call <>{2}
Set <Fill colour red, no border>{7}
Right MouseButton Menu
Properties
Assigning weight factors to represent the user’s effort in the case of ideal user
Total effort value
Total effort = MRi.W
• MRi.W : Message received. Weight factor
Click <right>
Click <left on Properties option>
{1}{1}
Click <left on Fill tab>Click <left on on colour red>Click <left on Outline tab>Click <left No Line button>Click <left no Ok button>
{1}{1}{1}{1}{1}
Call <>{2}
Right MouseButton Menu
Properties
5 2 = 7+2
Assigning weight factors in case of real user
Correction for inefficiency of higher and lower components
Visual Drawing Objects
Properties
Right MouseButton Menu
Assigning weight factors in case of real user
Assign weight factors as if lower components operate optimal
Visual Drawing Objects
Properties
Right MouseButton Menu
Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required
Assigning weight factors in case of real user
Visual Drawing Objects
Properties
Right MouseButton Menu
Inefficiency of higher level components: more messages are requested than ideally required
UE : User effort
MRi.W : Message received. Weight factor
#MSUreal :Number of messages sent upward by real user
#MSUideal :Number of messages sent upward by ideal user
MRi.W
#MSU real
#MSU ideal
UE =
Ideal User versus Real User
Extra User Effort = User Effort - Total effort
The total effort an ideal user would make
The total effort a real user made
The extra effort a real user made
Calculate for each component:
Prioritize
Experimental validation
40 users40 mobile telephones2 components were manipulated
(Keypad only Repeated-Key Method)
1. Function Selector 2. Short Text Messages
Results
010
20304050
6070
Broad & Simple
Narrow & Simple
Broad &Complex
Narrow &Complex
1 2 3 4
Function Selector
Send Text Message
Mobile phones
Ext
ra U
ser
Eff
ort
Results
Measure Function Selector
Send Text Message
Objective
Extra keystrokes 0.64** 0.44**
Task duration 0.63** 0.39**
Perceived
Overall ease-of-use -0.43** -0.26*
Overall satisfaction -0.25* -0.22
Component-specific ease-of-use -0.55** -0.34**
Component-specific satisfaction -0.41** -0.37**
Partial correlation between extra user effort regarding the two components and other usability measures
*p. < .05. **p. < .01.
Comparison with other evaluation methods
Overall measuresSequential Data
analysisGOMSThinking-aloud,
Cognitive Walkthrough and heuristic evaluation
Example: Keystrokes, task duration, overall perceived usability
Relatively easy to obtain
Unsuitable to evaluate components
Overall measuresSequential Data
analysisGOMSThinking-aloud,
Cognitive Walkthrough and heuristic evaluation
Based only on lower-level events
Pre-processing: selection, abstraction, and re-coding
Relation between higher-level component and compound message less direct
Components’ status not recorded
Comparison with other evaluation methods
Help to understand the problem
Only looking at error-free task execution
Considers the system only at the lowest-level layer
Overall measuresSequential Data
analysisGOMSThinking-aloud,
Cognitive Walkthrough and heuristic evaluation
Comparison with other evaluation methods
Quicker
Evaluator effect (reliability)
Overall measuresSequential Data
analysisGOMSThinking-aloud,
Cognitive Walkthrough and heuristic evaluation
Comparison with other evaluation methods
Topics Introduction
Whether and how the usability of components can be tested empirically.
- Testing different versions of component- Testing different components
Whether and how the usability of components can be affected by other components.
- Consistency- Memory load
Consistency experiments
48 Users Used 3 applications:
1. 4 Room Thermostats
2. 4 (2 Web-Enabled TV sets 2 Web Page Layouts)
3. 4 Applications (2 Timers 2 Application domains)
Within one layer – Experimental Design
Day time TemperatureN
ight
tim
e T
em
pera
ture
Moving Pointer
Moving Scale
Movin
g
Poin
ter
Movin
g
Sca
le
Daytime version
moving scalemoving pointer
Mes
sage
s N
ight
time
Tem
pera
ture
24
21
18
15
12
moving scale
moving pointer
Within on layer - Results
Web Pages version
matrixlist
Mes
sage
s B
row
ser
40
30
20
10
0
linear-oriented
plane-oriented
Between layers - Results
Between Application domain – Experimental Design
ApplicationTim
er
Alarm radio MicrowaveM
ech
an
ical
ala
rmH
ot
dis
h
Application domain
MicrowaveAlarm radio
Num
ber
of M
ode
mes
sage
s re
ceiv
ed27
24
21
18
15
12
9
hot dish
mechanical alarm
Application domain - Results
Topics Introduction
Whether and how the usability of components can be tested empirically.
- Testing different versions of component- Testing different components
Whether and how the usability of components can be affected by other components.
- Consistency- Memory load
Equation
difficulteasy
Log-
tran
sfor
med
HR
V 0
.1 H
z B
and
-2.5
-2.6
-2.7
-2.8
-2.9
-3.0
-3.1
large display
small display
Mental Effort - Heart-rate variability
Equation
difficulteasy
tran
sfor
med
num
ber
of s
tore
req
uest
s+1 1.2
1.0
.8
.6
.4
.2
small display
large display
Mental Effort - Control of higher-level layer
Conclusions Whether and how the usability of
components can be tested empirically.- Testing different versions of component : more powerful- Testing different components : prioritized on potential
improvements
Whether and how the usability of components can be affected by other components.- Consistency : components on the same or on higher-
level layers can activate wrong mental models- Memory load : lower-level interaction affects higher-
level interaction strategy
Top Related