1
Human Technology Interaction: Methods
Cato A. Bjørkli, Associate professor, UIO
http://www.regjeringen.no/nb/dep/jd/dok/NOUer/2008/nou-2008-8.html?id=504445
«Undersøkelseskommisjonen skal foreta de undersøkelser den finner nødvendig for å kartleggehendelsesforløp og årsaksfaktorer, og utrede forhold avbetydning for å forebygge at en slik ulykke kan skje i fremtiden.
Kommisjonen har i egen regi foretatt en MTO (Man/Technology/Organisation)-analyse med gjennomgang av hendelsesforløpet fra fartøyet bleplanlagt bygget til forliset for å avdekkesikkerhetsbarrierer og brudd på slike.
p.127-133: Anbefalinger
En ulykkeshendelse er ofte et resultat av flere samvirkende årsaker –både direkte og indirekte årsaker, det vil si forhold som ikke direktehar medført at hendelsen har funnet sted, men har vært medvirkendetil hendelsen eller til at denne ikke ble avverget.
p.78-87: Hendelser forut og under forliset
Background for the lecture
• S.G.Charlton (1996) Mental Workload Test ...
• B.Nardi (1997) The Use of Ethnography Methods...
• (X.Faulkner (2000) Usability Engineering ...)
2
Art is ‘I’
Science is ‘we’
Claude Bernard
(why method?)
-- method --
a way of doing something, especially a systematic way; implies an orderly logical arrangement (usually in fixed steps)
-- science --
ability to produce solutions in some problem domain
approximate d(f)
The Science ofHuman Technology Interaction
(MMI)
What characterizes a good product?
What characterizes a safe system?
What characterizes a efficient system?
How do we know that something is what we designed it to be?
3
Field of Study
Product
Social
Individual
Systems
Your examples?
Life of Products
• How do a product come to life? • How does an artifact come in to existence?• When is a product completely designed?
CS60/65
Know the user
Know the taskKnow thecontext
Usability goalsetting
Know the existingproducts
Design
PrototypingEvaluation
Follow upstudies
ANALYSE DESIGN
COGNITIVEENGINEERING
USABILITYDESIGN
INTERACTIONDESIGN
Methods for all phases of product cycles
4
Evaluation & Redesign
Innovation & Design
Methods underlying all aspects of HTI
A way of gaining insight into the system studied
What insight is useful?
What insight is not useful?
Method is choices
How you approach the phenomenon is guiding for
the methods you chose to investigate it.
So, beyond the type of project, you must consider your perspective
to the system i question.
“... theoretical interpretations serve to provide much of the direction for the research questions and methodologies.”S.G.Charlton, 1996, p.181.
Traditional Information Processing
Embodiment
Social Matrix
Political
Ergonomics
?Engineering
Sociology
Ethical
5
Good news / Bad news
Fact:There are a number of ways for modeling human –technology interaction and equally many ways of measuring it.
Consequence:You must be able to provide arguments for your specific approach and make sure that it corresponds with your choice of measurement.
Considerations
• What system?– Complex– Product– Social
• What phase and focus of your project?– Evaluation of existing system– Redesign– Innovation & Development
• What theoretical perspective?– Traditional Infor Pro– Embodiment– Cognitive Engineering
Concepts and techniques
6
W O R K L O A DCharlton, S. (1996) Mental Workload Test and Evaluation
7
Workload
• Quantitative measurement of a specific transitory subjective state or temporal feature of mental processes describing human technology interaction.
• What is ‘workload’? – Definition? – No 1-to-1 coupling between performance and workload– Multifaceted?– Related to performance– Related to motivation– Related to strategies/timing– Situatedness / Contextuality / Lab versus world– No direct external manifestation
Workload is a multidimensional concept, and should not be estimated with single indicators, but with multivariate measures
NASA TLXSWAT
Yet, some univariateglobal measures yieldgood information
Strengths and weaknesses ?
How to describe a
house
Height? Weight?Shape?
Describe a house
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
8
1. Sensitivity: Can the test show changes in workload?
2. Intrusiveness: Do the test disturb the operator during work?
3. Diagnostical: Which aspect of several is the most important?
4. Convenience: How much logistics are necessary?
5. Relevance: How well do the measurement fit the work?
6. Operator acceptance: Is the test ok for the operator?
Criteria for good workload tests
Types of Tests: Overview
Projective tests• Timeline Analysis• Scheduling Theory• Pro-SWAT
Empirical tests• Modified Cooper-Harper (MCH)• SWAT• NASA TLX• Crew Status Survey
Projective Tests
9
PROJECTIVE TESTS
Time available
T-0 T-N
a b c
Time-Line Analysis and Scheduling
Task Time versus Time Available
Task Difficulty
Demanded skills
Frequencies
Percept/Motor channel
Criteria: task time < 75% of total time
PROJECTIVE TESTS
Pro-SWAT
Task Enactment w/prototype
Detailed Briefing
SWAT scores
Debriefing
(During prototyping)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Cashpoint Interface
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Cellphone Interface
Know the user
Know the taskKnow thecontext
Usability goalsetting
Know the existingproducts
Design
PrototypingEvaluation
Follow upstudies
ANALYSE DESIGN
COGNITIVEENGINEERING
USABILITYDESIGN
INTERACTIONDESIGN
Where do projective tests fit in?
10
Empirical Tests
Empirical Techniques
“... collection of data from one or more subjects actually
performing the task(s) of interest.”
Q: What is relevant data?
Types of Data
• Primary task: Objective variables: – Speed, lateral position, observable
performance
• Secondary task: Additional tasks– Mental calculus, counting backwards
• PsychoPhys: Changes in patterns– Heart, eye, startle, skin conducance
• Subjective methods– Self-reports (NASA, SWAT, etc)
Workload by Inference
Workload by Measure
?
11
Examples from different projects
Example I: Adaptive Front Lights
The objectives of the CLARESCO project is to carry out an extensive assessment of advanced lighting technologies by studying advanced front-lighting systems for both carand truck lighting.
Example I: Adaptive Front Lights
15
20
25
30
35
40
45
50
55
-80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30
Distance from object(m)
Spee
d(km
/h)
AFSNon-AFSDay
Objective variablesSpeed, position
PhysiologicalHeart, eye movement
Subjective variablesNASA TLX, Acceptance
Secondary TaskAlarm detection
12
Example II: Invehicle Information System
Objective variablesSpeed, position
Subjective variablesNASA TLX
GUI vs Speech Recognition
Example III: HSC Sailing
How can this be approached?
-Workload?
-Performance?
-Secondary task?
-Primary Task?Focus:
Design of electronic charts and other navigation equipment
Example III: HSC Sailing
PhysiologicalHeart, Skin Conduct
Subjective variablesNASA TLX, SWAT
Secondary TaskAdHoc Attention Task
Video RecordingLink Analysis
13
Know the user
Know the taskKnow thecontext
Usability goalsetting
Know the existingproducts
Design
PrototypingEvaluation
Follow upstudies
ANALYSE DESIGN
COGNITIVEENGINEERING
USABILITYDESIGN
INTERACTIONDESIGN
What kind of data do we need?
Specific empirical tests
Assumes a three dimensional workload construct
1. Time Load 2. Mental Effort 3. Psych Stress
Procedure:1) Normalize for subjects (scale development)2) Workload Scores (data collection)
Evaluation1) Subscales2) Total score in accordance to scale development
Empirical Technique:SWAT
14
• Simple dimension structure– Subjective fatigue– Workload estimate
• Seven point Likert-scale (1 to 7)– 1 = Fully Alert / Nothing to do– 7 = Completly exhausted / Unacceptable
• Compared to other system information– Length of shifts– Task Complexity– Multitasking
• Later elaborated with more dimensions in the ARWES Questionnaire
Empirical Technique:Crew Status Survey
Empirical Technique:NASA TLX
Procedure
1. Rate all six subscales from 1 to 1002. Make pairwise comparison of subscales3. Multiply rating with weigthing
Evaluation
A) Add all 6 subscales, divide by 15 (comparisons) for total scoreand/or
B) Evaluate workload profile of all 6 subscales
NASA TLX RATING SCALE DEFINITIONS
MENTAL DEMAND (Low/High) How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, etc.)? Was the task easy or demanding, simple or complex?
PHYSICAL DEMAND (Low/High) How much physical activity was required (e.g., pushing, pulling, turning, controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious?
TEMPORAL DEMAND (Low/High) How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? Was the pace slow and leisurely or rapid and frantic?
EFFORT (Low/High) How hard did you have to work (mentally and physically) to accomplish your level of performance?
PERFORMANCE (Good/Poor)How successful do you think you were in accomplishing the goals of the task set by the experimenter? How satisfied were you with your performance in accomplishing these goals?
FRUSTRATION LEVEL (Low/High) How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?
15
see example from excel and program
Empirical Technique:NASA TLX
One step back, again
Types of Tests: Overview
Projective tests• Timeline Analysis• Scheduling Theory• Pro-SWAT
Empirical tests• Modified Cooper-Harper (MCH)• SWAT• NASA TLX• Crew Status Survey
16
1. Sensitivity: Can the test show changes in workload?
2. Intrusiveness: Do the test disturb the operator during work?
3. Diagnostical: Which aspect of several is the most important?
4. Convenience: How much logistics are necessary?
5. Relevance: How well do the measurement fit the work?
6. Operator acceptance: Is the test ok for the operator?
Criteria for good workload tests
SWAT vs NASA TLX?
Alternatives?
Beyond quantitative methods ...
Nardi and Ethnography
• Sometimes you don’t know what you are looking at
• Cognition in the wild (Welcome to the Jungle)
• Golden Rule: Before measuring a phenomenon, you should understand it.
• Main ethnographic activities: – Interviews– Observation– Participation
17
Ethnography: The Choices
• How long should I study this?
• What role should I have?
• What enables me to access information?
• What tools do I have?
• What data do I collect? How? When?
• How do I report ethnographic data?
Ethnography, cont
Your role and your mission
• Goal: Reflect and state your goal for observation. What do I want to achieve? What is success?
• Method: How do can I get to the information and by what means? Tools? Methods?
• Conditions: What resources do I have available? Time and money? Equipment?
• Analysis: What will be the format of data? How do I make sense of experiences? What do I bring back?
Know the user
Know the taskKnow thecontext
Usability goalsetting
Know the existingproducts
Design
PrototypingEvaluation
Follow upstudies
ANALYSE DESIGN
COGNITIVEENGINEERING
USABILITYDESIGN
INTERACTIONDESIGN
What kind of data do we need?
18
Ethnography: How to report?
Analysis is difficult with qualitative data (show dvd)
The choice of methods ...... in accordance to what?
Questions?
19
PSY2403 Man Machine Interaction:Method I
Cato A. BjørkliPsychologist
Top Related