Measuring users’ experiences
Evangelos Karapanos
or, the memory of them?
Skopje, 5 June 2012
Trajectory reminders EmoSnaps Footprint tracker iScale
Wednesday, June 6, 12
BSc Physics / microelectronics, U Patras, Greece (2004)Thesis: Model based design and evaluation of walk-up-and-use interfaces (HCI Group, ECE department)
My background
MSc HCI / UCL Interaction Centre, UK (2005)Thesis: User acceptance of nomadic user interfaces (Philips Research, Eindhoven)
PhD HCI / TU Eindhoven, NL (2010)Title: Quantifying diversity in user experience
Assist. Prof. HCI / Madeira ITI (2010-)Design for Experience in pervasive computing
Wednesday, June 6, 12
48% of returned products are not attributed to a violation of product specifications
Soft Reliability
Wednesday, June 6, 12
problems rooted early in (concept) design phase
Wednesday, June 6, 12
failure to truly incorporate it in one’s life
Wednesday, June 6, 12
1
Modeling Users' Experiences w
ith Interactive Systems
436
Over the past decade the field of Human-Computer Interaction has evolved from the study of the usability of interactive products towards a more holistic understanding of how they may mediate desired human experiences.
This book identifies the notion of diversity in users? experiences with interactive products and proposes methods and tools for modeling this along two levels:(a) interpersonal diversity in users? responses to early conceptual designs, and(b) the dynamics of users? experiences over time.
The Repertory Grid Technique is proposed as an alternative to standardized psychometric scales for modeling interpersonal diversity in users? responses to early concepts in the design process, and new Multi-Dimensional Scaling procedures are introduced for modeling such complex quantitative data.
iScale, a tool for the retrospective assessment of users? experiences over time is proposed as an alternative to longitudinal field studies, and a semi-automated technique for the analysis of the elicited experience narratives is introduced. Through these two methodological contributions, this book argues against averaging in the subjective evaluation of interactive products. It proposes the development of interactive tools that can assist designers in moving across multiple levels of abstraction of empirical data, as design-relevant knowledge might be found onall these levels.
Foreword by Jean-Bernard Martens and Closing Note by Marc Hassenzahl.
issn 1860-949X
isbn 978-3-642-30999-1
springer.com
The series Studies in Computational Intelligence (SCI) publishes new developments and advances in the various areas of computational intelligence – quickly and with high quality. The intent is to cover the theory, applications, and design methods of computational intelligence, as embedded in the fields of engineering, computer science, physics and life sciences, as well as the methodologies behind them. The series contains monographs, lecture notes and edited volumes in com putational intelligence spanning the areas of neural networks, connectionist systems, genetic algorithms, evolutionary computation, artificial intelligence, cellular automata,self-organizing systems, soft computing, fuzzy systems, hybrid intelligent, and virtual reality systems. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution, which enable both wide and rapid dissemination of research output.
Studies in Computational Intelligence
1 3
436
Modeling Users'Experiences withInteractive Systems
Evangelos Karapanos
Karapanos
9 7 8 3 6 4 2 3 0 9 9 9 1
Coming!June 2012
Foreword: Jean-Bernard MartensClosing note: Marc Hassenzahl
Wednesday, June 6, 12
Wednesday, June 6, 12
• 20 faculty– 14 countries, 8 languages
• Areas:– 11 CS, 2 physics/electronics, 2 psychology, 2 architecture, 2 design, 2 art, 2 other
Leonel Nóbrega
Ian Oakley Luis Gomes
ValenIna Nisi
Paulo Sampaio
Gonçalo GouveiaLaura Rodríguez
Vassilis Kostakos
Jos van Leeuwen
Nuno Nunes
Larry ConstanIne
Eduardo Fermé
Néstor Catano
Monchu Chen
Pedro Campos
Barbara Pizzileo
Ron Salden
Evangelos Karapanos
David Aveiro Luis Gomes Yoram Chisik
Wednesday, June 6, 12
MSc HCI & Entertainment Technology
Wednesday, June 6, 12
Industry Involvement
Wednesday, June 6, 12
Socially translucent eco-feedback technologies
!
How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.
Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance
Location-aware narratives: Does locality matter?
Design for Experience in pervasive compu3ng
Does the coupling between physical and virtual space result to increased immersion in the narrative world?
Awareness technologies for parents, children and schoola) Using mobile sensors to infer the physical, verbal and social activity of a pupilb) To support interpersonal connectedness within familyc) To engage parents and school in ad-hoc communication
Technologies for Social Inclusion in primary schoolsa) Using sociometric technologies to assess the inclusiveness of school communitiesb) Designing Persuasive technologies that challenge pupils’ perceptions of diversity
ABSTRACT We introduce Senseµ (application that aims at supporting awareness in parent
Wednesday, June 6, 12
Socially translucent eco-feedback technologies
!
How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.
Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance
Location-aware narratives: Does locality matter?
Design for Experience in pervasive compu3ng
Does the coupling between physical and virtual space result to increased immersion in the narrative world?
Awareness technologies for parents, children and schoola) Using mobile sensors to infer the physical, verbal and social activity of a pupilb) To support interpersonal connectedness within familyc) To engage parents and school in ad-hoc communication
Technologies for Social Inclusion in primary schoolsa) Using sociometric technologies to assess the inclusiveness of school communitiesb) Designing Persuasive technologies that challenge pupils’ perceptions of diversity
ABSTRACT We introduce Senseµ (application that aims at supporting awareness in parent
Wednesday, June 6, 12
Socially translucent eco-feedback technologies
!
How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.
Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance
Location-aware narratives: Does locality matter?
Design for Experience in pervasive compu3ng
Does the coupling between physical and virtual space result to increased immersion in the narrative world?
Awareness technologies for parents, children and schoola) Using mobile sensors to infer the physical, verbal and social activity of a pupilb) To support interpersonal connectedness within familyc) To engage parents and school in ad-hoc communication
Technologies for Social Inclusion in primary schoolsa) Using sociometric technologies to assess the inclusiveness of school communitiesb) Designing Persuasive technologies that challenge pupils’ perceptions of diversity
ABSTRACT We introduce Senseµ (application that aims at supporting awareness in parent
Wednesday, June 6, 12
Socially translucent eco-feedback technologies
!
How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.
Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance
Location-aware narratives: Does locality matter?
Design for Experience in pervasive compu3ng
Does the coupling between physical and virtual space result to increased immersion in the narrative world?
Awareness technologies for parents, children and schoola) Using mobile sensors to infer the physical, verbal and social activity of a pupilb) To support interpersonal connectedness within familyc) To engage parents and school in ad-hoc communication
Technologies for Social Inclusion in primary schoolsa) Using sociometric technologies to assess the inclusiveness of school communitiesb) Designing Persuasive technologies that challenge pupils’ perceptions of diversity
ABSTRACT We introduce Senseµ (application that aims at supporting awareness in parent
Wednesday, June 6, 12
Socially translucent eco-feedback technologies
!
How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.
Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance
Location-aware narratives: Does locality matter?
Design for Experience in pervasive compu3ng
Does the coupling between physical and virtual space result to increased immersion in the narrative world?
Awareness technologies for parents, children and schoola) Using mobile sensors to infer the physical, verbal and social activity of a pupilb) To support interpersonal connectedness within familyc) To engage parents and school in ad-hoc communication
Technologies for Social Inclusion in primary schoolsa) Using sociometric technologies to assess the inclusiveness of school communitiesb) Designing Persuasive technologies that challenge pupils’ perceptions of diversity
ABSTRACT We introduce Senseµ (application that aims at supporting awareness in parent
Wednesday, June 6, 12
Measuring users’ experiences
Skopje, 5 June 2012
or, the memory of them?
Trajectory reminders EmoSnaps Footprint tracker iScale
Wednesday, June 6, 12
Technology Assisted Reconstruction
Skopje, 5 June 2012
Trajectory reminders EmoSnaps Footprint tracker iScale
Wednesday, June 6, 12
User Experience defined
a momentary, primarily evaluative feeling (good-bad) while interacting with a product or service
Hassenzahl, 2008
Wednesday, June 6, 12
Most of our evaluation tasks rely on memory
Wednesday, June 6, 12
• Peak-and-end phenomenon
• Summary judgments can be best predicted by a simple average of the most extreme and the last experience (Fredrickson and Kahneman, 1993)
• Replicated in HCI - Summarizing mental effort to perceived usability: end matters (Hassenzahl and Sandweg, 2004)
• Other biases (e.g., Rosy retrospection, Focusing illusion etc.)
Heuristics and biases
Wednesday, June 6, 12
Why do memory biases exist?Robison & Clore (2002)
“The emotional experience can neither be stored nor retrieved”
it is reconstructed from recalled contextual details
Wednesday, June 6, 12
Why do memory biases exist?
produce the exact past event, but instead, every attempt to recall results in a new, often altered representation of the event. Bartlett asked participants to recall an unfamiliar story that they were told 20 hours before. Recalled stories differed from the original one in missing details, altering the order and importance of events, or in applying rationalizations and interpretations to the original story. Stories were further distorted through repeated reconstruction.The notion that remembering is an act of reconstruction instead of mere reproduction has received wide support. At the heart of reconstruction lies the distinction between episodic and semantic memory [69]. While episodic memory “is specific to a particular event from the past, semantic memory is not tied to any particular event but rather consists of certain generalizations (i.e. beliefs) that are rarely updated” (p. 935). These two types of memory serve different needs such as learning new information quickly - a capacity of episodic memory - or developing relatively stable expectations about the world - a capacity of semantic memory [63]. Reconstruction happens through the retrieval of cues from episodic memory. In the absence of contextual cues in episodic memory, beliefs found in semantic memory may be used to reconstruct the past, resulting in distortions such as the ones found in Barlett’s study. Thus, overall, the accuracy of one’s remembered events lies in the degree to which contextual cues are still present in the person’s episodic memory.
But, how do we reconstruct emotional experiences that contain not only contextual details of the experienced event, but also value-charged information such as emotions or overall evaluative judgments on the event?
Robinson and Clore [63] argue that “emotional experience can neither be stored nor retrieved” (p. 935), but can only be reconstructed on the basis of recalled contextual cues. They propose an accessibility model that distinguishes between four types of knowledge used to construct an emotion (see figure 1). First, experiential knowledge is used when an emotion is constructed online, i.e. as the experience takes place. It is considered the only “authentic” source of information, yet it is accessible for only a small fraction of time, about 3 seconds [37]. Self-reports tapping into experiential information, such as ESM, are considered the “gold standard” of experience measurement [36] as they avoid biases caused by retrospection.
When experiential knowledge is inaccessible, people will resort to episodic information, i.e. they will recall contextual cues from episodic memory and infer their emotions on the basis of these contextual cues. For instance, in recalling one’s emotions while experiencing a roller coaster, one might recall having his hands raised and screaming, and on this basis to infer an aroused experience. The accuracy with which people may recall their emotions will thus depend on the richness of recalled episodic
information. Daniel Kahneman and colleagues [36] relied on this proposition to develop the day reconstruction method. DRM asks participants to list their daily activities as a continuous list of episodes [36]. This is assumed to form stronger temporal and semantic links across the distinct experiences, thereby minimizing bias from overly relying on semantic information, which is detached from the actual experience. It has been shown that DRM provides a reasonably good approximation to experience sampling data, both in between-subjects [36] and within-subject [20] analyses. When episodic memories become inaccessible, people will shift to semantic memory. People will first access situation-specific beliefs, i.e. “a belief about the emotions that are likely to be elicited in a particular type of situation” [63]. If event-specific beliefs are inaccessible, e.g. due to rarity of the event, people will access identity-related beliefs, i.e. “beliefs about their emotions in general”. Such information is detached from the actual experience and is associated with retrospection biases found in global reports of experience such as online surveys and questionnaires used at the end of empirical studies [65].
A REVIEW OF EXPERIENCE SAMPLING AND DAY RECONSTRUCTION STUDIESThe experience sampling method has been popular in HCI for over a decade. Yet, to our knowledge, no attempt has been made to review the use of the method by HCI researchers. In the following section we attempt a review of empirical studies, published within HCI, that employ either the experience sampling or the day reconstruction method. The questions we will attempt to adress are: How are these methods used within HCI? What are the benefits these methods offer in addressing particular HCI problems? And, what drawbacks are experienced in practice?
Experience samplingAs ES studies have been published across a wide range of HCI venues we decided that narrowing our search down to
2
Figure 1. Four sources of information in emotional self-report according to Robinson and Clore [63]. Figure adapted from [63].
Experiential Knowledge
Online emotion, e.g.Experience Sampling
Episodic memory
Situation-specific belief
Identity-related belief
Retrospective, e.g.Day Reconstruction
Exit questionnaires
Type of Knowledge Source of information Type of Self-Report
Exit questionnaires
Semantic
Semantic
Episodic
Episodic
Robison & Clore (2002)
“The emotional experience can neither be stored nor retrieved”
it is reconstructed from recalled contextual details
Robinson & Clore (2002)
Wednesday, June 6, 12
bottom line...
If you want to know what the user really experiences, ask her at that exact moment!
Wednesday, June 6, 12
Experience Sampling Method
Prompts at random, or computationally estimated times, to self-report on ongoing behaviors and experiences.
– Where are you?– What are you doing?– How far is your mobile phone? – How do you feel?
Karapanos, E. (2012) Experience Sampling, Day Reconstruction, what’s next? Towards Technology-Assisted Reconstruction. M-ITI internal report.
Wednesday, June 6, 12
Experience Sampling Method
a few relevant venues would still miss a substantial number of studies. We instead used the ACM Guide to Computing Literature querying for the term “experience sampling” without constraining to particular venues. This query returned 284 papers, published in more than fifty venues.
Figure 2. 243 papers referring to experience sampling over a ten-year period. Retrieval took place on August 26th, 2011.
Fourty-one papers were excluded from further processing. These were: references to proceedings of conferences (n=13), papers that dealt with research problems that were beyond the scope of HCI such as computer networks, transport economics and organizational psychology (n=16), papers where no traces of the term “experience sampling” could be found (n=6), most likely caused by errors in the retrieval process such as due to stemming algorithms, papers that were written in languages other than English (n=3), papers that used the term ‘experience sampling’ to refer to concepts other than the method (n=2), and a single paper that was found to be a duplicate entry.
The remaining 243 papers were further classified with respect to being in one of the following types (Interrater agreement of two coders on a random subset of 60 papers, Fleisch K=0.74): empirical studies using the experience sampling method (n=59), empirical studies that considered but decided against the use of ESM (n=21), tools for carrying out experience sampling studies (n=20), systems that borrow concepts from ESM (n=10), papers discussing particular methodological aspects of ESM such as psychometric scales (n=13), reflection papers discussing conceptual issues of in-situ measurement (n=17), and papers discussing ESM as prior (n=85) or future (n=18) work.
For the scope of this paper we will analyze in detail only two categories of papers:
a) 59 papers reporting empirical studies that used the experience sampling method, and
b) 21 papers reporting empirical studies that considered, but decided against the use of the experience sampling method.
Out of the 59 papers that reported empirical studies, eight were found to report a study that was published earlier, and two cited incorrectly the use of experience sampling
0
15
30
45
60
2001 2003 2005 2007 2009 2011
method while following a user-initiated diary approach. The analysis of the remaining 49 studies is being reported below.
Study length, sampling frequency, and response rateThe majority (80%) of the studies had a duration of several days up to one month with 14 studies (34%) lasting between four and seven days (see figure 3). Only two studies had a duration of more than a month.
Figure 3. Distribution of 41 experience sampling studies over their length. Eight papers did not report study length.
Thirty nine papers reported the particular sampling approach that was followed in the study. Eighteen of these studies (46%) followed a random sampling approach. Often researchers would assign constraints with regard to the minimum and/or maximum (e.g. [64]) temporal distance between two samples, or the maximum samples taken during a day (e.g. [12]). In seven studies (18%) the sampling was scheduled for particular moments, while the remaining 14 (36%) followed an event-based sampling approach, triggered by incoming call handling in the user’s mobile phone [29], changes in the user’s location [25], attempts to connect to a wifi-access point [44] and other user interactions. The frequency of sampling varied considerably across studies, ranging from 10 minutes to a couple of hours, with most studies that spanned throughout the day to limit sampling frequency to less than 10 per day. Previous research has revealed that a sampling frequency of five to eight times per day may yield an optimal balance of recall and annoyance, while sampling at predetermined times is preferred over random as interruptions can be anticipated [45].
Only 11 papers reported users’ response rate to experience samples. Reported response rate was high, ranging from 17% to 93% (median=71%). Some papers reported varying response rate depending on the sampled day with weekdays displaying higher response rate (63%) than days of the weekend (55%) [52], or the type of questions with qualitative ones, requiring free-text responses, displaying lower response rate (73%) as opposed to quantitative such as scales and selection tasks (89%) [2].
0
5
10
15
≤1h < 24h ≤ 3d ≤ 1w ≤ 2w ≤ 1m > 1m
3
No of papers referring to Experience Sampling
Prompts at random, or computationally estimated times, to self-report on ongoing behaviors and experiences.
– Where are you?– What are you doing?– How far is your mobile phone? – How do you feel?
Karapanos, E. (2012) Experience Sampling, Day Reconstruction, what’s next? Towards Technology-Assisted Reconstruction. M-ITI internal report.
Wednesday, June 6, 12
Experience Sampling Method
a few relevant venues would still miss a substantial number of studies. We instead used the ACM Guide to Computing Literature querying for the term “experience sampling” without constraining to particular venues. This query returned 284 papers, published in more than fifty venues.
Figure 2. 243 papers referring to experience sampling over a ten-year period. Retrieval took place on August 26th, 2011.
Fourty-one papers were excluded from further processing. These were: references to proceedings of conferences (n=13), papers that dealt with research problems that were beyond the scope of HCI such as computer networks, transport economics and organizational psychology (n=16), papers where no traces of the term “experience sampling” could be found (n=6), most likely caused by errors in the retrieval process such as due to stemming algorithms, papers that were written in languages other than English (n=3), papers that used the term ‘experience sampling’ to refer to concepts other than the method (n=2), and a single paper that was found to be a duplicate entry.
The remaining 243 papers were further classified with respect to being in one of the following types (Interrater agreement of two coders on a random subset of 60 papers, Fleisch K=0.74): empirical studies using the experience sampling method (n=59), empirical studies that considered but decided against the use of ESM (n=21), tools for carrying out experience sampling studies (n=20), systems that borrow concepts from ESM (n=10), papers discussing particular methodological aspects of ESM such as psychometric scales (n=13), reflection papers discussing conceptual issues of in-situ measurement (n=17), and papers discussing ESM as prior (n=85) or future (n=18) work.
For the scope of this paper we will analyze in detail only two categories of papers:
a) 59 papers reporting empirical studies that used the experience sampling method, and
b) 21 papers reporting empirical studies that considered, but decided against the use of the experience sampling method.
Out of the 59 papers that reported empirical studies, eight were found to report a study that was published earlier, and two cited incorrectly the use of experience sampling
0
15
30
45
60
2001 2003 2005 2007 2009 2011
method while following a user-initiated diary approach. The analysis of the remaining 49 studies is being reported below.
Study length, sampling frequency, and response rateThe majority (80%) of the studies had a duration of several days up to one month with 14 studies (34%) lasting between four and seven days (see figure 3). Only two studies had a duration of more than a month.
Figure 3. Distribution of 41 experience sampling studies over their length. Eight papers did not report study length.
Thirty nine papers reported the particular sampling approach that was followed in the study. Eighteen of these studies (46%) followed a random sampling approach. Often researchers would assign constraints with regard to the minimum and/or maximum (e.g. [64]) temporal distance between two samples, or the maximum samples taken during a day (e.g. [12]). In seven studies (18%) the sampling was scheduled for particular moments, while the remaining 14 (36%) followed an event-based sampling approach, triggered by incoming call handling in the user’s mobile phone [29], changes in the user’s location [25], attempts to connect to a wifi-access point [44] and other user interactions. The frequency of sampling varied considerably across studies, ranging from 10 minutes to a couple of hours, with most studies that spanned throughout the day to limit sampling frequency to less than 10 per day. Previous research has revealed that a sampling frequency of five to eight times per day may yield an optimal balance of recall and annoyance, while sampling at predetermined times is preferred over random as interruptions can be anticipated [45].
Only 11 papers reported users’ response rate to experience samples. Reported response rate was high, ranging from 17% to 93% (median=71%). Some papers reported varying response rate depending on the sampled day with weekdays displaying higher response rate (63%) than days of the weekend (55%) [52], or the type of questions with qualitative ones, requiring free-text responses, displaying lower response rate (73%) as opposed to quantitative such as scales and selection tasks (89%) [2].
0
5
10
15
≤1h < 24h ≤ 3d ≤ 1w ≤ 2w ≤ 1m > 1m
3
No of papers referring to Experience Sampling
What variables do ES studies measure?We distinguish below between self-reported measures of behavior and experience (see Table 1). This distinction is relevant as their reconstruction follows a different process whereas behavioral information may be directly accessible through episodic memory while experiential information has to be further inferred from recalled episodic cues [63].
Table 1. Number of ES studies eliciting self-reported measures of behavior, experience, or both.
Type of measures that studies elicit No
Self-reported measures of behavior 5
Self-reported measures of experience 22
Self-reported measures of behavior & experience 18
Behavioral measures related most frequently to the activity (n=18) that the participant was engaged with prior to being interrupted (e.g. [30]), its duration (n=2) (e.g. [35]), the participant’s current physical location (n=15) (e.g. [17]) and the social context (n=10), e.g. the number or nature of relationship of people that are in close proximity or participate with in a conversation (e.g. [34]). Other measures of behavior related to mode of transit [25], participants’ current physical engagement [18] and mode of convrersation (e.g. f2f, fixed/mobile phone etc.) [32].
Experiential measures related to:• Attitudes towards behaviors or events (n=15) such as
being interrupted (e.g. [55]), disclosing information to relevant others (e.g. [17]), or being video recorded (e.g. [58]).
• Measures of affect and experience (n=18) such as mental engagement (e.g. [18]) and concentration [14], satisfaction [27], mood and emotional states (e.g. [53], [23], experienced stress [61] and fun [14], interpersonal connectedness [19] and telepresence [14].
• Cognitive judgments (n=10) relating to stimuli, such as judging the relevance or usefulness of an advert [64], ranking the features of the product under use [2], making credibility assessment of provided information [62], forming judgments of the outcome of an undertaken activity [49], or one’s self-efficacy on the sampled activity [4].
• Motivations for exhibited behaviors (n=9) such as charging one’s mobile phone [6], handling incoming calls [28], posting information on social networking sites [50], tracking the location of family members [49].
What are the most frequent reasons for not selecting ESM?Twenty-one papers reported empirical studies that considered, but decided against the use of the experience
sampling method. Two of these provided no justifications for their choice. Analyzing the remaining 19 papers resulted to a total of 11 reasons for choosing alternative methods to ESM (see table 2).
Table 2. Reasons for not selecting the Experience Sampling Method along with frequency of occurence (No of papers).
Reason No
Disrupts the activity 6
Imposes high burden to participants 3
Requires high effort from researchers 3
Inappropriate for eliciting rich qualitative data 3
Misses rare and brief events 3
The user should be in control of when, what and how often to report
2
Limits sample size 2
Depends on participants’ ability to articulate ongoing experience
2
Poses privacy concerns 2
Limits number of measured variables 1
Technology limitations 1
As expected, the most frequent reason for not selecting the ES method was the interruptions that the method imposes on the user’s activity (eg. [48], [5]). For instance, Lindley and Monk [48] attempted to inquire into the social experiences of storytelling and reminiscing while revisiting photos with friends. While ESM would enable the momentary assessment of these experiences, it would have a strong impact on group behavior. Instead, they argued that behavioral measures can be used to infer aspects of these social experiences.
Other reasons for selecting alternative methods were the high burden that ESM imposes on participants, especially when typing in mobile situations is required (e.g. [47]), but also the high effort required to run the study (e.g. [31]). Three papers reported a need for eliciting rich qualitative accounts as a reason for choosing an alternative method such as Day Reconstruction diaries combined with self-reported experience narratives [41] and “lightweight capture of memory-trigger media in-the-moment, followed by in-depth annotation and review from a PC” [42].
4
Prompts at random, or computationally estimated times, to self-report on ongoing behaviors and experiences.
– Where are you?– What are you doing?– How far is your mobile phone? – How do you feel?
Karapanos, E. (2012) Experience Sampling, Day Reconstruction, what’s next? Towards Technology-Assisted Reconstruction. M-ITI internal report.
Wednesday, June 6, 12
Day Reconstruction Method
produce the exact past event, but instead, every attempt to recall results in a new, often altered representation of the event. Bartlett asked participants to recall an unfamiliar story that they were told 20 hours before. Recalled stories differed from the original one in missing details, altering the order and importance of events, or in applying rationalizations and interpretations to the original story. Stories were further distorted through repeated reconstruction.The notion that remembering is an act of reconstruction instead of mere reproduction has received wide support. At the heart of reconstruction lies the distinction between episodic and semantic memory [69]. While episodic memory “is specific to a particular event from the past, semantic memory is not tied to any particular event but rather consists of certain generalizations (i.e. beliefs) that are rarely updated” (p. 935). These two types of memory serve different needs such as learning new information quickly - a capacity of episodic memory - or developing relatively stable expectations about the world - a capacity of semantic memory [63]. Reconstruction happens through the retrieval of cues from episodic memory. In the absence of contextual cues in episodic memory, beliefs found in semantic memory may be used to reconstruct the past, resulting in distortions such as the ones found in Barlett’s study. Thus, overall, the accuracy of one’s remembered events lies in the degree to which contextual cues are still present in the person’s episodic memory.
But, how do we reconstruct emotional experiences that contain not only contextual details of the experienced event, but also value-charged information such as emotions or overall evaluative judgments on the event?
Robinson and Clore [63] argue that “emotional experience can neither be stored nor retrieved” (p. 935), but can only be reconstructed on the basis of recalled contextual cues. They propose an accessibility model that distinguishes between four types of knowledge used to construct an emotion (see figure 1). First, experiential knowledge is used when an emotion is constructed online, i.e. as the experience takes place. It is considered the only “authentic” source of information, yet it is accessible for only a small fraction of time, about 3 seconds [37]. Self-reports tapping into experiential information, such as ESM, are considered the “gold standard” of experience measurement [36] as they avoid biases caused by retrospection.
When experiential knowledge is inaccessible, people will resort to episodic information, i.e. they will recall contextual cues from episodic memory and infer their emotions on the basis of these contextual cues. For instance, in recalling one’s emotions while experiencing a roller coaster, one might recall having his hands raised and screaming, and on this basis to infer an aroused experience. The accuracy with which people may recall their emotions will thus depend on the richness of recalled episodic
information. Daniel Kahneman and colleagues [36] relied on this proposition to develop the day reconstruction method. DRM asks participants to list their daily activities as a continuous list of episodes [36]. This is assumed to form stronger temporal and semantic links across the distinct experiences, thereby minimizing bias from overly relying on semantic information, which is detached from the actual experience. It has been shown that DRM provides a reasonably good approximation to experience sampling data, both in between-subjects [36] and within-subject [20] analyses. When episodic memories become inaccessible, people will shift to semantic memory. People will first access situation-specific beliefs, i.e. “a belief about the emotions that are likely to be elicited in a particular type of situation” [63]. If event-specific beliefs are inaccessible, e.g. due to rarity of the event, people will access identity-related beliefs, i.e. “beliefs about their emotions in general”. Such information is detached from the actual experience and is associated with retrospection biases found in global reports of experience such as online surveys and questionnaires used at the end of empirical studies [65].
A REVIEW OF EXPERIENCE SAMPLING AND DAY RECONSTRUCTION STUDIESThe experience sampling method has been popular in HCI for over a decade. Yet, to our knowledge, no attempt has been made to review the use of the method by HCI researchers. In the following section we attempt a review of empirical studies, published within HCI, that employ either the experience sampling or the day reconstruction method. The questions we will attempt to adress are: How are these methods used within HCI? What are the benefits these methods offer in addressing particular HCI problems? And, what drawbacks are experienced in practice?
Experience samplingAs ES studies have been published across a wide range of HCI venues we decided that narrowing our search down to
2
Figure 1. Four sources of information in emotional self-report according to Robinson and Clore [63]. Figure adapted from [63].
Experiential Knowledge
Online emotion, e.g.Experience Sampling
Episodic memory
Situation-specific belief
Identity-related belief
Retrospective, e.g.Day Reconstruction
Exit questionnaires
Type of Knowledge Source of information Type of Self-Report
Exit questionnaires
Semantic
Semantic
Episodic
Episodic
Can a retrospective method help participants in recalling more accurately their experiences?
Kahneman et al. (2004)
Robinson & Clore (2002)
Wednesday, June 6, 12
Day Reconstruction Method
Can a retrospective method help participants in recalling more accurately their experiences?
Kahneman et al. (2004)
Wednesday, June 6, 12
Technology Assisted ReconstrucIon
Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts?
Trajectory reminders EmoSnaps Footprint tracker iScale
Wednesday, June 6, 12
! !
Control condition With trajectory reminders
Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process?
Wednesday, June 6, 12
Can self-face pictures assist in recalling momentary emotions?If so, is it through a recognition or a reconstruction process?
Emosnaps - inferring emotion from self-face pics
Wednesday, June 6, 12
Can self-face pictures assist in recalling momentary emotions?If so, is it through a recognition or a reconstruction process?
Emosnaps - inferring emotion from self-face pics
Wednesday, June 6, 12
Time-Daycontrol condition
Photo-dayEmotion reconstruction
Photo-weekEmotion recognition
Experience Sampling (Ground truth)
78% of pictures could be used for inferring emotions
Can self-face pictures assist in recalling momentary emotions?If so, is it through a recognition or a reconstruction process?
Emosnaps - inferring emotion from self-face pics
Wednesday, June 6, 12
Time-Daycontrol condition
Photo-dayEmotion reconstruction
Photo-weekEmotion recognition
Experience Sampling (Ground truth)
78% of pictures could be used for inferring emotions
Can self-face pictures assist in recalling momentary emotions?If so, is it through a recognition or a reconstruction process?
Emosnaps - inferring emotion from self-face pics
Wednesday, June 6, 12
1. Sensecam
2. Location logging3. Context logging
(SMS/calls made or received)
Footprint trackerHow do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?
Wednesday, June 6, 12
Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
Wednesday, June 6, 12
“Longitudinal” paradigms in HCI
Cross-‐sec8onal Repeated sampling
Longitudinal Retrospec8ve
Wednesday, June 6, 12
Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
Wednesday, June 6, 12
Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
Wednesday, June 6, 12
33
Constructive Value-Account Control (no-graphing)
Constructive iScale, but not the Value-Account, performed better than control condition
•More experience reports•With more details (references to temporal information, discrete events)•Higher test-retest consistency of time estimation (i.e., when did an experience take place)
•Higher test-retest consistency of graphed patterns (over Value-Account)
Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
Wednesday, June 6, 12
1
Modeling Users' Experiences w
ith Interactive Systems
436
Over the past decade the field of Human-Computer Interaction has evolved from the study of the usability of interactive products towards a more holistic understanding of how they may mediate desired human experiences.
This book identifies the notion of diversity in users? experiences with interactive products and proposes methods and tools for modeling this along two levels:(a) interpersonal diversity in users? responses to early conceptual designs, and(b) the dynamics of users? experiences over time.
The Repertory Grid Technique is proposed as an alternative to standardized psychometric scales for modeling interpersonal diversity in users? responses to early concepts in the design process, and new Multi-Dimensional Scaling procedures are introduced for modeling such complex quantitative data.
iScale, a tool for the retrospective assessment of users? experiences over time is proposed as an alternative to longitudinal field studies, and a semi-automated technique for the analysis of the elicited experience narratives is introduced. Through these two methodological contributions, this book argues against averaging in the subjective evaluation of interactive products. It proposes the development of interactive tools that can assist designers in moving across multiple levels of abstraction of empirical data, as design-relevant knowledge might be found onall these levels.
Foreword by Jean-Bernard Martens and Closing Note by Marc Hassenzahl.
issn 1860-949X
isbn 978-3-642-30999-1
springer.com
The series Studies in Computational Intelligence (SCI) publishes new developments and advances in the various areas of computational intelligence – quickly and with high quality. The intent is to cover the theory, applications, and design methods of computational intelligence, as embedded in the fields of engineering, computer science, physics and life sciences, as well as the methodologies behind them. The series contains monographs, lecture notes and edited volumes in com putational intelligence spanning the areas of neural networks, connectionist systems, genetic algorithms, evolutionary computation, artificial intelligence, cellular automata,self-organizing systems, soft computing, fuzzy systems, hybrid intelligent, and virtual reality systems. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution, which enable both wide and rapid dissemination of research output.
Studies in Computational Intelligence
1 3
436
Modeling Users'Experiences withInteractive Systems
Evangelos Karapanos
Karapanos
9 7 8 3 6 4 2 3 0 9 9 9 1
Coming!June 2012
Foreword: Jean-Bernard MartensClosing note: Marc Hassenzahl
Wednesday, June 6, 12
Can we really trust 6-month old memories?
• Validity? i.e. do memories reflect what we really experienced?
• Reliability? i.e. in a second trial, will we recall the same experiences?
Wednesday, June 6, 12
Memories are (sometimes) more important than experiences
• Memories define how you evaluate your past and how you decide on your future
• What do we measure for?
• Why do people drive irresponsibly?
• Why do people recommend their products
Wednesday, June 6, 12