Ensuring Validity in StrategicUX Research Methods Dr. Carine Lallemand

University of Luxembourg @carilallUX STRAT Europe 2016

01

WHO AM I?CARINE LALLEMAND

« I DESIGN, ADAPT AND VALIDATE

UX RESEARCH METHODS »

Carine LallemandGuillaume Gronier

Méthodes de design

30 MÉTHODES FONDAMENTALES POUR CONCEVOIR ET ÉVALUER LES SYSTÈMES INTERACTIFS

DESIGN

32 € Conc

eptio

n N

ord

Com

po

DESIGN

Articulant théorie et pratique, cet ouvrage présente 30 fiches méthodologiques couvrant l’essentiel du design UX et de l’ergonomie des interactions homme-machine (IHM). Vous serez guidé pas à pas à travers les étapes de réalisation de chaque méthode et accompagné pour prendre les décisions les plus adaptées à votre projet. Chaque fiche méthode intègre égale-ment une partie théorique et des illustrations concrètes pour faciliter la compréhension.

Véritable portfolio théorique et méthodologique, cet ouvrage est un guide indispensable à toute personne impliquée dans la conception de systèmes interactifs. Professionnels, chefs de projets, étudiants, enseignants et chercheurs y trouveront de précieuses ressources pour mener à bien leurs projets.

Grâce aux méthodes d’UX design, créez des produits et des services qui attirent, qui captivent, qui enchantent et inspirent pour améliorer la vie de ceux qui les utilisent !

AU SOMMAIREIntroduction au design UX ⍟ Planification ⍟ Définition du projet ⍟ Recrute-ment des utilisateurs ⍟ Déontologie et éthique ⍟ Exploration ⍟ Entretien ⍟ Focus group ⍟ Observation ⍟ Questionnaire exploratoire ⍟ Sondes culturelles ⍟ Idéation ⍟ Brainstorming ⍟ Cartes d’idéation ⍟ Design studio ⍟ Experience maps ⍟ Personas ⍟ Techniques génératives ⍟ Génération ⍟ Design persuasif ⍟ Gamification ⍟ Iconographie ⍟ Maquettage ⍟ Storyboarding ⍟ Tri de cartes ⍟ Évaluation ⍟ Complétion de phrases ⍟ Courbes d’éva-luation UX ⍟ Échelles d’utilisabilité ⍟ Échelles UX ⍟ Éva-luation des émotions ⍟ Évaluation experte ⍟ Inspection cognitive ⍟ Journal de bord UX ⍟ Test des 5 secondes ⍟ Tests utilisateurs

Code

G14

143

ISBN

978

-2-2

12-1

4143

-6

« Aucun ouvrage francophone ne rassemble autant

de savoir-faire ! Simple, pratique et pédagogique,

c’est LE guide essentiel de l’UX au quotidien. »

Corinne Leulier, Psychologue - Ergonome,

directrice UX chez Klee Group

« Ergonomie, psychologie, ingénierie, design, sociologie,

ethnographie… Ce livre est une formidable proposition

pragmatique, claire et actualisée des méthodes pour

la conception et l’évaluation de l’expérience utilisateur ! »

Julien Kahn, responsable pôle ergonomie

chez Orange

Chercheur à l’université de Luxembourg, Carine Lallemand est spécialisée dans les méthodes de conception et d’évaluation de l’expérience utilisateur (UX). Impliquée depuis 2010 dans l’association FLUPA, elle est également conférencière et enseigne l’UX design. Guillaume Gronier est chercheur ergo-nome au Luxembourg Institute of Science and Technology. Ses principales activités portent sur l’amélioration de l’expérience utilisateur, l’acceptation technologique et l’implication des utilisateurs dans le processus de conception. Il est l’un des fondateurs de l’association FLUPA.

Méthodes de design UX

UX

Car

ine

Lall

eman

dG

uilla

ume

Gro

nie

rM

étho

des

de d

esig

n UX

Préface d’Alain Robillard-Bastien

FORMER VICE-PRESIDENT

CURRENT COUNCIL MEMBER

RESEARCHER AT THE UNIVERSITY OF LUXEMBOURG

@CARILALL

01

WE NEED TO BASE STRATEGIC DECISIONSON VALID FINDINGS

01

COLLECTING VALID AND RELIABLE DATA?

—> VALID & RELIABLE —> CONTEXTUALIZED & DYNAMIC

Asking SIRI?

Making a good guess?

On-site live wind measurement

Meteorological information provided by official services

How can you safeguard a UX strategy by ensuring the quality of research

conclusions? 

Whenever we measure or observe we should be concerned with whether we are measuring what we intend to measure or with how our observations are

influenced by the circumstances in which they are made

© Adam Cooper (2014)

UNDERSTANDING VALIDITY AND RELIABILITY

Validity: are we measuring what we intend to measure?

Reliability: how stable or constant is our measure?

UX UX UX UX

01WORLD IA DAY 2016 PRESENTATION TITLE HERE

<STRATEGIC THINKING

>Doing the right things = Using the right methods

TACTICAL THINKING

Doing things right = Using the methods right

01WORLD IA DAY 2016

Some kind of illustration or image?

HEADER OPTIONSUB HEAD OR SHORT DESCRIPTION

Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.

USING THE RIGHT METHODS

1

© UX Booth (2016)

« Strangely, while I find the proposition to consider the experience before the thing quite a radical change, many practitioners and

academics of HCI happily embrace experience – however, without changing much in their approach. »

-Prof. Marc Hassenzahl (2013)

The nature and complexity of UX involves a deep change in the methods we use

UX is highly dynamic

The memory of an experience matters more than the experience itself

UX is highly contextual

UX is holistic1

2

3

4

5 UX is about emotions and psychological needs

UX is holistic

Thüring & Mahlke, 2007

A system’s perceived attractiveness is based on the perception of its pragmatic and hedonic qualities

System

User

Context

Interaction characteristics

Perception of non-instrumental qualities

Emotions

Perception of instrumental qualities

Components of User Experience

Consequences

overall evaluation, acceptance, intention to

use, choice of alternatives

1

System

User

Context

Interaction characteristics

Perception of non-instrumental qualities

Emotions

Perception of instrumental qualities

Components of User Experience

Consequences

overall evaluation, acceptance, intention to

use, choice of alternatives

Usability scales (SUS, QUIS, SUMI, WAMMI, etc)

established usability questionnaires focus on pragmatic aspects only… this is not enough!

1UX is holistic

Thüring & Mahlke, 2007

We need to assess both pragmatic and hedonic perceived qualities of a system

AttrakDiff scale (Hassenzahl et al., 2003)

User Experience Questionnaire

(Laugwitz et al., 2008)

meCUE scale (Minge & Riedel, 2013)

1

Using standardized and validated UX scales

The AttrakDiff scale: a standardized UX assessment tool

28 items (word pairs) divided into four subscales: • Pragmatic qualities • Hedonic qualities - stimulation • Hedonic qualities - identification • Attractiveness

• Single evaluation • Comparison product A - product B • Comparison Before - After

An abridged version (10 items) Portfolio of results (comparison A vs. B)

(Hassenzahl et al., 2003)

UX is highly contextual

Context

User System

Social context

Technical context

Temporal context

Task context

Physical context

Time

2

user testing in a controlled

environment

expert evaluation

traditional evaluation methods assess UX in an artificial environment

Context

User System

Time

2UX is highly contextual

How does UX alter laboratory evaluation?

Study conducted in 2015 (Lallemand et al., 2015)

Experiment involving 70 users, who were asked to evaluate their UX with two systems

Research objective: assessing the quality and limitations of « lab testing » for the evaluation of UX.

Results: - validity of our assessment was limited to only the

pragmatic aspects of the interaction - significant order effects - impact of the scenarios of use on the felt experience - impact of the environment and the lack of ecological

validity - lack of evaluation of the dynamics of the experience

We need to evaluate UX in a natural or realistic setting

Field testing and observation

"In-sitro" user testing (Kjeldskov et al., 2004)

Experience sampling (Csikszentmihalyi , 1990)

2

Ecological validity and the « turn to the wild »

Before usage

Anticipated UX

Imagining experience

During usage

Momentary UX

Experiencing

After usage

Episodic UX

Reflecting on an experience

Over time

Cumulative UX

Recollecting multiple periods of use

When:

What:

How:

UX White Paper, 2010

There are several time spans of UX

UX starts before the interaction and doesn’t end immediately after the interaction

UX is highly dynamic3

traditional or psychophysiological evaluation methods focus on momentary UX… this is

not enough!

UX White Paper, 2010

user testing psychophysiological measurements

Before usage

Anticipated UX

Imagining experience

During usage

Momentary UX

Experiencing

After usage

Episodic UX

Reflecting on an experience

Over time

Cumulative UX

Recollecting multiple periods of use

When:

What:

How:

UX is highly dynamic3

The memory of an experience matters more than the experience itself

Episodic UX is a reconstruction, a remembered experience biased by cognitive processes

The momentary experience is not as important as the way it is remembered.

It’s the memory of an experience that influences user’s behavior and the way he talks or recommends the product to someone

4

We need to assess UX across time and to focus on the memory of experiences

UX Curve (Kujala et al., 2011)

Diary methods

Retrospective UX assessment

Analytic scale (Karapanos et al., 2010)

Longitudinal study

3 4

The UX Curve method: retrospective UX evaluation

What they will tell you is biased by their memory, it is not similar to how they really felt

What matters is how they remember the experience with your system because they will behave on this basis.

Unvalid, yet reliable?

Author's personal copy

performed simply on the basis of whether the starting point of thecurve was higher or lower compared to the end point. For example,the curve in Fig. 2 was categorized as being improving as its start-ing point was lower than its ending point, even though the curvedeteriorates in the middle. If the starting and ending points wereat the same level, the curve was categorized as stable. As the curves

were freehand drawings, they were categorized as stable if therewas a very small deviation (less than one millimeter) betweenthe vertical values of the starting and ending points of the curve.However, it can be seen from Figs. 3–10 that the categorizationwas rather straight-forward to do with the three trend type catego-ries. The relationships between the curve types and the key

Fig. 4. The deteriorating and stable general UX Curves with user IDs.

Fig. 5. The improving Attractiveness curves with user IDs.

Fig. 6. The deteriorating and stable Attractiveness curves with user IDs.

Fig. 7. The improving ease of use curves with user IDs.

Fig. 8. The deteriorating and stable ease of use curves with user IDs.

Fig. 9. The improving utility curves with user IDs.

478 S. Kujala et al. / Interacting with Computers 23 (2011) 473–483

Author's personal copy

performed simply on the basis of whether the starting point of thecurve was higher or lower compared to the end point. For example,the curve in Fig. 2 was categorized as being improving as its start-ing point was lower than its ending point, even though the curvedeteriorates in the middle. If the starting and ending points wereat the same level, the curve was categorized as stable. As the curves

were freehand drawings, they were categorized as stable if therewas a very small deviation (less than one millimeter) betweenthe vertical values of the starting and ending points of the curve.However, it can be seen from Figs. 3–10 that the categorizationwas rather straight-forward to do with the three trend type catego-ries. The relationships between the curve types and the key

Fig. 4. The deteriorating and stable general UX Curves with user IDs.

Fig. 5. The improving Attractiveness curves with user IDs.

Fig. 6. The deteriorating and stable Attractiveness curves with user IDs.

Fig. 7. The improving ease of use curves with user IDs.

Fig. 8. The deteriorating and stable ease of use curves with user IDs.

Fig. 9. The improving utility curves with user IDs.

478 S. Kujala et al. / Interacting with Computers 23 (2011) 473–483

Results: Mean attractiveness curves8

3.6.2011

Facebook Mobile phoneLong-term UX curves (for a specific UX dimension)

Kujala et al., 2011

UX is about emotions and psychological needs

Thinking about the experience first

Designing for emotions and psychological needs

Using science-based design tools

5

Using science-based (yet pragmatic) design tools

5

UX theories as powerful triggers of innovative design ideas —> Defining your Unique Value Proposition

PLEX Cards playful experiences

(Lucéro & Arrasvuori, 2010)

Positive Emotional Granularity Cards

(Yoon, Desmet, & Pohlmeyer, 2013)

UX Cards psychological needs

(Lallemand et al., 2015)

PLEX Cards playful experiences

(Lucéro & Arrasvuori, 2010)

Positive Emotional Granularity Cards

(Yoon, Desmet, & Pohlmeyer, 2013)

UX Cards psychological needs

(Lallemand et al., 2015)

Download them now!

http://uxmind.eu/portfolio/ux-design-and-evaluation-cards

www.funkydesignspaces.com/plex/

www.diopd.org/emotioncards

Established evaluation methods only explore a limited part of UX

single user testing sessions

psychophysiological measurements

expert evaluationusability scales

As we gain a deeper understanding of UX, we have to adapt the methods we use to ensure validity

01WORLD IA DAY 2016

Some kind of illustration or image?

HEADER OPTIONSUB HEAD OR SHORT DESCRIPTION

Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.

USING THE METHODS RIGHT

2

UX RESEARCH METHODS ARE « Ingredients and Meals Rather Than Recipes »

…just as the quality of what is cooked reflects the quality of its ingredients, so does the quality of UX work reflect the quality of

resources as configured and combined. Woolrych et al., 2011

01

TARGETING REPRESENTATIVE USERS

THE IMPORTANCE OF SAMPLING

01

SAMPLING: TARGETING THE RIGHT USERS

Making the most out of opportunistic sampling?Probability sampling: process that gives all the individuals in the population equal chances of being selected

Opportunistic sampling: the availability of participants guides on-the-spot sampling decisions

Sample size vary in different research settings. All else being equal, large sized sample leads to increased precision in estimates of various properties of the population.

01

WHAT ABOUT GUERRILLA RESEARCH?

A reasonable option?Fast and cheap way to get a certain type of feedback

• Only for consumer-oriented product

• Testing the understanding of the Value Proposition or the usability of one specific feature

Not always ‘better than no research’

01

IDENTIFYING CONFOUNDING VARIABLES

AVOIDING RESEARCH BIASES

01

RESEARCH BIASES A few examples

Selection bias: one relevant group in the population has a higher probability of being included in the sample. —> Choosing a random or representative sample

Experimenter / interviewer bias: differential treatment of participants—> Standardized procedures and instructions.

Expectancy / observer bias: the researcher’s expectations affect the outcome of a study—> Having independent observers and computing inter-raters agreement

Social desirability bias: the tendency of respondents to answer questions in a manner that will be viewed favorably by others—> Careful formulation of questions and items. Use of projective techniques.

01

VALIDITY THROUGH DATA TRIANGULATION

USING MIXED-METHOD

01

COMBINING QUANTITATIVE & QUALITATIVE

Understanding « how » and « why »

Quantitative research methods: rely on using large sample sizes to establish trends and conclusions.

Qualitative research: appropriate for getting a more in-depth, contextual understanding of why those trends occur.

The best research strategies incorporate both approaches

Profile Unfinished sentence UX dimension

non ebook-reader

Compared with a paper book, a digital book is… Comparison between products

In my opinion, digital books are addressed to… Identity / product image

I have never read digital books because… Frustrations / Barriers to use

I would read a digital book if… Expectations and needs

I expect a digital book to / that… Expectations and needs

When I read a paper book, I feel… Affects

ebook reader

Compared with a paper book, a digital book is… Comparison between products

The reading experience on a digital book is… Global UX

The problem with ebooks is… Issues and frustrations

What I love about ebooks is… Positive aspects / Appropriation factors

What frustrates me the most with a digital book is… Issues and frustrations

I find that the interface of a digital book is… Specific UX - Interface

I dream of a digital book that… Expectations / Dreams

Ongoing study (Lallemand & Mercier, 2015)Designing an optimal e-reading experience

01

LIKERT SCALE VS. SENTENCE COMPLETION

On a 7-points Likert scale, how would you rate your overall e-reading experience? (N = 1284)

Self-reported overall e-reading experience(7 points Likert scale)

Valence Frequency PercentNegative 228 17,8 %

Positive 817 63,9 %

Neutral 160 12,5 %

Mixted 74 5,8 %

« The reading experience on a digital book is… »

Valence analysis of sentence L_SC_2

Ongoing study (Lallemand & Mercier, 2015)

Designing an optimal e-reading experience

01

LIKERT SCALE VS. SENTENCE COMPLETION

The problem with ebooks is…- the price- the lack of availability and choice- the absence of a sensual experience (feeling the

paper in one’s hands)- the navigation and information architecture- the battery / the need for a network connexion- their bad quality- the impossibility to lend the book to a friend- DRM (digital rights management)- the bad reading experience- the screen and visual fatigue- it is dematerialized- …

that you don’t see what people are reading because you don’t see the book cover…

you can't skim or flip through easily

I’m not able to physically track my progress in the book

Designing an optimal e-reading experienceOngoing study (Lallemand & Mercier, 2015)

01

VALIDITY & RIGOR, YET SCALED TO THE NEED

Basing strategic decisions on valid findings

Rigor should be proportional to the risk

Catching up on emerging research and using freely available valid yet lightweight UX methods developed in Academia

Rethinking your Unique Value Proposition thanks to UX theories

THANK YOU FOR YOUR ATTENTION

Dr. Carine Lallemand Twitter @carilall http://uxmind.eu

01

REFERENCES• Csikszentmihalyi, M. (1990). Flow. The psychology of optimal experience, Harper and Row. • Hassenzahl, Marc (2013): User Experience and Experience Design. In: Soegaard, Mads and Dam, Rikke Friis (eds.). "The Encyclopedia of

Human-Computer Interaction, 2nd Ed.". Aarhus, Denmark: The Interaction Design Foundation. • Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff : Ein Fragebogen zur Mes- sung wahrgenommener hedonischer und pragmatischer

Qualität. In J. Ziegler & G. Szwillus (Eds.) Mensch & Computer 2003. Interaktion in Bewegung, 187–196. Stuttgart: B.G. Teubner. • Kahneman, D., et al., (2004). A survey method for characterizing daily life experience: The Day Reconstruction Method, Science, CCCVI(5), 702. • Karapanos, E., Martens, J.-B., & Hassenzahl, M. (2010). On the Retrospective Assessment of Users’ Experiences Over Time : Memory or Actuality

? Proc. of CHI 2010, 2689-2698. • Kjeldskov, J., & Skov, M.B. (2007). Studying Usability In Sitro : Simulating Real World Phenomena in Controlled Environments. International Journal

of Human-Computer Interac- tion, 22(1-2), 7–36. • Kujala,S., Roto,V., Väänänen-Vainio-Mattila,K., Karapanos,E., &Sinnelä,A. (2011). UX Curve: A method for evaluating long-term user experience.

Interacting with Computers, 23, 473-483. • Lallemand, C. (2015). Towards Consolidated Methods for the Design and Evaluation of User Experience. (Doctoral dissertation). University of

Luxembourg. https://publications.uni.lu/handle/10993/21463 • Laugwitz, B, Held, T., & Schrepp, M. (2008). Construction and evaluation of a user expe- rience questionnaire. In A. Holzinger (Ed.) USAB 2008,

LNCS 5298. Berlin: Springer Verlag. • Lucero, A., & Arrasvuori. J. (2010) PLEX Cards : a source of inspiration when designing for playfulness. Proc. of Fun and Games 2010. New York,

USA: ACM, 28-37.

01

REFERENCES• Minge, M., & Riedel, L. (2013). meCUE – Ein modularer Fragebogen zur Erfassung des Nutzungserlebens. Presented at Mensch und Computer

2013, Bremen. • Roto, V., Law, E., Vermeeren, A., & Hoonhout, J. (2011) User Experience White Paper: Bringing clarity to the concept of user experience. Result

from Dagstuhl Seminar on Demar- cating User Experience, Finland. • Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4),

253-264. • Yoon, J., Desmet, P. M. A., & Pohlmeyer, A. E. (2013). Embodied Typology of Positive Emotions: The Development of a Tool to Facilitate Emotional

Granularity in Design (pp. 1195–1206). Presented at the 5th International Congress of International Association of Sciences of Design Research, Tokyo, Japan.

• Woolrych, A., Hornbæk, K., Frøkjær, E. & Cockton, G. (2011). “Ingredients and meals rather than recipes : a proposal for research that does not treat usability evaluation methods as indivisible wholes”. International Journal of Human-Computer Interaction, 27(10), 940-970

• Adam Cooper, Cetis Blog 2014 - http://blogs.cetis.org.uk/ • http://www.uxbooth.com/articles/complete-beginners-guide-to-design-research/

Download the UX Cards : http://uxmind.eu/portfolio/ux-design-and-evaluation-cards Download the PLEX Cards : http://www.funkydesignspaces.com/plex/Download the Positive Emotional Granularity Cards : www.diopd.org/emotioncards