Evaluating Usability and User Experience

in Non-Traditional Environments

IHC 2008 Porto Alegre, Brasil

21 to 24 October, 2008

Regina Bernhaupt

Think about it ….

*How to evaluate the usability of this presentation …

* And how to evalute the User Experience during this

presentation …

*Stressed

*Not being well

*Critical Situations

*Relax

* Fun

*Operational Situations

Non-Traditional Environments

Evaluating Usability & UX in Non-Traditional Environments

Overview

*Usability and User Experience as Key Concepts

*Current Evaluation Approaches

* Three Case Studies: Adoption and Development of

Evaluation Methods

*How to evaluate in non-traditional environments

Usability and UX as Key Concepts

Definition of Usability

*Usability is concerned with making

systems easy to learn and use.

* A usable system is one which

enables users to perform their job

effectively and efficiently.

*Usability is the „effectiveness, efficiency

and satisfaction with which a specified

set of users can achieve a specified

set of tasks in a particular environment“.

ISO 9241-11

Usability Decomposed

* Flexibility: Dialog Initiative, Multi-threading,

Task migratability, Substitutivity, Customizability, …

* Learnability: Predictability, Synthesizability, Familiarity,

Generalizability, Consistency, ….

*Robustness: Observability, Recoverability,

Responsiveness, Task Conformance, …..

Fun as Factor of Usability (Caroll and Thomas, 1988)

„We realize that many people will readreadreadread this article as a

joke. To this extend, we are the victims of our own

analysis: there are risks in being serious about fun.

Still though, we continue to see, without humor, the

prospect of a decade of research analysis possibly failing

to provide the leverage it could on designing systems

people will really want to use by ignoring what could be a

very potent determinant of subjective judgements of

usability – FUN“ (p. 23)

Is User Experience different from Usability?

For Playing the Game:press any keyCongratulations – you won!!

Defining UX

*Experiences, Feelings and Needs of the user before,

during and after interacting with the product in various

contexts.

*How do we evaluate?

Current Evaluation Approaches

Evaluation: Process Model

Limitations of Evaluation Methods

*Real usage and user experiences differ from lab usage

* Field studies: adoption of methods needed

*UX is difficult to evalute: lots of factors

* UX: when do we evaluate what and how?

Categories of Evaluation: Usability

HeuristicEvaluation

DesignRationale

Model-basedEvaluation

User Testing

Experiments

User Studies

Adoption and Development of New Evaluation Methods

Case Study 1: Satellite Control Room

*Usage of new interaction technique

· Multimodal Interfaces

· Multi-touch including voice

· Usability Problems lead to enormous damage/loss

Case Study 1: Satellite Control Room

*Problems:

· You are not allowed to access the users

· You can not test in real situations

· Usability Problems lead to considerable

damage/loss

*Solution:

· Full modeling of the interaction

· Combine formal modeling state description of

the model to investigate usability problems (*)

· (*) Bernhaupt, Palanque, Winckler, Navarre (2007); Bernhaupt, Palanque (in preparation)

Adapting Usability Evaluation Methods

*Combination of Methods:

· Usability Studies with model-based evaluation (*)

* Adapting Methods:

· Heuristic Evaluation with usage scenarios (videos) (**)

· Cognitive Walkthrough with usage scenarios (*)

· Performing Heuristic Evaluation in the field (**)

*Bringing the „Field“ into the „Lab“

· Sports equipment to simulate running (***)

* In general: Combination and embedding of methods

* (*) Bernhaupt et al. (2007) (**) Po, Howard, Vetere, Skov (2004) (***) Kjeldskov and Stage (2003)

Adopting Usability Evaluation Methods

for Non-Traditional Environments

HeuristicEvaluation

DesignRationale

Model-basedEvaluation

User Testing

Experiments

User Studies

User Experience Evaluation

Categories of Evaluation: User Experience

Focus GroupsQuestionnaire

Probing

Experiments User Studies

Interviews

UX Sampling

Play Testing

Case Study 2: Playful Probing

Cultural Probing and Variations

*Probing Approach (Gaver et al., 1999)

*Several Variations: Mobile Probes, Technology Probes, …

* Insights into daily life e.g. to understand user experiencefactors

Playful Probing

*Problem:

· Privacy

· Real usage situations: long-term usage

· Adressing all users during study

*Solution (*) :

· Make Probing „Playful“

· Develop Games to extend the methods

· Use creative/playful material/questions

* (*) Bernhaupt, Weiss, Obrist, Tscheligi (2007),

Playful Probing: Material

*Games, especially developed to address the research

questions, to increase participation

Playful Probing: Material

Playful Probing: Results (*)

* more involvement

* more insights

* more „material“

* helps investigating UX

(*) e.g. Bernhaupt et al. 2007

·

Case Study 3: Evaluating UX in Games

*Developing/Using Games to evaluate aspects of UX

*Effects of Emotion

*Emotional Flower can be played by a group of people

*We use an ambient display to enable the players to see

the progress of others

* All flowers including the chosen nickname are displayed,

the screen is in a public area

Social and Ambient Factors related to UX

*Need for attentive user interfaces

*Emotions are a major

influence on peoples attitude

*Emotional Interfaces are

dealing with human emotions

*Goal: make UX positive

Investigation aspects of UX: Manipulating Emotions

Results (*)

*Emotion can be influenced (but not on a longer term)

*Social influences are important for UX in games

*UX depends on usage situations, time is an important

factor

*UX design must take into account different characters of

users (e.g. more/less play-oriented users)

Categories of Evaluation: User Experience

Focus GroupsQuestionnaire

Playful Probing

Experiments User Studies

InterviewsEmotionalFlowers

How to Evaluate in Non-Traditional Environments

*Choose the right set of evaluation methods based on the

context the final product will be used in

*Evaluation should take place in real usage context with

real users – but this is not realistic for several domains

(security, airtraffic, cockpits, satellite control rooms,

homes, cars, public places, …)

*Modify and develop methods to allow the evaluation in

non-traditional environments or adapt methods to reflect

the real usage environment

Evaluation in Non-Traditional Environments

*Real usage context vs. Lab/Field Study

* Investigate the real usage problems and the user

experiences

* Add valuable information for the development of products

that are usable and that are perceived with a positive user

experience

Problems for Evaluation in

Non-Traditional Environments

*Users not accessible

*Non-standardized technologies (e.g. Set-top box in living

room)

*Evaluation is typcially expensive

*Sometimes not reproducable insights – especially for UX.

Solutions - Examples

* Adaption of Methods:

· Playful Probing

· Creative cultural probing

· User Experience Sampling

· Field usability studies

*Development of new methods and learning from „related“

disciplines:

· Evaluating User Experience in Games [Bernhaupt, R., Springer

Verlag, Mid 2009]

Future Work

* Identification of factors influencing the evaluation

in non-traditional environments

*Development of new or adapted methodologies and their

validation

* Framework for comparing evaluation methods with

respect to usage in non-traditional environments

Contacts

*Regina Bernhaupt

* IRIT ruwido

*Groupe IHCS

*118, Route de Narbonne Köstendorferstr. 8

*31062 Toulouse 5321 Neumarkt a. W.

*Regina.Bernhaupt@irit.fr

*Regina.Bernhaupt@ruwido.com

Literature

* IF you have DIFFICULTIES in finding any of the

publications I authored, please send me an e-mail to

request the article!

*Regina.Bernhaupt@sbg.ac.at

Literature: Playful Probing and Playful Evaluation Methods

(Bernhaupt et al.)

* PLAYFUL PROBING

* Bernhaupt, R., Weiss, A., Obrist, M., Tscheligi, M.(2007). Playful Probing: Making Probing more Fun, INTERACT 2007, 10 -14 Sep 2007, Springer LNCS, 606-619.

* Beck, E., Obrist, M., Bernhaupt, R., Tscheligi, M., (2008). Instant Card Technique: How and Why to apply in User-Centered Design. Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear.

* Weiss, A., Wurhofer. D., Bernhaupt, R., Beck, E., and Tscheligi, M. (2008) “This is a Flying Shopping Trolley”- A Case Study of Participatory Design with Children in a Shopping Context, Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear.

* STUDIES USING PLAYFUL PROBING

* Bernhaupt, R., Obrist. M., Weiss, A., Beck, E. & Tscheligi, M. (2008). Trends in the Living Room and Beyond. Computers in Entertainment, Special Issue, Vol. 6, 1, online.

* Obrist, M., Bernhaupt, R. and Tscheligi, M. (2008). Interactive TV for the Home: An ethnographic study on users’ requirements and experiences. International Journal of Human-Computer Interaction, Volume 24, Issue 2, 174-196.

* Bernhaupt, R., Wilfinger, D., Weiss, A., and Tscheligi, M. (2008) An Ethnographic Study on Recommendations in the Living Room: Implications for Design of iTV Recommender Systems, Proc. of EuroiTV 2008, Springer LNCS 5066, 92-101.

Literature: Evaluation Methods – Introduction/Overviews

* Bernhaupt, R., Mihalic, K. and Obrist M. (2008). Methods for Usability

Evaluation of Mobile Applications. to appear in: Handbook of

Research on User Interface Design and Evaluation for Mobile

Technology, Lumsden, J. et al. (eds.), IGI Global, p. 742 - 755.

* Bernhaupt, R, Palanque, P., Winckler, M. and Navarre, D. ( 2007)

Usability Study of Multi-Modal Interfaces using Eye-Tracking, Interact

2007, 10 – 14 Sep 2007, Springer LNCS, p 412-424.

* Bernhaupt, R., Palanque, P., Winkler, M. and Navarre, D. (2007).

Supporting Usability Evaluation of Multimodal Safety Critical

Interactive Applications using Dialogue and Interaction Models. In

book: Maturing Usability: Quality in Software, Interaction and Value.

Law, E., et al. (eds.) Springer 95-127.

Literature: User Experience

* Hassenzahl, M. and Tractinsky, N. (2006) User Experience – a

research agenda, Behaviour and Infomration Technology, Vol. 25, No.

2, March-April 2006, 91-97.

* Blythe, M., Hassenzahl, M. and Wright, P. C. (2004), More Funology,

Interactions, 11, 36-77.

* Carroll, J.M. and Thomas, J. C. (1988) Fun. SIGCHI Bulletin, 19. 21-24.

* Norman, D. (2004) Emotional Design: Why we love (or hate) everyday

things (New York: Basic Books).

* Blythe, M., Overbeeke, C., Monk, A.f., and Wright, P. C. (Eds), (2003)

Funology: From Usability to Enjoyment (Dordrecht: Kluwer).

* Tractinsky, N., Katz, A.S. and Ikar, D. (2000). What is beautiful is

usable. Interacting with Computers, 13, 127-145.

Literature: User Experience - Emotion/Affect

* Desmet, P.M.A., Overbeeke, C.j. and Tax, S.J.E.T (2001) DesigningProducts with added emotional value: development and application of an approach for research through design. The Design Journal, 4. 32-47.

* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., Tscheligi, M. (2007). Emotional Flower: How Social Community Building is Influenced by an Emotional Interface Game; to AoiR 2007, online version at: http://conferences.aoir.org/papers.php?first_letter=B&cf=6.

* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., and Tscheligi, M. 2007. Using emotion in games: emotional flowers. In Proceedings of the international Conference on Advances in Computer Entertainment Technology (Salzburg, Austria, June 13 - 15, 2007). ACE '07, vol. 203. ACM Press, New York, NY, 41-48.

Literature: Evaluating Games

*R. Bernhaupt (to appear) Evaluating User Experience in

Games: Methods and Approaches, Springer Verlag, HCI-

Series.

*Bernhaupt, R., Ijsselsteijn, W., Mueller, F., Tscheligi, M.

and Wixon, D. (2008). Evaluating User Experiences in

Games, In CHI '08 Extended Abstracts on Human Factors

in Computing Systems (Florence, Italy, April 05 - 10,

2008). CHI '08. ACM, New York, NY, 3905-3908.

*Position Papers at:

http://workshops.icts.sbg.ac.at/chi2008uxgames/

Literature: Evaluation in Non-Traditional Environments

* Bernhaupt, R. (accepted) Application of Usability and User Experience

Evaluation Methods for the Mass-Customization of Products and

Services. (Eds).Mourlas, C., & Germanakos, P. Mass Customization for

Personalized Communication Environments: Integrating Human

Factors, IGI Global Publication.

* Bernhaupt, R., Obrist, M., Tscheligi, M. (accepted) Methods for

involving users in the development of social interactive TV: Enhancing

usability and user experience in non-traditional environments, 15.

pages, Ed. Caesar, P., Geerts, D., and Chorianopoulous, K. Social

Interactive Television: Immersive Shared Experiences and

Perspectives. IGI Global Publication.