1. A brief overview of UX research and evaluation methods Dr. Panagiotis Zaharias 13/5/2014 @ CoLab Athens

2. The UX Mantra To measure is to know. If you cant measure it, you cant improve it 3. Overview of the talk Examples of UX evaluation studies Current trends in UX evaluation UX Research: Main Concepts 4. Main Concepts in UX Research First things first 5. The Ancestor: Human Computer Interaction Human Computer Interaction, is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and the study of major phenomena surrounding them. 6. User friendly? Usable? Once upon a time we were talking about user friendly systems so general and vague Then usability emerged and was originally derived from the term user friendly but usability is still a general and vague term Many definitions and different approaches to how usability should be measured 7. What is usability anyway? The ISO definition (9241-11) Usability is defined as: the extent to which a product can be used by specified users to achieve specified goals with effectiveness; the extent to which the intended goals of use are achieved, efficiency; the resources that have to be expended to achieve the intended goals and satisfaction; the extent to which the user finds the use of the product acceptable, in a specified context of use. 8. Examples of usability measures (1) Measures of effectiveness: Task completion (number of correct tasks, number of tasks where users failed to finish within a set time, number of tasks where users gave up) Accuracy measures (quantify the number of errors users make during the process of completing tasks) Recall measures (how much information users can recall after the use of an interface) Quality of outcome (attempts to measure the outcome of the tasks, measures of understanding e.g. tests of what has been learned from an e-learning system) 9. Examples of usability measures (2) Measures of efficiency Time (refers to measures of how long users take to complete tasks) Input rate (e.g. words per minute) Mental effort (mental resources spend on interaction, e.g. heart rate variability, subjective time estimation) Usage patterns (number of times a certain action has been performed, how much information users access when solving tasks, deviation from the optimal solution etc.) 10. Examples of usability measures (3) Measures of satisfaction Preference measures capture which interface users prefer using Specific attitudes towards the interface (liking, fun, annoyance etc.), toward the content of interface (quality of information, organization of information etc.) Perception of outcomes (users assessment of their performance, users perception of learning, users confidence in the solution to tasks) 11. From Usability to User Experience (1) UX - A relatively new field - Lack of unique definition But lets take for example the ISO definition on UX: A person's perceptions and responses that result from the use or anticipated use of a product, system or service 12. From Usability to User Experience (2) Usability Focused on optimizing task efficiency Removing problems in the user interface Studying interactive products mainly used for work UX Dynamic context- dependent and subjective Focuses also on hedonic qualities of use (identification, s timulation, attra ctiveness etc.) 13. Current trends in UX evaluation 14. As we shift to User Experience Challenge#1: Need for new approaches and methods Existing methods are somewhat unable -it is claimed- to capture experience Challenge#2: The choice of UX evaluation method Qualitative or quantitative methods? Only few studies combine qualitative and quantitative methods 15. What Dimensions of Experience Are Studied? Usability Effectiveness Efficiency Satisfaction UX Emotions and affect Enjoyment Aesthetics And many more 16. What Dimensions of Experience Are Studied? Source: Bargas- Avila and Hornbaek, 2012 17. How do we measure UX? What are the methods applied in UX Qualitative (50 %), quantitative (33%), combination (17%) How UX data are collected? Questionnaires are still the dominant UX evaluation method (53%). Source: Bargas- Avila and Hornbaek, 2012 18. Popular UX evaluation methods Source: Bargas- Avila and Hornbaek, 2012 19. When to use which UX evaluation methods Decision Qualitative vs. Quantitative Attitudinal vs. Behavioral Context of system/ product use 20. The Attitudinal vs. Behavioral Dimension to understand, measure, or inform change of people's stated beliefs e.g. self-reported methods such as surveys, interviews, focus groups, etc. Attitudinal to understand "what people do" with minimal interference from the method itself e.g. A/B testing, observation and video recording, first click testing, eyetracking, etc. Behavioral 21. The Qualitative vs. Quantitative Dimension - the data is usually being gathered directly - they are much better suited for answering question about why or how to fix a problem Qualitative - the data is gathered indirectly, through an instrument, such as a survey or a web server log etc. - they do a much better job answering how many and how much type of questions. Quantitative 22. The Context of Use Dimension This dimension can be described by the following options: Natural or near-natural use of the system/product Scripted use of the system/product Not using the system/product during the study A hybrid of the above 23. What about the use of UX evaluation methods across the development phases ? 24. UX evaluation methods across development phases Product Development Phase Strategize Optimize Assess Goal: Inspire, explore and choose new directions and opportunities Inform and optimize designs in order to reduce risk and improve usability Measure product performance against itself or its competition Approach: Qualitative and Quantitative Mainly Qualitative (formative) Mainly Quantitative (summative) Typical methods: Ethnographic field studies, focus groups, diary studies, surveys, data mining or analytics etc. e.g. cardsorting, field studies, observation, paper prototype and usability studies, etc. e.g. Usability benchmarking, surveys, A/B testing, etc. 25. Some examples from UX research studies 26. Walls of Nicosia The context: "The Walls of Nicosia is an interactive application that runs on a multi-touch table at the Leventeio Museum, Nicosia Students from elementary schools as museum visitors Goal: To conduct a comparative summative evaluation Methods used: Multiple choice knowledge tests UX questionnaires (FunToolkit) Observation Indicative results: Mainly quantitative analysis (pre and post test analysis, t-tests) 27. UX questionnaire for children: the FunToolkit The Again-Again table The Smileyometer 28. Long term UX evaluation of League of Legends (1) The context: The well known MMOG League of Legends Players were students at TEI of Kavala Goal: to identify how UX of players changed over time Methods used: An adaptation of UX Curve Self reported method by drawing curves that depict changes in UX over time 29. Long term UX evaluation of League of Legends (2) Indicative results: Reason Categories Positive Negative General UX 43 14 Ease of use 41 26 Immersion 43 10 Social interaction 38 12 Engagement 32 10 Total 197 72 General UX Curve for each player Number of reasons for UX dimensions 30. Heuristic Evaluation of LMS The context: A well known cloud-based Learning Management System platform Goal: to identify usability problems and possible areas for UX improvement Methods used: Heuristic evaluation Heuristic evaluation protocol adapted by Mehlenbacher et al. (2005) Indicative results: An evaluation report that contains the identified usability and UX problems, their severity, and relevant redesign suggestions 31. Typical structure of the report of heuristic evaluation A numeric identifier of the problem A short description of the problem (and likely difficulties for the user) Specific context (defined as location of the problem in the interface): we identified this generically in terms of affecting the: Structure of the system, including layout, navigation, and basic help functions such as print, help, etc. Content, including actual content for study, exploration, examples, exercises, and so on Global issues, i.e., affecting both structure and learning content The heuristic(s) used The severity rate: severe, moderate, and minor 32. First Click testing with mockups The context: A project for the design of new MOOC interfaces Goal: to create, inform and optimize MOOC interface design Methods used: Interviews and first click testing with users 33. First Click testing - indicative results 34. First Click testing - indicative results 0 2 4 6 8 10 12 35. Some UX resources: Worlds premier magazine for both academics and industry: http://interactions.acm.org/ A comprehensive list of UX methods: http://www.allaboutux.org/ An indicative list of UX tools: http://www.smashingmagazine.com/2011/10/20/compreh ensive-review-usability-user-experience-testing-tools/ Other sources on the web: http://uxmag.com/ http://www.usability.gov/ http://www.uxmatters.com/index.php 36. Key takeaways: You cant afford NOT to evaluate UX Always use a combination of methods Dont forget Usability Do it properly 37. Thank you! Lets keep in touch: Web: about.me/panagiotis_zaharias Email: pz@aueb.gr LinkedIn: gr.linkedin.com/pub/panagiotis- zaharias/0/bb6/a91 Twitter: twitter.com/pan_sugarman