1. Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not UXPA-DC Conference Redux 2013 October 25, 2013Kristen Davis Dick Horst kdavis@userworks.comdhorst@userworks.com1738 Elton Rd., Suite 138, Silver Spring, MD 20903 (301) 431-0500 www.userworks.com

2. Presentation Objectives To explore ways to become a more efficient/effective notetaker by: Developing notetaking and test planning skills Recognizing whats important to note and whats not Understanding the pros and cons of various styles Determining what metrics to capture and how to define them Discussing a list of best practices and tipsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20132 3. Why is Effective Notetaking Important? To improve the data analysis process by: Capturing the appropriate data the first time Making analyses more efficient Increasing accuracy and completeness of data Identifying trends across participantsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20133 4. Typical Context One-on-one, moderated sessions (participant in-person or remote) Moderator interacting with participant, notetaker elsewhere Participant attempting typical task scenarios at the direction of the moderator Participant thinking aloud Some usability test principles and best practices may apply to taking notes in focus groups, user interviews, or ethnographic settings, but not allUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20134 5. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 6. Misconceptions No preparation needed Just take notes Anyone can do it Fast typing is only key skill necessary One-size-fits-all approachUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20136 7. Set Goals Be familiar with the product under test Understand overall study goals Understand specific task goals Set goals prior to data collection Record appropriate data Target type(s) of dataUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20137 8. Notetaking Styles/Approaches Determine appropriate style Verbatim (court stenographer) rarely ideal High level interpretation only, not behavior or comments Coding behaviors in addition to free form notesDetermine number of notetakers Moderator is only notetaker not ideal Multiple notetakers looking for different things Multiple notetakers covering the same things (redundancy) Taking notes from playback of a video/audio recording (not live)Determine data collection tools Checklist (word processing or spreadsheet based, or paper-based) Automated tool (e.g., Morae, Ovo, Silverback)UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20138 9. Data Collection Define task completion rating scale Participant performance to be categorized Systematic and consistent scoringNotetaker can quickly record Task scoring (success vs failure) Errors, wrong paths Pre-identified issues of interest Occasional verbatimsData logging software records Clickpaths Time on task Shorthand codes for behaviors or incidentsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 20139 10. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 11. Study Context Product: NIHSeniorHealth.gov website Participants: Seniors who look for health information online Study goals: What navigation path do participants use when looking for specific pieces of information? Do participants notice and use the pagination? Does the visual treatment used in left hand navigation clearly indicate location within specific health topic?Task goals: Are participants able to locate information on the 2nd page? Are the left hand navigation labels clear?Task: Why do people with diabetes need to check their feet?UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201311 12. Video ExampleUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201312 13. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 14. Study Context Product: HealthIT.gov website Participants: Private physicians Study goals: Are navigation labels clear? Does information architecture make sense? What suggestions do participants have to improve content?Task goal:What navigation path do participants use when looking for specific pieces of information?Task: Medical professionals and hospitals are encouraged to participate in the Government EHR reimbursement programs early to receive the maximum payment. Find out when the Government will start or has started these reimbursement payments. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201314 15. Video ExampleUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201315 16. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 17. Study Context Product: Blood glucose meter Participants: People with diabetes who use glucose meter Rating scale for scoring task completion: Completed with ease: Participant easily completed the task Completed with minor difficulty: Participant struggled somewhat to complete the task (e.g., attempted more than two paths) Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) Failed to complete: Participant was unable to complete the taskTask: Change meter default settingsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201317 18. Video ExampleUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201318 19. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 20. Pitfalls Participant says one thing and does another Participant thinks they have been successful when they really havent Participant doesnt want to look bad, so bluffs Participant initially struggles, then succeeds, and speaks highly of product Participant blames self and rates product highly despite disastrous task performance Participant is overly chatty and goes off on tangentsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201320 21. Considerations Being selective in information noted, without introducing biases (seeing what you want/expect to see) Noting participant behavior versus comments Whether to try to capture participants clickpath, menu choices, data entry Whether to capture timing Whether to use a shorthand code for observations Whether to capture participants non-verbal behavior Whether to note good clips for a highlights videoUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201321 22. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 23. "Take-Aways" for Effective Notetaking (1 of 2) Ensure the project team values role of notetaker Choose someone with appropriate education and experience Include notetaker in test planning discussions and project debriefs Establish project-specific notetaking procedures and coding schemes Be sure notetaker is thoroughly familiar with: Product Project goals Task scenarios Task goalsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201323 24. "Take-Aways" for Effective Notetaking (2 of 2) Focus on what participant does, not just what they say Just dont note observations; interpret what you are seeing Review your notes and fill in any gaps as soon as possible after sessions Consult with the team and client throughout project Check available storage space on recording devices before each session Spot check recordings throughout study in-between sessions Discuss findings with notetaker/moderator after each session, if time allowsUXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201324 25. Questions/Comments Kristen DavisDick HorstUserWorks, Inc.UserWorks, Inc.Senior User Experience Specialist/ Note-Taker ExtraordinairePresident/ Principal User Experience Specialistkdavis@userworks.comdhorst@userworks.comwww.userworks.com (301) 431-0500UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 201325