UXPA International 2013 The Note-Taker's Perspective

www.userworks.com(301) 431-0500

Kristen Davis & Dick Horst

UXPA International Conference 2013 July 10, 2013

The Note-Taker's Perspective The Note-Taker's Perspective During Usability Testing: During Usability Testing:

Recognizing What's Important, What's Recognizing What's Important, What's NotNot

The Note-Taker's Perspective During Usability TestingUXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc.

OverviewOverview

Presentation Objectives: To collectively explore ways to become a more efficient and effective note-taker by:

• Developing the skills and planning necessary

• Recognizing what’s important to note and what’s not

• Understanding the pros and cons of various note-taking styles

• Determining what metrics to define and utilize in note-taking

• Creating a list of note-taking best practices and tips

2


Why is Effective Note-Taking Why is Effective Note-Taking Important?Important? To improve the data analysis process by:

• Capturing the appropriate data the first time; less need for reviewing recordings

• Making analyses more efficient by categorizing observations on the fly

• Increasing accuracy and completeness of notes (so you don’t miss anything important)

• Identifying trends across participants

3


Typical ContextTypical Context

One-on-one, moderated sessions (with the participant in-person or remote)

Moderator interacting with the participant, note-taker in the background taking notes

Participant attempting typical task scenarios at the direction of the moderator

Participant thinking aloud

Many of the same principles and best practices would also apply to taking notes in focus groups, user interviews, or ethnographic settings, but not all

4


ExerciseExercise

Think about and/or record your observations as you watch this video clip

6


DiscussionDiscussion

What did you note? Did you capture what you thought were the key findings? Did you miss anything?

7


Note-Taking MisconceptionsNote-Taking Misconceptions

No preparation needed Take the notes, then figure out how to analyze and

summarize the data Anyone can do it Attention to detail, fast typing are the key skills A one-size-fits-all approach will work for all

practitioners and for all products under evaluation

8


The ContextThe Context

Product: NIHSeniorHealth.gov website Participant: Seniors who look for health information online Task: What types of food or drink are related to balance

problems? Study goals:

• What navigation path do participants use when looking for specific pieces of information?

• Do participants notice and use the pagination?

• Does the visual treatment used in the left hand navigation clearly indicate location within a specific health topic?

Task goals: Are participants able to locate information on the 2nd page? Are the left hand navigation labels clear?

10


ExerciseExercise


11



What did you note? Did you capture the key findings? Did you miss anything? Did you note any of these:

• Study goals: What navigation path do participants use when looking for

specific pieces of information? Do participants notice and use the pagination? Does the visual design treatment used in the left hand navigation

clearly indicate location within the topic?

• Task goals: Are participants able to locate information on the 2nd page? Are the left hand navigation labels clear?

12


Lessons Learned: Understand the Lessons Learned: Understand the GoalsGoals Important to be familiar with the product and

understand the overall goals of the study and the specific goals of the tasks

Goals should be set prior to data collection in order to:• Record appropriate data • Target the type of data that will be useful when summarizing

and reporting the findings

13


Different Note-Taking ApproachesDifferent Note-Taking Approaches

Would you try to note:

• Everything the participant says?

• Everything the participant does? Pages visited Links visited Scrolling, hovering Participant’s demeanor Non-verbal signs

Or would you be more selective?

• Task scoring (success vs failure)

• Errors, wrong paths

• Pre-identified issues of interest

• Occasional verbatims

14


Different Note-Taking StylesDifferent Note-Taking Styles

Moderator also takes notes The court stenographer, verbatim High level interpretation only, not behavior or comment Coding behaviors in addition to free form notes Working from a checklist Multiple note-takers looking for different things Multiple note-takers working redundantly Taking notes from playback of a video/audio recording of the

session Using a tool like Morae, Ovo, Silverback, free form typing

into a word processing or spreadsheet program, or paper-based (checklist)

15


Practice ExercisePractice Exercise

Product: HealthIT.gov website Participant: Private physician Task: Medical professionals and hospitals are encouraged to

participate in the Government EHR reimbursement programs, early, to receive the maximum payment. Find out when the Government will start or has started these reimbursement payments.

Study goals:• Are the labels used for navigation clear?

• Does the information architecture make sense?

• What suggestions do participants have to improve the content?

Task goal:• What navigation path do participants use when looking for specific

pieces of information?

17


ExerciseExercise


18



What did you note? Did you capture the key findings? Did you miss anything?

19


Lessons Learned: Record Lessons Learned: Record Only Key Only Key PointsPoints The method of note-taking impacts the type of data

collected during the evaluation Depending on the evaluation objective, one style or a

combination of styles may be more appropriate

20


Challenges for the Note-taker (and Challenges for the Note-taker (and Moderator)Moderator) Participant says one thing and does another Participant thinks they have been successful when they

really haven’t Participant doesn’t want to look bad, so bluffs Participant initially struggles, then succeeds, and

speaks highly of the product Participant blames self and rates product highly despite

disastrous task performance Participant is overly chatty and goes off on tangents

21


More Challenges for the Note-More Challenges for the Note-TakerTaker Being selective in the information you note, without

introducing your biases (seeing what you want/expect to see)

Noting participant behavior versus comments Whether to try to capture participant’s clickpath, menu

choices, data entry Whether to capture timing Whether to use a shorthand “code” for observations Whether to capture participant’s non-verbal behavior Whether to note good clips for a highlights video

22


Practice ExercisePractice Exercise

Product: Blood glucose meter Participant: Person with diabetes who uses a meter Task: Change default settings in the meter Rating scale for scoring task completion:

• Completed with ease: Participant easily completed the task (Score: 3)

• Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2)

• Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1)

• Failed to complete: Participant was unable to complete the task (Score: 0)

24


ExerciseExercise


25



What did you note? Did you capture the key findings? Did you miss anything? What score did you give the participant?

• Completed with ease: Participant easily completed the task (Score: 3)

• Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2)

• Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1)

• Failed to complete: Participant was unable to complete the task (Score: 0)

26


Lessons Learned: Lessons Learned: Define MetricsDefine Metrics

Using a task completion rating scale allows: • Participant performance to be categorized• Note-taker to record the scores in a systematic and

consistent way Carefully defining the performance categories before

the fact helps the note-taker• Know what to look for• Efficiently categorize on the fly Some additional metrics to consider (facilitated by

commercial data logging software): • Clickpaths• Time on task• Shorthand codes for behaviors or incidents

Example: Number of times a participant clicked the pagination

27


"Take-Aways" for Effective Note-"Take-Aways" for Effective Note-TakingTaking Create a mindset that values the role of the note-taker Choose someone with appropriate education and

experience to appreciate the design issues of interest Include the note-taker in test planning discussions and

debriefs Establish project-specific note-taking procedures and

coding schemes; anticipate analysis needs when planning note-taking

Be sure the note-taker is thoroughly familiar with the product interface, task scenarios, and project goals

29


And a Few Tips for the Effective And a Few Tips for the Effective Note-TakerNote-Taker Focus on what the participant does, not just what they say Strive to interpret what you are seeing on the fly and take

notes about that, not just “raw” observations Review your notes and fill in any gaps as soon as possible

after the session while observations are fresh in your mind Consult with the team and client mid-course to be sure

your notes reflect project goals and team intent Check available storage space on recording devices

before each session Spot check recordings throughout the study to ensure that

everything is working as intended

30


Questions/Comments?Questions/Comments?

Kristen DavisUserWorks, Inc.

User Experience Specialist/ Note-Taker Extraordinaire

[email protected]

31

Dick HorstUserWorks, Inc.

President/Principal User Experience Specialist

[email protected]

www.userworks.com(301) 431-0500

UXPA International 2013 The Note-Taker's Perspective

Technology

Transcript of UXPA International 2013 The Note-Taker's Perspective