You want me to what a practical guide to diary studies
-
Upload
michael-kennedy -
Category
Presentations & Public Speaking
-
view
347 -
download
0
Transcript of You want me to what a practical guide to diary studies
What is a Diary Study?
Definition*: “Research method that involves providing participants with the materials and structure to record daily events, tasks and perceptions around a given subject in order to gain insight into their behaviour and needs over time.”
*http://www.usability.gov/what-and-why/glossary/diary-study.html
Mike’s Study
Purpose: to examine an existing system to prepare for a global redesign
Why a diary study?
• To address the difficulty of determining the impact of known usability issues during onboarding
• What prevents a motivated user from forming patterns of use?
• Examined patterns of use, sticking points, goals and motivations
Vicky’s Study
Purpose: Understand how different user types work with an Email Marketing System over time.
Why a diary study:
● Diary study enabled us to track user actions and satisfaction level OVER TIME – what changes for the user as they become more skilled?
● Factors we were gauging: blockers to success, satisfaction points, business goals and motivations, which features were used most frequently
Liz’s Study Purpose: Gain an understanding of the experience of current customers when they are
upgraded to a newer version of the product. Why a diary study: ● Study conditions recreate the context for transitioning users as closely as possible ● Can gauge how quickly participants adjust to the new version ● Participants have the freedom to report any type of issue or concern ● Longer timeline distinguishes initial learning from persistent issues
What can you learn from a Diary Study?
It can provide a better understanding of usage patterns over time: • Learnability • Acceptance of a product or product features • Issues or challenges that impede usage
Generally: most any type of change in behavior or attitude
Advantages of Diary Studies
• Authenticity of information collected: – User’s environment rather than a lab setting – Participants feel safer so they’re more likely to be open – Can get past initial learning
• Benefits for participants: – Less intrusive – More flexible (e.g., scheduling)
• Can uncover unexpected issues
Disadvantages & Challenges
• Overall: – Time- and resource-intensive – Logistically unwieldy – Less control over study environment
• Participants: – Recruitment, Management, & Retention
• Data: – Self-reported, but use the resources available to you – Overwhelming!
A Practitioner’s Checklist:
❏ Designing the Study ❏ Recruiting ❏ Managing Participants ❏ Managing the Data ❏ Analyzing the Data ❏ Reporting the Results
Designing the Study
Managing Length of Study • Long enough to capture desired
behaviors • Stay meaningful for participants and
stakeholders • Reflect users’ natural work cadence as
much as possible
Designing the Study
Components • Ongoing feedback on user experience
– Diary entries • Interim check-ins
– Surveys / interviews • Final review touch point
– Follow-up interviews
Designing the Study
Unstructured • Allows for more unexpected, possibly more realistic data
• BUT, may not get desired data, and lots of manual labor sorting through results
Structured • Less time consuming data analysis, higher probability of
getting relevant data
• BUT, may not reflect user’s natural workflow, and a lot of dictating may feel too formal and restrictive to users
Example for study design - Diary Questions
• Name
• What did you do with the tool today?
• What did you want to accomplish?
• Did you accomplish what you wanted?
• What specific features did you use?
• What did you like about your experience today?
• What did you not like about your experience today?
• What else would you like us to know?
• Screen shots (optional)
Example for study design - Setup
“Is the tool meeting your business goals?” “What features have you used?” “Which features do you like most / least?”
“How has your experience changed since 3 weeks ago?” “Have you tried any new features?” “Do you feel more skilled than you did 3 weeks ago?”
Example for study design - Phone Interviews
• Follow-up on diary entries and surveys
• Kept conversation less structured • Gave deeper understanding of user
goals, issues, etc. • Benefit of hearing user’s tone
(excitement or frustration)
A Practitioner’s Checklist:
✔ Designing the Study ❏ Recruiting ❏ Managing Participants ❏ Managing the Data ❏ Analyzing the Data ❏ Reporting the Results
Recruiting
Be realistic about how long this is likely to take • It will probably take a lot longer than you think to hit your number • Be prepared to be flexible, prioritize screener criteria if necessary
Strategies for minimizing attrition:
• Graduated payments – Have payments tied to something you can control (e.g., interviews)
• If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it
A Practitioner’s Checklist:
✔ Designing the Study ✔ Recruiting ❏ Managing Participants ❏ Managing the Data ❏ Analyzing the Data ❏ Reporting the Results
Managing Participants
• Make it as easy as possible for them – Online data entry – Include links in every communication – Have some structure (not completely open-ended)
• Give participants permission to be brief and/or repeat themselves, and let them be themselves!
– Remember, it’s about a truer contextual perspective
• Guard against being too invasive – Let participants choose the frequency of reminders – Make it explicit that reminders are automated
• Monitor for compliance and to answer questions or redirect to support
A Practitioner’s Checklist:
✔ Designing the Study ✔ Recruiting ✔ Managing Participants ❏ Managing the Data ❏ Analyzing the Data ❏ Reporting the Results
Managing the Data
There are tons of great online resources -- make use of them! • Online form for diary entries
– Easy for participants to access – Researcher has access to data in real time to monitor
• Online self-scheduler for interviews/checkins • Online form for data entry (e.g., for interviews) • Survey tools for questionnaires, rating questions etc.
But watch out for:
• Privacy issues • Appropriateness for your participants
A Practitioner’s Checklist:
✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ❏ Analyzing the Data ❏ Reporting the Results
Analyzing the Data
It’s largely a manual process: 1. Read through the data looking for issues, patterns etc. 2. Make a list of tags and cross reference with your data 3. Repeat
Qualitative data analysis tools: • Lots of them available • Traditionally have been more helpful for ongoing data collection needs than
for a one-time study • Worth keeping an eye out
Analyzing the Data
Strategies: • Try to Identify categories and tags in advance as part of defining
the study • “Find” is your friend but watch out for:
– misspellings – different word forms – different words for the same/similar concepts – different meanings for the same words
• Have a method for collecting or flagging verbatims as you go • Use learnings from other touch-points to help identify tags &
categories
Analyzing the Data
Don’t wait until the end! • Keep the job manageable • Keep analysis timely • Allow for mid-course corrections
Include some selected-response measures: • Easy to analyze • Consistent across participants and over time • Provide metrics to supplement open-ended responses Look for the surprises!
My Approach Two spreadsheets:
1. Raw data: diary entries, numbered chronologically 2. “Processed” data, with columns for:
1. Tags 2. Entry numbers 3. Verbatims
Same 3 Likert questions in all diary entries: • “easy” results
My approach: Get your participants to help you!
Two weeks before the final interview: • Created individual spreadsheets • Asked participants to review and think about:
1. Their overall experience making the transition 2. What they liked best 3. What was the biggest challenge 4. What we could have done to make the transition easier
DIY vs. DIFM
• Emerging tools like dscout can reduce overhead – The trade-off can be
flexibility – Decide what you want to
accomplish with your methodology before you approach any tools
• We were curious, so we’re running a study right now with thanks to our UXPA organizers!
A Practitioner’s Checklist:
✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ❏ Reporting the Results
Reporting the Results Changes over time • Learnability
– (Qual) Frequency with which issues or problems are reported – (Quant) How easy/difficult was it for you to meet your goals in this session? To
what extent were you able to meet your goals in this session?
• Long-term patterns of use (what features remain important over time)
– (Qual) Changes in users’ stated goals – (Quant) % of use of different features
• Changes in user opinions over time – (Qual) % of comments classified is positive/negative/neutral – (Quant) How satisfied/dissatisfied are you with the product?
A Practitioner’s Checklist:
✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ✔ Reporting the Results
Lessons Learned
Mike: 1. Planned vs. Unplanned learnings (be prepared to dive back in) 2. It all comes down to Codifying open responses 3. Visualize learnings for broad adoption
Vicky:
1. Try to start all participants at the same time 2. Automate as much as possible 3. Review the data as you go - don’t wait until the end
Liz: 1. Structure can be your friend 2. Control what you can but don’t try to control what you can’t 3. Embrace the chaos!
References
1. http://www.uxbooth.com/articles/jumpstart-design-research-with-a-diary-study/
2. http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/
3. https://uxmag.com/articles/making-the-most-of-ethnographic-research
4. http://www.spotless.co.uk/insights/6-things-for-ux-diary-study
5. http://www.uxbooth.com/articles/dear-diary-hard-say-goodbye/
6. https://blinkux.com/blog/dear-digital-diary-a-powerful-tool-in-user-research/
7. http://uxpamagazine.org/dear-diary/
8. http://www.slideshare.net/jaremfan/the-goodness-of-diary-studies