Growing pains: Launching our first longitudinal study

Post on 31-Oct-2014

1.003 views 0 download

Tags:

description

In our ongoing quest to discover how our users think, our UX group conducted a four-month-long study. As we would soon discover, this was no mean feat, as there was very little precedent from the UX community on the nuts and bolts of longitudinal studies. We learned the hard way how to actually set up a longitudinal study, how to maintain it, how to collect and analyze data, and perhaps most importantly, how to get real insights that can go into design solutions. In this session, I'll present a case study of our methodology, our insights, what mistakes we made, and what you can do to make qualitative research a success in your organization.

Transcript of Growing pains: Launching our first longitudinal study

Growing painsLeanna Gingras - @leegoesplacesInternet User Experience 2011

Hi.

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

Grab ‘n Go Gus

Savvy Sally

“that’s not it…that’s not it either…”

“this looks good enough.”

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

1. Our research need

http://www.flickr.com/photos/lwr/2446626352/sizes/l/in/photostream/

Grab ‘n Go Gus

“this looks good enough.”

IUE October 2011 @leegoesplaces

Selecting our tools

IUE October 2011 @leegoesplaces

“A longitudinal study is a correlational research study that involves repeated observations of the same variables over long periods of time.”

--Wikipedia

http://dux.typepad.com/dux/2009/05/usability-over-time-longitudinal-research-studies.html“I participated in the "Best Practices in Longitudinal Research" workshop at CHI 2009 with a small group of academics and practitioners from five countries. … Here's a brief sampling of a few topics from the day's discussions.”

User experience over time: An initial framework“This paper presents an in-depth, five-week ethnographic study that followed 6 individuals during an actual purchase of the Apple iPhone". We found prolonged use to be motivated by different qualities than the ones that provided positive initial experiences.”

(and many abstracts and workshops)

IUE October 2011 @leegoesplaces

UX resources are sparse

IUE October 2011 @leegoesplaces

To learn how to retain participants: Minimizing respondent attrition in longitudinal research: Practical implications from a cohort study of adolescent drinking

More on retaining participants: Retaning Participants in Longitudinal Community Research: A Comprehensive Protocol

Examples of how much to pay: Psychology Department at the University of Vermont

IUE October 2011 @leegoesplaces

Psychology resources galore

IUE October 2011 @leegoesplaces

2. Planning

http://www.flickr.com/photos/dacotahsgirl/299198494/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces

“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”

IUE October 2011 @leegoesplaces

Decide the research question

“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”

IUE October 2011 @leegoesplaces

Decide who to study

“All of it.”

IUE October 2011 @leegoesplaces

Decide what data to collect

“We will be collecting data from usability tests, interviews, Twitter entries, diary entries, and focus group activities.”

IUE October 2011 @leegoesplaces

Decide what data to collect

1 - Initial interview & usability test1 - Contact the recruits for longitudinal study1 - Orientation, 1 – Follow-up email 3 - check-in emails1 - Mid-semester interview based on tweets1 - End-semester interview1 - Focus group

IUE October 2011 @leegoesplaces

Plan touchpoints and activities

IUE October 2011 @leegoesplaces

Plan compensation strategy

IUE October 2011 @leegoesplaces

Plan retention strategy

http://www.flickr.com/photos/goincase/2720494170/sizes/o/in/photostream/

3. Execution

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

The actual study

1 usability test10-30 tweetsAt least 3 diary entries3 in-person interviewsFocus group

IUE October 2011 @leegoesplaces

Screener usability tests

IUE October 2011 @leegoesplaces

Screener usability tests

Agenda:

• Explain study expectations and compensation

• Plan dates for touchpoints throughout semester

IUE October 2011 @leegoesplaces

Kickoff meeting

IUE October 2011 @leegoesplaces

Twitter guidelines

IUE October 2011 @leegoesplaces

Diary due dates

IUE October 2011 @leegoesplaces

Diary entries

Standard set of questions:“Tell me about your toughest moment.”

Observations from Twitter/diary to build interview from:“Does she understand where to find scholarly stuff?”

Exercise:“Let’s revisit one of your earlier Tweets. Walk me through finding resources on Faust. Is there anything you’d do differently?”

IUE October 2011 @leegoesplaces

Interviews

“so how did your research change over the semester?”

IUE October 2011 @leegoesplaces

End of semester focus group

“so how did your research change over the semester?”

IUE October 2011 @leegoesplaces

End of semester focus group

“so how did your research change over the semester?”

IUE October 2011 @leegoesplaces

End of semester workshop

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

Building activities

IUE October 2011 @leegoesplaces

Articulating activities

YAY! We’re done, right?

http://www.flickr.com/photos/markfive/4121088992/ IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

4. Analysis

• Does she understand where to get academic content?

• What keywords did she try?

http://www.island94.org/2009/09/boston-subway-in-vector-format-svg/

Analyze as we go

IUE October 2011 @leegoesplaces

Asked specific questions of the data

“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”

What do they learn? What tools and hacks are useful to them and why? What do they hate and why?

IUE October 2011 @leegoesplaces

“What do they hate and why?”

IUE October 2011 @leegoesplaces

Archived everything

IUE October 2011 @leegoesplaces

What’d we find out?

IUE October 2011 @leegoesplaces

Finding: Emotions run high

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

Finding: Wikipedia is a broad resource

IUE October 2011 @leegoesplaces

Finding: Resources drive research

IUE October 2011 @leegoesplaces

Yeah, but what about the deliverables?

IUE October 2011 @leegoesplaces

Cross-unit brainstorming

IUE October 2011 @leegoesplaces

Persona-building

Project steering

IUE October 2011 @leegoesplaces

Recycle, recycle, recycle

IUE October 2011 @leegoesplaces

IUE October 2011 @leegoesplaces

5. Lessons learned

IUE October 2011 @leegoesplaces

Study management takes time

Retain ‘em

IUE October 2011 @leegoesplaces

Nag, nag, nag

IUE October 2011 @leegoesplaces

Make it easy for them

IUE October 2011 @leegoesplaces

Adapt.

IUE October 2011 @leegoesplaces

1. Define research need2. Planning 3. Execution4. Analysis and findings5. Lessons learned

IUE October 2011 @leegoesplaces

Questions?

http://www.flickr.com/photos/picken/5694078937/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces

Leanna Gingras@leegoesplacesusabilityfail.com

Thanks