Exploring the research toolbox: what to use, when and why.

Calgary - Vancouver - Toronto

Anthony HempellDirector, User Experience

Tara Franz,Director, Research

Part 1: Why do research, anyway?Part 2: Understand the landscapePart 3: Pushback & pitfallsPart 4: Exploring the toolboxPart 5: Case Study: ATB

Part 1:

Q: Why do research, anyway?

Part 1:

Q: Why do research, anyway?

“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein

Part 1:

Q: Why do research, anyway?

“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein

“I believe in innovation and that the way you get innovation is you fund research and you learn the basic facts” - Bill Gates

Why Research?

Why Research?

measurements, facts, observations

“know nothing”words, numbers, symbols

Why Research?

descriptions, correlation, comparison, relationships

“know what”

meaning & purpose

Why Research?

framed experiences, values, context, expertise, intuition;

organizational routines, processes, practices and norms

understanding & capability

“know how”

Why Research?confidence to act

judgement, ethical and aesthetic values

“know why”

So how do we do that?

gather server logs, conduct surveys, perform usability tests

comparison & analysis of data

communicate and evangelize findings

turn insights into policies, methods and actions

Part 1: A: We want to fill our gaps in the data to create useful information, knowledge & wisdom which can be used to make the world a better place.

Part 2: Understand the landscape

Part 2: Understand the landscape

“If you know the enemy and know yourself, you need not fear the result of a hundred battles.” - Sun Tzu

Some practical questions before we

begin...What do we already know?

Who are the users? Who are the stakeholders?

Where are they located?What are we trying to discover?When do we need our findings?

How are the findings going to be used?

What do we already know?

What research results do we already have access to?

How recent is it?

How was it collected?

Is it valid?

What does it tell us?

Has it been converted into organizational assets like policies, procedures, shared knowledge?

Who are the users?

What research has been done on current audience?

Is any of it qualitative?

How has the current audience / user / customer base been determined?

Are there demographic aspects to customer base that are of interest -- language, age, mobility, cultural, technology use, etc.

Who are the stakeholders?

Who has ultimate accountability (financial and otherwise)?

Who is the primary point of contact / responsibility for coordination?

Who else needs to be consulted? What are their roles?

Who needs to be informed and when?

Who would be most affected if the project goes poorly?

What are the desired outcomes + effects?

Where users + stakeholders located?

Local? Provincial?

National? International?

Urban / Suburban / Rural

Regional attitudes / behaviours?

When do we need our findings?

Deadlines -- what is driving them?

Possibility of future research?

How are our findings going to be used?

Creating requirements?

Concept creation or validation?

Setting policies / product direction?

Within small project team or communicating across departments?

Part 2: Gather as much data about your surroundings as possible.

Part 3: Pushback & Pitfalls

Part 3: Pushback & Pitfalls

“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law

Part 3: Pushback & Pitfalls

“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law

“We don’t devote enough scientific research to finding a cure for jerks” - Bill Watterson

Some common pitfalls“We already have a research department

that does that”

“Marketing did some market research last quarter, we can use that”

“Let’s put a survey up on our website”

“That sounds like it will take too long / be too expensive -- let’s talk to some of my

friends / co-workers / kids instead”

“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)

Some common pitfalls“We already have a research department

that does that”

“Marketing did some market research last quarter, we can use that”

“Let’s put a survey up on our website”

“That sounds like it will take too long / be too expensive -- let’s talk to some of my

friends / co-workers / kids instead”

“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)

Some common pitfalls - why they happen

Organizational pressure for concrete results, yesterday

Research not seen as valuable / waste of time

Lack of knowledge about research methods

Lack of understanding of research benefits

Quote

Research

Strategy

Design

Culture

Testing

Novice Beginner Intermediate Advanced Enlightened

NoneAd-hoc,

AnecdotalData-driven

Qualitative + quantitative

Triangulation: data & observation;

passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value

Focus on business requirements

"Best practices"

Occasionally

Individual champions;

traction dif!cult

Balance needs of business & users

Style guides + processes de!ned; ideation & creativity

Sometimes

Some managerial advocates +

success

Business is customer-driven

Agile / Lean: small teams, rapid design sessions, iterative

Always

Empowered by C-level executive

Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx

"We don't have time / money for

that!""Just do it."

"Think before you act."

"Use lessons learned to predict, plan and create opportunity."

"Review, learn, adjust, execute, review, learn…"

Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/

0 1 2 3 4

UX Maturity Model

Quote

Research

Strategy

Design

Culture

Testing

Novice Beginner Intermediate Advanced Enlightened

NoneAd-hoc,

AnecdotalData-driven

Qualitative + quantitative

Triangulation: data & observation;

passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value

Focus on business requirements

"Best practices"

Occasionally

Individual champions;

traction dif!cult

Balance needs of business & users

Style guides + processes de!ned; ideation & creativity

Sometimes

Some managerial advocates +

success

Business is customer-driven

Agile / Lean: small teams, rapid design sessions, iterative

Always

Empowered by C-level executive

Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx

"We don't have time / money for

that!""Just do it."

"Think before you act."

"Use lessons learned to predict, plan and create opportunity."

"Review, learn, adjust, execute, review, learn…"

Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/

0 1 2 3 4

Quote

Research

Strategy

Design

Culture

Testing

Novice Beginner Intermediate Advanced Enlightened

NoneAd-hoc,

AnecdotalData-driven

Qualitative + quantitative

Triangulation: data & observation;

passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value

Focus on business requirements

"Best practices"

Occasionally

Individual champions;

traction dif!cult

Balance needs of business & users

Style guides + processes de!ned; ideation & creativity

Sometimes

Some managerial advocates +

success

Business is customer-driven

Agile / Lean: small teams, rapid design sessions, iterative

Always

Empowered by C-level executive

Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx

"We don't have time / money for

that!""Just do it."

"Think before you act."

"Use lessons learned to predict, plan and create opportunity."

"Review, learn, adjust, execute, review, learn…"

Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/

0 1 2 3 4

Company A

Company B

Company C

Company D

Some common pitfalls - how to avoid

Know your research question. Always have concrete, measurable goals for your research that you can refer to.

Build a research strategy and plan.

Be willing to listen to alternative points of view, but don’t deviate from sound research practices.

Protect the validity of your methods. It is your basic currency.

If you’re getting pushback, take baby steps.

Focus on the value.

Don’t force it. If the glass slipper doesn’t fit, it doesn’t fit.

Part 3: Observe and understand the organization you’re working with. Propose measured improvements. Make friends.

Part 4: Exploring the toolbox

“If all you have is a hammer, everything looks like a nail.” - Abraham Maslow

Focus groupsIn-depth interviews (in person / remote)EthnographyCard sortingUsability testingUsability benchmarkingSurveysObservationMessage board miningCoding customer feedback emailsA/B testingWeb AnalyticsDiariesEye trackingOnline Panels / Communitiesand more...

http://nform.com/cards/

Let’s look in the toolbox

Research dimensionsBasic vs AppliedCross-sectional vs LongitudinalResearch purpose: Exploratory, descriptive, explanatory

Data collection techniques:Quantitative vs QualitativeSmall vs Large sampleModerated vs UnmoderatedAttitudinal vs Behavioural

http://qualitative.wikidot.com/dimensions-of-research

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Moderated• Can observe subtle cues:

non-verbal behaviour, tone of voice, etc.

• More chances for further inquiry based on context

• Harder to eliminate biases• Much higher quality of data

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Unmoderated• Can  facilitate  much  larger  sample  

sizes• Not  as  many  constraints  with  

different  9me  zones• Limited  context• Media  biases• Respondents  have  more  9me  to  

par9cipate  (send  photos,  diaries,  ponder  thoughts)

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

• Small  test  group  but  deeper  understanding• Gathering  informa9on  or  themes  from  texts,  conversa9ons  or  

interviews• Open  ended;  changeable.  There  is  a  maybe.  • Opportunity  for  flexibility;  serendipity

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

• More closed questioning with directed response (pick Yes or No, there is no maybe)

• Gathering data with an “instrument”• Derive measures or variables (operationalization)• Error of measurement

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

• Hybrid:  gather  data  qualita9vely  and  then  code  into  variables  to  make  inferences  quan9ta9vely.

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Discovering  mo9va9ons  in  context

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix BLUE  =  measures  aUtudes

RED=  measures  behaviours

Focus  groups

In  person  interviews

Ethnography

Usability  tes9ng

Telephone  interviews

Card  sor9ng

Diaries

Online  panels

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Observing  and  tracking  behaviours  in  context

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Observa9on

Usability  benchmarking

Eye  tracking

BLUE  =  measures  aUtudes

RED=  measures  behaviours

Online  panels

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Discovering  general  themes  and  aUtudes

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Message  board  mining

Customer  feedback  emails

BLUE  =  measures  aUtudes

RED=  measures  behaviours

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Gathering  data  about  behaviours

Mod

erated:  high  eff

ort  /  high  context U

nmoderated:  low

er  effort  /  lower  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Web  Analy9cs  

A/B  tes9ng

Customer  feedback  emails

Unmoderated  usability  tes9ng

BLUE  =  measures  aUtudes

RED=  measures  behaviours

Mod

erated:  high  eff

ort  /  high  context

Unm

oderated:  lower  effort  /  low

er  context

Quan8ta8ve:  how  many  and  how  much

Qualita8ve:  why  and  how  to  fix

Focus  groups

In  person  interviews

Ethnography

Surveys

Web  Analy9cs  

Observa9on

Usability  benchmarking

Usability  tes9ng

A/B  tes9ng

Message  board  mining

Customer  feedback  emails

Telephone  interviews

Card  sor9ng

Unmoderated  usability  tes9ng

Eye  tracking

BLUE  =  measures  aUtudes

RED=  measures  behaviours

Diaries

Digital  Ethnography

Online  panels

Opportunities:Triangulation

What data do you currently have? How was it collected? Consider using a method that is complimentary.

Do you have the opportunity to run two or more research methods? Use different methods to look at the same problem from different angles.

Part 4: Know the strengths and weaknesses of every method and pick wisely. If possible, research with more than one.

Part 5: Case Study: ATB Financial

“Our youth now love luxury. They have bad manners, contempt for authority; they show disrespect for their elders and love chatter in place of exercise” - Socrates

• Problem: –Required insight and

understanding of the millennial demographic to increase market share within the group

–Make something

awesome

ATB Case Study

• what do we already know? – Lots of data about ATB customer needs and

current use –Also were aware of Millennial research and

some overarching themes

ATB Case Study: Landscape

• Who are the users? –Target of urban Millennials and drafted up a

demographic framework to contain the sampling

• Who are the stakeholders? –Emerge group at ATB and big wigs –Ensure we had a plan to have them involved

ATB Case Study: Landscape

• Where are they located?–Calgary, Alberta

• What are we trying to discover? –Something we can take to market to engage a new

market for ATB

ATB Case Study: Landscape

• When do we need our findings? –We had 3 months time

• How are the findings going to be used? – To create a new offering

ATB Case Study: Landscape

• The plan:- Secondary research review on Millennials and ATB data - Ethnographic Immersion – be one with the animals

- DEBRIEF (this is where the real insight starts to happen and stakeholders are involved) - User Interviews – understand the ways and the why

- DEBRIEF again – what have we learned? What else do we want to know? Let’s get our persona on!

- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)

- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote - In field feedback (this is when you call your friends)

- RDL and prototyping- Concept testing - Finalize and forge ahead!

ATB Case Study: Toolbox

- How was everyone kept in the loop?

Collaboration

Remember the triangle?

- Secondary research review on Millennials and ATB data

- Ethnographic Immersion – be one with the animals - DEBRIEF (this is where the real insight starts to happen and

stakeholders are involved)

Stage One: Foundation

- User Interviews – understand the ways and the why - DEBRIEF again – what have we learned? What else do we

want to know? Let’s get our persona on!

• Semi structured phone and in person ethnographic interviews: we dined, we coffee-d, we chatted, we made life plans together

• Debrief goal: Personas

Stage Two: Patterns

- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote

Stage Three: Innovation

• Explored creative ways to solve the problems we had: personal organizers, teachers, support workers, rehabilitation counselors

• Spoke with people who were on the extreme side of everything we discovered: wealthy and homeless

• Debrief goal: 5 solid ideas with a framework. • Opportunity to vote on favorite idea • In field feedback

- RDL: include your stakeholders and sometimes users - Prototyping- Concept testing:

- recruit the right people - Test on the right screen

- Finalize and forge ahead!

Stage Four: Building

Anthony Hempellanthony@uxguys.com

Web: www.uxguys.comTwitter: @uxguys

Thank You!

Tara Franztarafranz@gmail.com

Web: about.me/tarafranzTwitter: @tarafranz