MIXED METHODS AND THE

UNKNOWN UNKNOWNS OF UX

JUDITH MÜHLENHOFF

@judithmp

HOW MIGHT WE MARRY QUANT AND QUAL METHODS

TO SEE THE BIG PICTURE?

UXCE15, BERLIN

User and Innovation Researcher

Sociologist

PhD candidate, “Culture-Driven Innovation”

Owner of the User Research Lab: enhancing audience engagement with qualitative

insights

@judithmp

Judith

HELLO, MY NAME IS…

“There are known knowns; there are things we know

we know.

We also know there are known unknowns; that is to

say we know there are some things we do not know.

But there are also unknown unknowns – the ones we

don't know we don't know.”

Donald Rumsfeld, 2002

THE UNKNOWN UNKNOWNS

R. D. Ward [Public domain], via Wikimedia Commons

https://commons.wikimedia.org/wiki/File%3ADefense.gov_News_Photo_020304-D-9880W-013.jpg

Donald Rumsfeld

1. GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

2. MIXED METHODS

2.1 WHY & HOW

2.2 CASES

3. TAKE AWAYS

AGENDA

Type of Knowns and Unknowns

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Knowns Unknowns

known

Known Knowns

Things we are aware of and

understand

Known Unknowns

Things we are aware of but

don‘t understand

unknown

Unknown Knowns

Things we understand but are

not aware of

Unknown Unknowns

Things we are neither aware

of nor understand

“With enough data, the numbers

speak for themselves.”

Chris Anderson, 2008

Big Data – The Holy Grail of Customer Experience?

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Anderson pic by James Duncan (Flickr)

https://commons.wikimedia.org/wiki/File%3AEtech05_Chris.jpg

Numbers do not speak for themselves, they have to be interpreted within their context.

Big Data Needs Deep Data

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Big Data are not objective, they inhibit high risks of hidden biases and faulty data.

Big Data Needs Deep Data

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Big Data tell us about “how many/how often”, but not about the “why” and “how”.

Big Data Needs Deep Data

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Qualitative methods, like ethnography, provide us with “Deep Data”

Tell us about the why and how of human behaviour

Put data into the context of the messy social world

Explore neglected research fields in depth and breadth

Big Data Needs Deep Data

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Types of Knowns and Unknowns and Data

GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

Knowns Unknowns

known

Known Knowns

Things we are aware of and

understand

• Known Data

• To be updated and refined

Known Unknowns

Things we are aware of but

don‘t understand

• Insufficient Big Data

Deep Data

• To be explored in depth

unknown

Unknown Knowns

Things we understand but are

not aware of

• Esp. Big Data

• To be made aware of

Unknown Unknowns

Things we are neither aware

of nor understand

• Deep Data + Big Data

• To be explored in depth

and breadth

Mixed Methods

Qualitative and quantitative methods each have its strengths and weaknesses.

With mixed methods, the strength of a single method outweighs the weakness of the

other.

Why Mixed Methods?

MIXED METHODS – WHY & HOW

https://www.flickr.com/photos/archeon/

Combining methods provide a bigger picture, like different angles in a movie.

These can lead to corroboration, verification…

… or might spur complete new thinking and theories. Unknown Unkowns

Why Mixed Methods?

MIXED METHODS – WHY & HOW

At which stage do you combine methods?

Do you choose an identical or non-identical sample?

What is your overall goal of mixing methods (e.g. corroboration, exploring)?

Will you focus on one method?

How to Conduct Mixed Methods?

MIXED METHODS – WHY & HOW

How to Conduct Mixed Methods?

MIXED METHODS – WHY & HOW

Sequential Design

Embedded Design

Parallel Design

Fully Integrated Design

Conversion Design

Help newsrooms to measure the quantitative and qualitative impact of their stories.

Two tasks:

Understanding what it means for a story to “do well” and what happened to cause

that.

Managing an event-tracking workflow while juggling other responsibilities

MIXED METHODS – CASES

NewsLynx (Columbia Journalism School/Tow Center)

Proposal of a taxonomy to measure impact beyond metrics

MIXED METHODS – CASES

NewsLynx Columbia Journalism School/Tow Center)

Ch

References

Sources

MIXED METHODS – CASES

Self-Observation: Diary Studies and Experience Sampling Method (ESM)

www.user-research-lab.org

Digital Media Diary (User Research Lab)

Self-observation of news consumption

Experience Sampling Method (e.g. Google)

Track experiences in the moment

Qualitative data at larger scale

MIXED METHODS – CASES

Self-Observation: Diary Studies and Experience Sampling Method (ESM)

www.pacoapp.com

Exploration of computer usage and temporality through tracking…

MIXED METHODS – CASES

Ethno-Mining (Intel)

Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining

…and participatory data interpretation

MIXED METHODS – CASES

Ethno-Mining (Intel)

Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining

Take Aways

1. Leave your method comfort zone.

2. Aim for highly integrated research designs with different angles.

3. Take advantage of participatory data interpretation.

4. Plan and decide deliberately.

5. It takes a lot of effort, but may lead you to your unknown

unknowns.

TAKE AWAYS

THANK YOU!

Judith Mühlenhoff @judithmp

www.user-research-lab.org

References

Newslynx: http://newslynx.org/

Big Data:

Kate Crawford: The hidden biases in data https://hbr.org/2013/04/the-hidden-biases-in-big-data/

Mikkel Krenchel and Christian Madsbjerg: Your big data is worthless if you don’t bring it into the real world http://www.wired.com/2014/04/your-big-data-is-worthless-if-you-dont-bring-it-into-the-real-world/

Experience Sampling:

Kathy Baxter, You too can collect big data! https://www.epicpeople.org/pblog10/

Overview of programmes and apps: Experience Sampling and Ecological Momentary Assessment with Mobile Phones http://www.otago.ac.nz/psychology/otago047475.pdf

Ethno-Mining:

Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining

Mixed Methods Research Design:

Miles MB, Huberman AM (1994) Qualitative Data Analysis: An Expanded Sourcebook. Sage Publications, Inc, Thousand Oaks

Tashakkori A, Teddlie C (2009) Integrating qualitative and quantitative approaches to research. In: Bickman L, Rog DJ (eds) The SAGE handbook of social research methods, 2d ed. Sage, Los Angeles, pp 283 – 317

Teddlie C, Tashakkori A (2006) A general typology of research designs featuring mixed methods. Res Sch 13:12–28.