User research for information-rich domains Rashmi Sinha, Consultant & Researcher-at-large Uzanto...

68
User research for information- rich domains Rashmi Sinha, Consultant & Researcher-at-large Uzanto Consulting www.rashmisinha.com www.uzanto.com Leadership Seminar, IA Summit 03, Mar 21

Transcript of User research for information-rich domains Rashmi Sinha, Consultant & Researcher-at-large Uzanto...

User research for information-rich domainsRashmi Sinha, Consultant & Researcher-at-large

Uzanto Consulting www.rashmisinha.com

www.uzanto.com

AIfIA Leadership Seminar, IA Summit 03, Mar 21

Special user research methods for information-rich domains. Why?

• Such domains offer unique challenges and opportunities.– design of cell phone vs. design of navigation for online cell

phone advisor.

• Important to understand information representation in user’s minds.

• Semi-structured methods borrowed from cognitive anthropology and psychology can be of use.

Project Vision

• A portfolio of methods to probe user information representations and needs.

• Reduced time for user research. • Same data collection phase for user categorization and

persona creation. • Reuse of results.• Suggestion not prescription.• Reliable methods.

Two key questions for user research

• Part 1: User Categorization– Scope & boundaries of the information domain– Structure of information domain– Differences between groups of people (different user groups,

different cultures, stakeholders)

• Part 2: User information needs– Are some needs more important than others?– Can users be differentiated into groups on the basis of such

needs? Can this grouping be used to form personas?

Part 1: Understanding the categorical structures of the mind• Overview

– Why people categorize?– The structure of semantic memory– Is understanding user categorization

important for IA’s?• Methods

– Free-listing.– Categorization Methods such category allocation & verification.– Card Sorting and alternatives.– Analysis of sorting data.– Identifying dimensions or facets.

Why people categorize• Cognitive Economy

– Russian Psychologist Luria’s case: “Mind of a mnemonic”; Borge’s “Funius the memorius”.

• Allow generalization from past experiences– By treating something as a kind, we can use what we have

learned from other examples of that kind.• Facilitate communication

– Describing something as a kind of thing may obviate the need to provide details of the thing itself.

• Permit different levels of abstraction – difference between objects and ideas, animals and plants, dogs

and cats, and terriers and collies.

Is understanding categorization useful for Information Architects• Direct use: when user categorization informs IA. • Indirect use: good to have broad understanding how

users think about product even when user categorization does not directly inform IA.

• Context is important, getting whole scheme right more important than specific category or node.

• Important to remember: – Categorization is not static. People are good at learning new

categories. If you provide the context and the right examples, they can learn new categories or alter boundaries of old categories.

Should interfaces always reflect user categories faithfully?• No.

– Categorization is far too important to depend only on what user thinks.

– Should also be influenced by business proposition, strategy, brand etc.

• Different user groups might differ in their perception of domain. No one scheme can serve them all perfectly.

• Depends on the type of site. More important to get categorization right for an architectural image database, than a small exclusive retail store e.g., Red Envelope which can push its own vision.

• User research should provide alternatives rather than the one answer, allowing information architects the freedom to make choices.

Structure of human memory and how categorization fits in

• Implicit / Explicit distinction• Short term / working memory limitations• Categorization is implicit

• Terms: categorization and semantic memory used interchangeably

The structure of semantic memory

• Semantic distance• Hierarchical structure• Graded structure (some members are more prototypical than others)• Semantic organization builds up user expectancies regarding how

things are organized– Go along with expectancies to help users understand your IA.– Violate these expectancies and you leave the user lost and confused.

Animal(has skin, can

move, eats)Birds(wings, flies,

feathers)

Fish(swims, gills, fins)

Canary(sings, yellow)

Ostrich(tall, does not fly)

Salmon(edible, pink)

Shark(dangerous, gray)

Not all categories follow these rules: Goal directed categories

• Example: “things to wear in winter to keep warm”.• Central tendency does not play a role in the graded

structure of goal directed categories.• Graded structure is organized around ideals. For

instance, exemplars are better members of category: “things to wear in winter”. – The best example of this category (down jacket) is not the

exemplar that is most like other category members; rather, it is the exemplar with the most extreme value on the goal-related dimension.

Individual differences in categorization

• Individual differences in categorization

– Factors such as experience, learning, culture have an impact.– There is enough consistency across people for effective design.

• Degree of closeness between items might vary between people, but we mostly recognize the idiosyncratic association made by others.

Grilled Salmon

Shark

Diving

Aquarium

SharkSushi

Fish

Fish

Person 1

Person 2

Semantic distance as basis of categorization• At root of all categorization techniques is question: “How far is A

from B?”• Proximity / similarity matrix can help visualize user semantic

networks (by using cluster analysis and other statistical techniques).

Animal(has skin, can

move, eats)Birds

(wings, flies, feathers)

Fish(swims, gills, fins)

Canary(sings, yellow)

Ostrich(tall, does not fly)

Salmon(edible, pink)

Shark(dangerous, gray)

Semantic Distance can be plotted as a similarity matrix

• Also called proximity matrix.• One type of similarity matrix is a correlation matrix (look in Excel guide for instructions for

creating correlation matrix)

Self correlation

Cheese Burger

Chicken McNuggets

Fish sandwich

French Fries

Shake

Cheese Burger 7 6 2 4 5

Chicken McNuggets 6 7 4 3 2

Fish sandwich 2 4 7 3 1

French Fries 4 3 3 7 4

Shake 5 2 1 4 7

(Table 1 in handout)

Plotting similarity / card sorting data: Cluster Analysis for dataset about cars

ContinentalBuick

CadillacMercedesCorvette

JaguarFirebirdCamaro

Monte CarloCapri

ChevyVegaDart

Volkswagen

• Cluster Analysis• Advantages

– Suggests a structural solution

• Disadvantages– Prescriptive– Averages over different

dimensions

Alternative method of analysis: Multidimensional Scaling (MDS)

Advantages of Multidimensional Scaling (MDS)

• Hints at possible solutions rather than prescribes. Tells you the possibilities, leaves specifics of solution to you.

• Dimensions (axis) can suggest facets.• Similarity maps are easy to understand• Helps identify what dimensions are important

– cluster analysis can be done on results of MDS. This gives cleaner results.

• Cluster Analysis and MDS are complementary

Part 1: Understanding the categorical structures of the mind• Overview

– Why people categorize– Is understanding user categorization important for IA’s.– The structure of semantic memory

• Methods– Free-listing– Categorization methods such category allocation & verification.– Card sorting and alternatives– Analysis of sorting data– Identifying dimensions or facets

Free-listing exercise to explore domain scope and boundaries

• Goals– Explore boundaries and scope of domain across a group of people.– Gain familiarity with user vocabulary for the domain.– Use as a precursor to card-sorting, to define and limit the domain, and frame

card items in the user’s language.

• Method– Can be conducted as part of interview, or as written exercise – Ask respondent, “Name all the x's you know.” Give sufficient time to do so. – How many respondents?

• Depends on how much agreement there is about the domain. more agreement > fewer respondents.

– Stability of average rank order is a good index.

Free-listing Exercise

• Think of the menu at Wendy’s. Write down all the menu items you can think of on a piece of paper. Take one minute to do this.

•Some items start repeating.

•Some items occur earlier in the list.

Data from free-listing exercise about McDonald’s menuUser No 1French friesCheese burgerShakeHamburgerCheese burgerFrench friesChicken sandwichChicken

McnuggetsFish sandwichShakeHamburger

User No 2French friesChicken Cheese burgerShake

User No 4Chicken McnuggetsCheese burgerBacon cheese burgerFrench fries

User No 5HamburgerQuarter pounderBig macChicken fajitaFrench friesApple pie

User No 3HamburgerCheese burgerFrench friesMc ribChicken sandwich

User No 6French friesSodaBig macQuarter pounderCheese burgerHamburgerDouble cheese burgerHappy mealChicken McnuggetsMc FlurryIce creamCookies

(Table 2 in handout)

Analysis of data• Create a list of all items, sorted by their average rank (of being listed

by a respondent). Examine how that rank order changes with the addition of each new respondent. If the ranks are relatively stable, then you can stop adding new respondents.

Items Times mentioned

% of all participants

Average list rank

Cheese burger

Chicken Mcnuggets

Chicken sandwich

Fish sandwich

French fries

Shake

6

7

4

4

10

3

60%

70%

40%

40%

100%

30%

1.6 (1,1,2,1,2,3,3)

3.2 (3,3,3,4,3,3,4)

4.5 (4,5,4,5)

6 (5,6,6,7)

4.2 (6,3,4,4,2,6,7,7,4,8)

7.3 (7,4,9,9)

(Table 3 in handout)

Concept structure–Plot items according to frequency of mention

% of times items were mentioned

0

5

10

15

20

25

30

35

40

Items

Core

Middle

Periphery

•Divide items into 3 concentric circles (use your own break points):

Similarity from co-occurrence

• How many times did two items occur in the same free-list?

– Assumption of similarity if same person mentions both• Crude similarity metric.

Cheese Burger

Chicken McNuggets

Fish sandwich

French Fries

Shake

Cheese Burger 7 6 2 4 5

Chicken McNuggets 6 7 4 3 2

Fish sandwich 2 4 7 3 1

French Fries 4 3 3 7 4

Shake 5 2 1 4 7

Other uses for free-listing

• Comparing cultural or other group differences– How do two groups perceive the same domain?

• Comparing two domains– How does perception of McDonald’s menu compare with

Wendy’s?

• Segment respondents into types based on familiarity:– Find respondents with greater domain familiarity or those who

perceive domain in idiosyncratic fashion?

• Recursive free-listing to explore concept structure

Variants of free-listing: Semantic Association

• To find out what lies nearest to target items. Helpful in understanding problem categories and items.– Exploring at same semantic level, rather than going deeper.

• Semantic Association– What words do you associate these target words with

pasta: _____ _____ _____ ______ _____ _____

Part 1: Understanding the categorical structures of the mind

• Overview– Why people categorize– Is understanding user categorization important for IA’s.– The structure of semantic memory

• Methods– Free-listing– Categorization methods such category allocation & verification.– Card Sorting and alternatives– Analysis of sorting data– Identifying dimensions or facets

Category Generation

• Goal: to understand the overall categorization scheme

• Method: Open card sort – Users given items. Asked to create categories

• Options:– Provide total number of categories to be created

(avoid problems with splitters and lumpers).– Successive card sorts.– Ask for labels for each grouping.

Category Allocation• Goal: to understand goodness of existing information architecture

and labels• Method: Closed card sort

– Users given items and category labels. Asked to place each item in a category.

– Do not allow creation of a miscellaneous category.• Useful for:

– Understanding user categorizations when category labels are a given– Refining existing categorization scheme.

• Options:– Allowing items to belong to multiple categories.– Providing category descriptions rather than category labels.

Category Verification

• Goal: test goodness of an existing categorization scheme.

• Method: Speed verification tests– Create the navigation scheme (the hierarchy / facets etc). Test

this scheme by asking users to find items from various levels of the hierarchy in a short time. Note time, number of steps and mistakes. Compare these metrics with another scheme.

– Search for utility called “Classify”.

• Useful for: Comparing two designs. Testing overall goodness of an existing scheme.

Part 1: Understanding the categorical structures of the mind• Overview

– Why people categorize– Is understanding user categorization important for IA’s.– The structure of semantic memory

• Methods– Free-listing– Categorization methods such category allocation &

verification.– Card sorting and alternatives– Analysis of sorting data– Identifying dimensions or facets

Example: Card Sorting exercise for online travel guide

• Example: Designing an online travel guide to help users plan their trips to various locations.

• Purpose of card sort: – How to structure the website to help users find

travel information, and create personalized travel guides.

• Items include – lodging, entertainment, local information, When to

Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.

Alternative for open card sorting: Online survey software

• Each item (card) occupies a row• Each potential (unnamed category) is represented by a column• Disadvantage: Cumbersome with large number of categories

Open card sorts: When user generates categories

Alternative for closed card sorting

• Each item (card) occupies a row• Each category is represented by a column• This can be done on Excel as well

Closed card sorts: When user works with given categories

Simple data analysis for card sorting

• Look at data, one user at a time

• At a glance you can look down a particular column, and see what items belong in it.

• Also possible to create hierarchies based on user created categories

Advantages and disadvantages of using online survey software for card-sorting

• Advantages– Easy to carry out, no software to install and figure out. Can be

done remotely using online survey software.– Easy to get data into Excel (or your favorite application).– Can be used to conduct either open / closed card sorts– Can also be adapted to individual / group sessions

• Disadvantages– Feels less intuitive to participants– Cumbersome with large number of categories

Options with card sorting and what to from future research• Options

– Should each item be in only one category?– Should card sorts be done individually or with groups?

• group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task.

• group card sorts can overshadow individual differences, highlighting opinion of a few.

– Combining card sorting with importance ratings?

• Future– Analyze differences between user groups, different cultures etc. – Individual Differences: How much consensus is there between individuals?– Stakeholder analysis.– Reuse of domain concept maps.

Part 1: Understanding the categorical structures of the mind• Overview

– Why people categorize– Is understanding user categorization important for IA’s.– The structure of semantic memory

• Methods– Free-listing– Categorization methods such category allocation & verification.– Card sorting and alternatives– Analysis of sorting data– Identifying dimensions or facets

Identifying dimensions: Method 1, 20 questions• The user tries to guess what the item is, by asking about aspects or facets

of the object. Facilitator can only answer yes or no. (method proposed by and example from Margaret Hanley on the faceted classification list)

• Example: Identify facets for microscopy images

– Questions asked: cellular / non-cellular, animal or not– Facets

• Species• Type of electron microscope• The magnification of the microscope• The biological process being demonstrated• Part of the organism or cell being shown• The method for taking the image

Identifying dimensions: Method 2, differentiating items• Randomly select two items from your list. Ask user what differentiates

them. Repeat question with other pairs till dimensions start repeating frequently.

• What differentiates pair below. List one or more ways that they are different.

– Pasta and rice: _______ ______ ______– Plane and train journey: _______ ______

______• Good way to explore domain you are unfamiliar with. Some of the

pairs will sound ridiculous!

Exercise: Identifying dimensions• 20 questions (5-6 minutes)

– Middle person at each table, think of a favorite movie. Movie should be a well known one. Ask partners to guess by asking yes/no questions.

– Not important if they get the correct answer or not.– Identify dimensions via questions

• Pair-wise differences (5-6 minutes)– Choose one of the pairs of recipes– List differences and similarities– Identify dimensions– Does listing differences work better than listing similarities?

Identifying dimensions: Method 3, Multidimensional Scaling

• Note: add text here

Conclusion of Part 1• Overview

– Why people categorize– Is understanding user categorization

important for IA’s.– The structure of semantic memory

• Methods– Free-listing– Categorization methods such category allocation &

verification.– Card sorting and alternatives– Analysis of sorting data– Identifying dimensions or facets

Part 2: Understanding user needs, creating personas• Are some needs more important than others?• Can users be differentiated into groups on the basis of such

needs?• Can this form the basis of personas?

• Personas as “User Archetypes” who represents needs and goals of many other users.– Using “Personas” facilitates design for one specific

person.– Who should that one person be? How should that

person/s be identified?• Not the average user• Not a real user• Derived from background user research (interviews

etc.)

Current methods for persona creation

• Method– Conduct interviews with various stakeholders.– Find patterns.– Pick a nugget, interesting tidbit and build persona around it.

• Problems with method:– Interviews not economical way to find representative users.– No tight coupling between user research & personas.– Would two designers using same user research create same

personas?

• This method works well when you have time on your hands, resources to interview many people, and skilled persona developers.

Will market segmentation techniques prove useful?

• Market segmentation is used to identify clusters of people product can appeal to.– Mostly on basis of demographics.– Sometimes on the basis of psychological variables (also

called psychographic variables).

• Techniques can forecast marketplace acceptance of products and services. Can also help convince executives to build product.

• Such techniques often not helpful in defining product

Deconstructing marketing techniques

• Questions focus on like / dislike of product concept

– what do you think of vanilla coke or green Heinz ketchup?

• Statistical techniques used in market segmentation cluster users according to demographic variables, not according to user needs from products.

• Concern themselves with product at a high conceptual level. For interaction design, one needs to focus on the specifics of the interaction: How will product satisfy needs / goals.

What is needed is a method that will…

• Ground the personas in reality.• Focus on motives/needs of users.• Complementary to existing methods e.g., interview,

observation.• Concrete series of steps that any designer can follow to

develop personas.• Reliable: When used by other designers, at other times

will lead to similar personas.

Creating personas for a Bay Area restaurant finder• Goals of site: Become the premier site for Bay Area residents & visitors to find

restaurants of their choice.• Design quick restaurant-finders (a la product advisors) apart from conventional

advanced search / browse options• Method:

– Step 1: Preliminary phone interviews– Step 2: Survey of User Needs– Step 3: General data analysis– Step 4: Identify user archetypes– Step 5: Add details to personas through interviews and

observation (optional)

Step 1: Preliminary phone interviews

• Goal: Initial exploration of the domain. Identify questions that need to be asked for next stage. Get acquainted with user vocabulary for a domain.– Phone Interviews: Ask people to describe a specific restaurant

experience. Note how they chose the restaurant, what factors they indicated as important.

• (Note: related to concept of schemas and scripts from social psychology)

– Review of other similar websites to identify factors that are important.

– User free-listing?

Step 2: Online survey of user needs• Conducted survey on factors that are used in finding

restaurants.– Rated importance of features in choosing restaurants on 5-point

scale.– Food quality, décor, wine selection, cuisine, service.

• Broad and shallow look. An effort to be comprehensive.• Tie response to behavior: Ask respondents to recall a

specific incident of choosing a restaurant, rather than answer questions in an abstract fashion.

• Option: Ask about scenarios of usage

Survey

Step 3: General data analysis to broadly understand user needsLook at Table 4 from Handout

Importance of various factors• What factors are important

– Most respondents thinks quality of food, type of cuisine, cleanliness is very important

– Few respondents think valet parking is important– Mixed feelings about features such buffet, location, wine

selection, waiting in lines etc.• Who do you visit with: Most report going to restaurants

with friends, or as a couple. Fewer with family. Even fewer alone.

• Frequency of visiting: Most report going more than once a week.

Relationship between restaurant factors

• Do some factors go together?• Exercise:

– Correlation Matrix with numbers– Correlation Matrix with colors

• Examine some of the relationships

Examining relationships through a numerical correlation matrix 

décor valet vegan wine selection

buffet child friendly

reservation

outdoor groups crowd romantic

décor

valet 0.34

vegan 0.33 0.33

wine selection 0.28 0.00 -0.11

buffet 0.24 0.57 0.32 -0.03

child 0.00 0.07 0.03 -0.05 0.22

reservation 0.39 0.30 0.07 0.34 0.09 -0.08

outdoor 0.32 0.28 0.06 0.35 0.22 -0.21 0.36

groups 0.19 0.27 0.15 0.19 0.39 0.22 0.20 0.14

crowd 0.45 0.29 0.14 0.13 0.33 -0.11 0.23 0.52 0.25

romantic 0.60 0.29 0.25 0.29 0.10 0.03 0.32 0.16 0.22 0.36

relaxed service 0.52 0.24 0.20 0.24 0.34 0.06 0.39 0.16 0.41 0.37 0.30

trendy 0.38 0.51 0.36 0.24 0.08 0.18 0.20 0.09 0.17 0.11 0.41

conver-sation 0.31 0.17 0.10 0.17 0.27 -0.19 0.39 0.42 0.08 0.32 0.32

bar 0.06 0.01 -0.02 0.38 -0.06 -0.26 0.23 0.41 -0.01 0.29 0.28

Examining relationships through a shaded correlation matrix

décor valet vegan wine selection

buffet child friendly

reservation

outdoor groups crowd romantic relaxed service

décor

valet

vegan

wine selection

buffet

child

reservation

outdoor

groups

crowd

romantic

relaxed service

trendy

conver-sation

bar

Step 4: Identifying user archetypes

• Features that are highly related to each other will form part of the same constellation, or group of needs

• Factor analysis (Principal Components analysis) techniques can identify such groupings.

• Each grouping can be said to represent a need type.• These are groupings of features not of users.

Exercise: From user archetypes to personas (10 minutes)

• Look at page showing user archetypes• Each archetype shows a bunch of factors that formed a

group through Principal Components Analysis• Examine each. Give a name/title to each• Choose one that seems suitable for personas

– Fill out more details for those

• Some archetypes seem similar (Archetype 1 and 2).

User Archetype 4: Lin looking for a romantic, relaxed, dining experience…• Important Features

– Good decor– Romantic setting– Relaxed service– Trendy– Caters to special Requirements (Low-fat, vegan etc.)– Bar

• Not Important Features– price, buffet-style– Spend more than $20, go out as a couple

• Date?• Food Quality, Type of Cuisine & Cleanliness does not appear in these

archetypes because it was important to everyone.

User Archetype 3: Chris, buffet style dining, with kids in tow…

• Important Features– Buffet style dining – Good for groups– Child Friendly– Relaxed Service– Valet

• Not Important Features– views, outdoor dining, people- watching, bar

• Many in age group 32-41, dine mostly with family, once a week or lesser

User Archetype 1: Jen, meet friends for drinks, eat some food too…• Important Features

– Outdoor seating– People Watching– Great view– Well stocked bar– Crowd Type– Valet – Conversation– Good for groups

• Not Important Features– long Lines, finding Parking nearby

After establishing some user archetypes

• Use user archetypes as “candidate personas”• Guided interviews and observations: of users who can

represent various archetypes• Generate sample scenarios, make final decision on

primary and secondary personas according to your design goals

• Do not treat user archetypes generated through survey as final word. Use them in conjunction with other information.

Advantages of technique

• Inexpensive in terms of time and expertise.– Statistical analysis requires SPSS, or SAS or Matlab.

• A reliable method, two designers using same method are likely to get similar results.

• Easier to get buy-in from management since personas are directly derived from user research.

• Retains creativity and fictional elements of creating personas, while grounding them in real data.

Other uses• Currently used personas (no matter how the personas were created) can

be verified for their accuracy. Use the correlation matrix to do so.– Personas make some key assumptions regarding what information needs go

together.– Conduct survey. Verify those individual relationships through correlation

matrix.

• Options– Identifying extreme cases– Or normative cases

• Frame survey in terms of scenarios.

Review of talk

• Part 1: User Categorization– Scope & boundaries of the information domain– Structure of information domain– Differences between groups of people (different user groups,

different cultures, stakeholders)

• Part 2: User information needs– Are some needs more important than others?– Can users be differentiated into groups on the basis of such

needs? Can this grouping be used to form personas?

Project Vision• Semi structured data collection

– Retain the flexibility of open-ended methods. – Some structure allows for deeper probing, easy summaries– Can find patterns in data that cannot be explored with the naked eye.

• Easy returns (looking at data summaries, correlations)– Sophisticated answers with more complex analysis

• Suggestion rather than prescription– methods that provide hints rather than one “right answer”

• Flexible methods– Can be used for both design from ground up or to improve existing

design (e.g., category verification, persona verification)• Reuse of results

– General domain specific concept maps– IA adapts for their own project

• Reliable methods.

Thanks!

• For more information– www.rashmisinha.com– www.uzanto.com

– Email: [email protected] – Phone: 510-703-9595