RUMM Handout or Final Paper
Transcript of RUMM Handout or Final Paper
-
8/8/2019 RUMM Handout or Final Paper
1/20
Information Architecture Summit 2004 Uzanto Consulting1
Rapid User Mental Modeling (RUMM)
at eBay.com: A Case Study
Jonathan Boutelle
Rashmi Sinha
& Kirsten Swearingen
Uzanto Consulting
Larry Cornett
&
Ido Dan
eBay, Inc.
-
8/8/2019 RUMM Handout or Final Paper
2/20
Information Architecture Summit 2004 Uzanto Consulting2
Background to Information Architecture Redesign of
eBay.com
eBay website had grown in organic fashion, leading to asomewhat inconsistent user experience
Although much design and usability work is done on site itself,this project was first effort to delineate user mental models forthe entire website.
Basic eBay facts Worlds largest online marketplace
16 million listings on any given day (as of Q2, 2003)
Almost 34 million active users worldwide
Buyers and sellers
New and experienced Occasional and full-time
Individuals and corporations
Different countries and cultures
-
8/8/2019 RUMM Handout or Final Paper
3/20
Information Architecture Summit 2004 Uzanto Consulting3
Challenges
New IA would need to: Support sites vision of global economic democracy
Serve a large and diverse user population with varying goals and
mental models
Be future-oriented, allowing for planned change and growth
Occur through evolution not revolution -- must accommodatecurrent users who are very used to the site as it exists today
Senior management at eBay needed to see proof of the
need for a drastic restructuring.
First phase of project (background research) to becompleted in only 6 weeks
-
8/8/2019 RUMM Handout or Final Paper
4/20
Information Architecture Summit 2004 Uzanto Consulting4
Overview of Design Research Methodology
Stage 1: Explore
Interviews and free-listing with
users and stakeholders
Stage 3: Verify / Refine
Large-sample closed
card-sorting study
Rapid UserMental Modeling
Stakeholders
User mental models
Preliminary IA
Stage 2: Understand
Open card-sorting study
site.com
TopCategory 3
TopCategory 1
TopCategory 2
Sub cat1
SubCat2
SubCat3
xx
SubCat1
Subcat4
Subcat3
Subcat2
xx
xx
xxx
SubCat3
SubCat5
SubCat4
SubCat6
SubCat4
xx
xxx
xx
xxx
xxx
xxx
SubCat4
SubCat5
SubCat1
SubCat2
xxx
xx
xx
Verified categorization
Scope of content& functionality
-
8/8/2019 RUMM Handout or Final Paper
5/20
Information Architecture Summit 2004 Uzanto Consulting5
Our Approach to Design Research: Rapid User
Mental Modeling (RUMM)
Rigorous, three-stage design research methodology to layfoundation for IA redesign
Bring together established techniques in a novel way
Three stages: Exploration Understanding Verification
Individual components complement and build upon each other
Process moves from open-ended exploration to focused
information-gathering and hypothesis testing
Addresses individual and group differences in a systematic way
Produces reliable results (can be reproduced)
Uses mix of qualitative and quantitative methods
Qualitative: rich, open-ended understanding of users and domain
Quantitative: generates recommendations firmly grounded in data
(can help make case to management)
-
8/8/2019 RUMM Handout or Final Paper
6/20
Information Architecture Summit 2004 Uzanto Consulting6
Towards Persuasive, Defensible Design
Research Important to create not just good design, butsuccessful design:
Serves user
Satisfies the business goals
Is accepted by management Results of design research phase must provide
rich picture of user, but also be persuasive anddefensible. Defensible: firmly grounded in data, can stand up to
scrutiny
Persuasive: persuades stakeholders that your solutionmakes sense and will provide benefit to the business
-
8/8/2019 RUMM Handout or Final Paper
7/20
Information Architecture Summit 2004 Uzanto Consulting7
Stage 1: Explore Domain Through Stakeholder
Interviews
Goals: Explore domain in open-ended fashion
Identify both current and planned content andfunctionality
Method: Interviews with eBay users about specific
incidents of site usage
Interviews with stakeholders: top management,members of design team etc.
Identify problems, constraints, and specific plansfor future
Generate list of current tasks and Horizon Tasks(tasks planned for future)
Stakeholder analysis
Task list development
-
8/8/2019 RUMM Handout or Final Paper
8/20
Information Architecture Summit 2004 Uzanto Consulting8
Stakeholder Interviews
Spoke to 16 eBay managers and designers Asked 3 questions:
What do you imagine a user doing on the eBay site two
to five years from now that they cant do today?
What one problem, if it were solved today, would causea large increase in the number of transactions executed
on the eBay platform?
What user population is least well served by the current
site?
-
8/8/2019 RUMM Handout or Final Paper
9/20
Information Architecture Summit 2004 Uzanto Consulting9
Purpose of Stakeholder Analysis
Gain valuableperspectives for design research: Insure that design aligns with business strategy.
Tap into a companys existing knowledge in a
systematic fashion.
Ensure success of project by getting input at theresearch phase from those who have ability to say yes
or no.
Helps stakeholders feel involved in and understand the design
research and process.
Generate task list: helps map the scope and
boundaries of the information domain
-
8/8/2019 RUMM Handout or Final Paper
10/20
Information Architecture Summit 2004 Uzanto Consulting10
Generating the Task List Using Free-Listing
Decided not to conduct inventory of site Incredibly time-consuming
Might tie new IA too closely to current sites design
Focused on tasks, since all eBay content is in context of
completing tasks:
Pay for an item
List an item
Generated initial list of 300 tasks by using free-listing
exercises with:
Users Designers
Stakeholders (especially important for identifying future tasks)
-
8/8/2019 RUMM Handout or Final Paper
11/20
Information Architecture Summit 2004 Uzanto Consulting11
The Free-Listing Exercise
This simple, semi-structured method helps: Determine the scope of a domain
Gain some insight into how the domain is structured
Example: List all the animals you can think of
Results: Indices of salience:
Frequency
Rank
Indirect indices of similarity:
Co-occurrence. How often are cat and dog both mentioned?
Average rank distance. Are cat and dog mentioned togethermore closely than cat and horse?
In this case, we asked List all the tasks at eBay.com aperson can do / might be able to do in the future.
-
8/8/2019 RUMM Handout or Final Paper
12/20
Information Architecture Summit 2004 Uzanto Consulting12
Refining the Task List
Consolidated task list, removing redundant items Divided tasks into groups:
Current Tasks (can be done on existing eBay website):
Core Tasks form core functionality of website
Boundary Tasks mark boundary of functionality offered
Horizon Tasks
Might be done on future versions of website
Important for creating a future-oriented IA
Selected representative items from each group
Result: comprehensive list of 100 tasks marking scopeand boundary of what users can accomplish on the eBay
site.
We were ready to begin studying user mental models!
-
8/8/2019 RUMM Handout or Final Paper
13/20
Information Architecture Summit 2004 Uzanto Consulting13
Stage 2: Understand User Mental Models
Goals: Gain further insight into usermental models and identify top-level
categories for site content and
functionality
Method: open card sorting
Used list of 100 tasks (including 20 Horizon
Tasks) generated in previous stage.
28 participants, representing the main types of
eBay users (sellers and buyers; new and
advanced users).
22 completed card-sorts individually
6 worked as a group to create a structure (mix of
buyers, sellers, new, and advanced)
-
8/8/2019 RUMM Handout or Final Paper
14/20
Information Architecture Summit 2004 Uzanto Consulting14
Identifying the Top Levels of the Hierarchy
Cluster analysis used to generateaggregate mental maps.
These were directly used to create
the hierarchical structure.
Looked at mental maps for fourmajor segments.
Identified the inconsistencies between
the mental maps of different groups.
Understood how to reconcile those to
create a new map. Result: preliminary IA, including 5
top-level categories that accounted
for all site content and functionality.
Bu d i ita l
Bid promo. option
Reverse auction
Negotiate price
Offer for unsold item
EsTrk car payment
Get buying tools
Contact seller
Contact buyer
Fn accessories
Fn items by seller
Search & browse
Fn BIN items
Fn items nearby
Fn specific item
Fn types of items
Browse items
Fn proFn services
View item desc.
Translate listingFn eBay internatLrn
Save items
Task Y
Task U
Task O
Task F
Task R
Task S
Task D
Task L
Task M
Task A
Task K
Task Ge
Task PTask X
Task B
Task Z
Task N
Dendrogram- User group !
Bu d i ita l
Bid promo. option
Reverse auction
Negotiate price
Offer for unsold item
EsTrk car payment
Get buying tools
Contact seller
Contact buyer
Fn accessories
Fn items by seller
Search & browse
Fn BIN items
Fn items nearby
Fn specific item
Fn types of items
Browse items
Fn proFn services
View item desc.
Translate listingFn eBay internatLrn
Save items
Task Y
Task U
Task O
Task F
Task R
Task S
Task D
Task L
Task M
Task A
Task K
Task Ge
Task PTask X
Task B
Task Z
Task N
Bu d i ita l
Bid promo. option
Reverse auction
Negotiate price
Offer for unsold item
EsTrk car payment
Get buying tools
Contact seller
Contact buyer
Fn accessories
Fn items by seller
Search & browse
Fn BIN items
Fn items nearby
Fn specific item
Fn types of items
Browse items
Fn proFn services
View item desc.
Translate listingFn eBay internatLrn
Save items
Task Y
Task U
Task O
Task F
Task R
Task S
Task D
Task L
Task M
Task A
Task K
Task Ge
Task PTask X
Task B
Task Z
Task N
Bu d i ita l
Bid promo. option
Reverse auction
Negotiate price
Offer for unsold item
EsTrk car payment
Get buying tools
Contact seller
Contact buyer
Fn accessories
Fn items by seller
Search & browse
Fn BIN items
Fn items nearby
Fn specific item
Fn types of items
Browse items
Fn proFn services
View item desc.
Translate listingFn eBay internatLrn
Save items
Task Y
Task U
Task O
Task F
Task R
Task S
Task D
Task L
Task M
Task A
Task K
Task Ge
Task PTask X
Task B
Task Z
Task N
Dendrogram- User group !
Aggregate maps fromcluster analysis
-
8/8/2019 RUMM Handout or Final Paper
15/20
Information Architecture Summit 2004 Uzanto Consulting15
Stage 3: Verify and Refine 5-Category
Scheme Goals:
Verify that the five top-level groupings cover all 100
tasks/concepts.
Identify tasks/concepts that do not fit in, as well as
where users do expect to find such information. Examine differences in groupings among various user
groups.
Further develop IA for eBay site.
Establish method for future verification and refinementof eBay IA.
-
8/8/2019 RUMM Handout or Final Paper
16/20
Information Architecture Summit 2004 Uzanto Consulting16
Stage 3: Verify and Refine 5-Category
Scheme Method: closed card sorting
Large sample study with more than 1,000 participants
representing all segments of eBay users
Study conducted online.
Participants asked to categorize tasks to one of the fivetop-level categories (or to select Other if it didnt
belong)
Result: site-map showing the structure for the
main parts of the eBay site.
-
8/8/2019 RUMM Handout or Final Paper
17/20
Information Architecture Summit 2004 Uzanto Consulting17
Analysis of Closed Card Sort Results
Category-belonging scores: Percent of users who think a task belongs to a particular category
Lack of consensus or high scores in Other category may signal a
design issue
Category Memberships across all participants
Task N
Category
1
Category
2
Category
3
Category
4
Category
5
Category
6
Task 1 870 48% 7% 21% 14% 3% 6%
Task 2 991 8% 1% 85% 1% 2% 3%
Task 3 959 7% 0% 86% 2% 1% 4%
Task 4 918 0% 0% 87% 3% 6% 4%
Task 5 929 0% 2% 83% 6% 6% 2%
Task 6 896 46% 5% 7% 10% 27% 5%
Task 7 908 55% 4% 15% 9% 11% 6%
Task 8 893 35% 2% 49% 1% 1% 12%
Task 9 870 1% 1% 67% 1% 3% 27%
Task 10 993 76% 1% 19% 0% 1% 4%
A color-coded visualization of closed card sorting data
-
8/8/2019 RUMM Handout or Final Paper
18/20
Information Architecture Summit 2004 Uzanto Consulting18
Reconciling Different Mental Models
Results suggeststructural possibilities--they donotprescribe one solution
In both open and closed card sorting, found
differences in mental models
Reconcile through design, if possible (revisitdendograms)
The most important, contentious issues may ultimately
require a decision driven by company brand and
business goals (revisit stakeholder analysis)
-
8/8/2019 RUMM Handout or Final Paper
19/20
-
8/8/2019 RUMM Handout or Final Paper
20/20
Information Architecture Summit 2004 Uzanto Consulting20
Impact of Project
Helped convince management that current site did notmatch user mental models and that IA redesign wasneeded.
Provided basis for a long-term redesign plan to implementa structured site-wide model.
Provided immediate guidance to near-term projects inorder to begin moving the site in the long-term direction.
Broadened eBay designers perspective on overall designdirection and goals by providing: Access to user mental models
Site-map for all content and functionality of eBay.com
Cleared the road towards a more consistent and unifieduser experience.