Information Architecture Summit 2004 Uzanto Consulting 1 Rapid User Mental Modeling (RUMM) at...

Information Architecture Summit 2004 Uzanto Consulting 1

Rapid User Mental Modeling (RUMM)at eBay.com: A Case Study

Jonathan BoutelleRashmi Sinha

& Kirsten Swearingen

Uzanto Consulting

Larry Cornett&

Ido Dan

eBay, Inc.


Background to Information Architecture Redesign of eBay.com eBay website had grown in organic fashion, leading to a

somewhat inconsistent user experience Although much design and usability work is done on site itself,

this project was first effort to delineate user mental models for the entire website.

Basic eBay facts World’s largest online marketplace 16 million listings on any given day (as of Q2, 2003) Almost 34 million active users worldwide

Buyers and sellers New and experienced Occasional and full-time Individuals and corporations Different countries and cultures


Challenges

New IA would need to: Support site’s vision of global economic democracy Serve a large and diverse user population with varying goals and

mental models Be future-oriented, allowing for planned change and growth Occur through evolution not revolution -- must accommodate

current users who are very used to the site as it exists today Senior management at eBay needed to see proof of the

need for a drastic restructuring. First phase of project (background research) to be

completed in only 6 weeks


Overview of Design Research Methodology

Stage 1: Explore

Interviews and free-listing with users and stakeholders

Stage 3: Verify / Refine

Large-sample closed card-sorting study

Rapid User Mental Modeling

Stakeholders

User mental models

Preliminary IA

Stage 2: Understand

Open card-sorting study

site.com

TopCategory 3

TopCategory 1

TopCategory 2

Sub cat1

SubCat2

SubCat3

xx

SubCat1

Subcat4

Subcat3

Subcat2

xx

xx

xxx

SubCat3

SubCat5

SubCat4

SubCat6

SubCat4

xx

xxx

xx

xxx

xxx

xxx

SubCat4

SubCat5

SubCat1

SubCat2

xxx

xx

xx

Verified categorization

Top-level

categories

Scope of content &

functionality


Our Approach to Design Research: Rapid User Mental Modeling (RUMM) Rigorous, three-stage design research methodology to lay

foundation for IA redesign Bring together established techniques in a novel way

Three stages: Exploration Understanding Verification Individual components complement and build upon each other

Process moves from open-ended exploration to focused information-gathering and hypothesis testing

Addresses individual and group differences in a systematic way Produces reliable results (can be reproduced) Uses mix of qualitative and quantitative methods

Qualitative: rich, open-ended understanding of users and domain Quantitative: generates recommendations firmly grounded in data (can

help make case to management)


Towards Persuasive, Defensible Design Research Important to create not just good design, but

successful design: Serves user Satisfies the business goals Is accepted by management

Results of design research phase must provide rich picture of user, but also be persuasive and defensible. Defensible: firmly grounded in data, can stand up to

scrutiny Persuasive: persuades stakeholders that your solution

makes sense and will provide benefit to the business


Stage 1: Explore Domain Through Stakeholder Interviews Goals:

Explore domain in open-ended fashion Identify both current and planned content and

functionality Method:

Interviews with eBay users about specific incidents of site usage

Interviews with stakeholders: top management, members of design team etc.

Identify problems, constraints, and specific plans for future

Generate list of current tasks and “Horizon Tasks” (tasks planned for future)

Stakeholder analysis Task list development


Stakeholder Interviews

Spoke to 16 eBay managers and designers Asked 3 questions:

What do you imagine a user doing on the eBay site two to five years from now that they can’t do today?

What one problem, if it were solved today, would cause a large increase in the number of transactions executed on the eBay platform?

What user population is least well served by the current site?


Purpose of Stakeholder Analysis

Gain valuable perspectives for design research: Insure that design aligns with business strategy. Tap into a company’s existing knowledge in a

systematic fashion. Ensure success of project by getting input at the

research phase from those who have ability to say yes or no.

Helps stakeholders feel involved in and understand the design research and process.

Generate task list: helps map the scope and boundaries of the information domain


Generating the Task List Using Free-Listing Decided not to conduct inventory of site

Incredibly time-consuming Might tie new IA too closely to current site’s design

Focused on tasks, since all eBay content is in context of completing tasks:

Pay for an item List an item

Generated initial list of 300 tasks by using free-listing exercises with:

Users Designers Stakeholders (especially important for identifying future tasks)


The Free-Listing Exercise This simple, semi-structured method helps:

Determine the scope of a domain Gain some insight into how the domain is structured

Example: “List all the animals you can think of” Results:

Indices of salience: Frequency Rank

Indirect indices of similarity: Co-occurrence. How often are cat and dog both mentioned? Average rank distance. Are cat and dog mentioned together

more closely than cat and horse? In this case, we asked “List all the tasks at eBay.com a

person can do / might be able to do in the future.”


Refining the Task List

Consolidated task list, removing redundant items Divided tasks into groups:

Current Tasks (can be done on existing eBay website): Core Tasks form core functionality of website Boundary Tasks mark boundary of functionality offered

Horizon Tasks Might be done on future versions of website Important for creating a future-oriented IA

Selected representative items from each group Result: comprehensive list of 100 tasks marking scope

and boundary of what users can accomplish on the eBay site.

We were ready to begin studying user mental models!


Stage 2: Understand User Mental Models

Goals: Gain further insight into user mental models and identify top-level categories for site content and functionality

Method: open card sorting Used list of 100 tasks (including 20 Horizon

Tasks) generated in previous stage. 28 participants, representing the main types of

eBay users (sellers and buyers; new and advanced users).

22 completed card-sorts individually 6 worked as a group to create a structure (mix of

buyers, sellers, new, and advanced)


Identifying the Top Levels of the Hierarchy Cluster analysis used to generate

aggregate mental maps. These were directly used to create

the hierarchical structure. Looked at mental maps for four

major segments. Identified the inconsistencies between

the mental maps of different groups. Understood how to reconcile those to

create a new map. Result: preliminary IA, including 5

top-level categories that accounted for all site content and functionality.

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help

Fn cusTrk SupportReq. tech. support

Read eBay guidance

See eBay overview

Get eBay appsRefer friend

Ask experts

Ask other members

Fn user groupsRead eBay msgs

View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image

Add res./BIN priceRevise listings

Create listing

Answer question

Send invoiceManage inventory

List regionally

List internationally

List item to donateCross-sell items

Manage eBay store

Trk shipments

IntLrn customsEsTrk shipping

Reg. as buyerReg. as seller

Report bad itemsReport accTrk theft

Report NPB/fraudRsrch. promotions

Rsrch. items sold

Rsrch. for listing

Rsrch. categoriesLrn manage bus.

Lrn seller benefits

Get selling tools

Be Power SellerFn sales by member

Fn items by bidderFn info. on eBay-OK code

Rsrch. partneringRsrch. eBay services

Be Trad. AssTrk / consultSee aucTrk cancels.

See bid retractions

View id history

Fn eBay memberMember contact info.

Lrn manage rep.Read feedback

Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price

Offer for unsold item

EsTrk car payment

Get buying tools

Contact sellerContact buyer

Fn accessoriesFn items by seller

Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items

Fn proFn servicesView item desc.

Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task XTask B

Task ZTask N

Dendrogram- User group !

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help


Read eBay guidance

See eBay overview


Ask experts

Ask other members


View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image


Create listing

Answer question


List regionally



Manage eBay store

Trk shipments





Rsrch. items sold

Rsrch. for listing


Lrn seller benefits

Get selling tools





See bid retractions

View id history



Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price


EsTrk car payment

Get buying tools



Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items


Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task XTask B

Task ZTask N

0.000

0.200

0.400

0.600

0.800

1.000

0.000

0.200

0.400

0.600

0.800

1.000

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help


Read eBay guidance

See eBay overview


Ask experts

Ask other members


View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image


Create listing

Answer question


List regionally



Manage eBay store

Trk shipments





Rsrch. items sold

Rsrch. for listing


Lrn seller benefits

Get selling tools





See bid retractions

View id history



Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price


EsTrk car payment

Get buying tools



Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items


Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task XTask B

Task ZTask N

See eBay time

Fn eBay help


Read eBay guidance

See eBay overview


Ask experts

Ask other members


View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image


Create listing

Answer question


List regionally



Manage eBay store

Trk shipments





Rsrch. items sold

Rsrch. for listing


Lrn seller benefits

Get selling tools





See bid retractions

View id history



Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price


EsTrk car payment

Get buying tools



Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items


Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task XTask B

Task ZTask N

Dendrogram- User group !

Aggregate maps from cluster analysis


Stage 3: Verify and Refine 5-Category Scheme Goals:

Verify that the five top-level groupings cover all 100 tasks/concepts.

Identify tasks/concepts that do not fit in, as well as where users do expect to find such information.

Examine differences in groupings among various user groups.

Further develop IA for eBay site. Establish method for future verification and refinement

of eBay IA.


Stage 3: Verify and Refine 5-Category Scheme Method: closed card sorting

Large sample study with more than 1,000 participants representing all segments of eBay users

Study conducted online. Participants asked to categorize tasks to one of the five

top-level categories (or to select Other if it didn’t belong)

Result: site-map showing the structure for the main parts of the eBay site.


Analysis of Closed Card Sort Results

Category-belonging scores: Percent of users who think a task belongs to a particular category Lack of consensus or high scores in “Other” category may signal a

design issue

Category Memberships across all participants

Task NCategory 1

Category 2

Category 3

Category 4

Category 5

Category 6

Task 1 870 48% 7% 21% 14% 3% 6%

Task 2 991 8% 1% 85% 1% 2% 3%

Task 3 959 7% 0% 86% 2% 1% 4%

Task 4 918 0% 0% 87% 3% 6% 4%

Task 5 929 0% 2% 83% 6% 6% 2%

Task 6 896 46% 5% 7% 10% 27% 5%

Task 7 908 55% 4% 15% 9% 11% 6%

Task 8 893 35% 2% 49% 1% 1% 12%

Task 9 870 1% 1% 67% 1% 3% 27%

Task 10 993 76% 1% 19% 0% 1% 4%

A color-coded visualization of closed card sorting data


Reconciling Different Mental Models

Results suggest structural possibilities--they do not prescribe one solution

In both open and closed card sorting, found differences in mental models Reconcile through design, if possible (revisit

dendograms) The most important, contentious issues may ultimately

require a decision driven by company brand and business goals (revisit stakeholder analysis)


Project Summary

Three-stage RUMM methodology used to: Understand business context Delineate hierarchical structure of

user mental models Final deliverable: site-map

showing structure for main parts of entire eBay website. This map is currently being used to

guide the stages of the IA redesign.

site.com

TopCategory 3

TopCategory1

TopCategory 2

TopCategory 4

SubCat1Sub cat1

SubCat2

xx

SubCat3

xx

xx

SubCat2

SubCat3

SubCat5

xx

xx

xx

xx

SubCat1

Subcat4

Subcat3

Subcat2

xx

xx

xxx

SubCat3

SubCat5

SubCat4

SubCat6

SubCat4

xxx

xx

xx

xxx

xx

xxx

xxx

xxx

SubCat7

SubCat4

SubCat5

SubCat6

SubCat1

SubCat2

xx

xx

xxx

xx

xxx

xx

xx

SubCat4

xx

Preliminary IA


Impact of Project Helped convince management that current site did not

match user mental models and that IA redesign was needed.

Provided basis for a long-term redesign plan to implement a structured site-wide model.

Provided immediate guidance to near-term projects in order to begin moving the site in the long-term direction.

Broadened eBay designers’ perspective on overall design direction and goals by providing:

Access to user mental models Site-map for all content and functionality of eBay.com

Cleared the road towards a more consistent and unified user experience.

Information Architecture Summit 2004 Uzanto Consulting 1 Rapid User Mental Modeling (RUMM) at...

Documents

Transcript of Information Architecture Summit 2004 Uzanto Consulting 1 Rapid User Mental Modeling (RUMM) at...