Measuring UX by Tim Bosenick, SirValUse

Bild

Measuring User ExperienceCombining the “Qualitative” with the “Quantitative”Some Examples from the Web

Tim Bosenick, SirValUse Consulting GmbH

What do I know about my site and the users?

I know the serverlogs, PIs and

sessions.

Sometimes a apply surveys

on my site.

I have a Web Analytics System and collect loads

of data.

But...

I have some data of users given during

order.I know that some users order.

I run regularely qualitative User

Experience Tests.

But still there are many questions…

Who exactly are my users? Do they act effectively on my

website?

Are my competitos better? Why?(scope, CR, ..)

How does the business

competition develop?

Does my marketing reach the people I want

to reach?

Where is my target group on the way in internet?

Where can I appeal to them?

Who are my most important

competitors?Do I reach my target groups?

Conclusion:

We need a 360 degree view on our costumers

User experience, usability and marketing (the whole “experience chain”) come

closer together

We often know the “what happens”, we sometimes know the

“why” – a combination would be great

The Idea

5WebWerte

Combining qualitative and quantitative observational and survey data.

Further requirements:

» Single source data collection for high validity

» Scalable to large numbers of participants

» Possibility of true-intent and task-based user experience tests, also experiments

» Measurement without cooperation of website owners or publishers(only Add-On)

StandardisedSurveys

FreeUser Feedback

User TrackingDetailed

analysis of user sessions

Quantitative Qualitative

Observation

Survey Data

for one specific website,site centric,

short time frame

Whole internet, user centricFeatures of the LEOtrace® Browser Add-Ons

Features:

» Single source measurement of reactive and non-reactive data

» Data collection without cooperation of website owners or publishers

» Allows data collection on third party sites

» Flexible setup» Customising of design and branding

» Remote control of all functions

» Extended research designs possible:» Task-based designs

» Experimental designs (manipulation of client-side HTML code)

» Data collection in SSL-encrypted areas possible (usually disabled)

Deliverable data:

» Non-reactive data ("Tracking")» Session information (e.g. duration)

» Visited URL’s

» Search queries (Google, ...)

» Precise ad impressions

» Screenshots

» Clicks (and mouse movements)

» Reactive data» Standardised surveys (also event-

triggered) with any survey software

» Free ad-hoc feedback

» Client information» Operating system, web browser, …

» User structure data provided from access panel

Whole internet, user centric Installation process

1DownloadAdd-On

2InstallAdd-On

Easy setup – justlike any other

browser add-on 3

Optional:Registration onfirst browser start

4

Optional: Identificationon every

browser start

Specific website, site centricThe user are invited on your website …

» To elevate the problems occuring while using the website,the LEOtrace® Remote Test has been developed:

… and use the website according to your natural performance.

Invitation by a layerPreliminary interviewOnline questionnaire

• "What is your today main reason of visit?"

Follow-Up InterviewOnline questionnaire

• "Did you achieve everything today you planned to?"

• "Please evaluate the website on the basis of following items."• User-Behaviour

• Clickway-Analysis

• User-Feedback

Remote Session

• "Please surf the website as you would do it normally."

Participants and website-providers don´t need to install software .Only the Invitationlayer has to be linked. Adjustments have only to be neccessary in the LEOtrace®-System itself

Specific website, site centric The Technology

Usability Expert

Recalls monitored actions of participants

Surveyserver

Give free and scaled feedback

Utility target-website

Recalls comments and evaluations of participants

Website

Content enquieries through

Participants

Proxy-Server

3

2 1

Via feedback-buttons the users are always able to call up a short survey or to give a positive or negative feedback .

Specific website, site centric The Feedback-Bar

To the finaly surveyCall up feedback-

sessions

Types of Studies

12WebWerte

Typical StudiesTasked based Benchmarking

Recruitmentfrom online access panel

(screening according totarget group criteria);

Installation of LEOtrace®Browser-Add-On

Initial survey(e.g. favourite search

engines, …)

Taskwith detailed recording of

user behaviour (clicks,screenshots) as well as free

feedback and event-triggeredquestionnaires

Final survey(e.g. rating of websites used);

Deinstallation of Add-On

1

2 3

4

Typical StudiesTasked based Benchmarking & Usage Monitoring





engines, …)

Taskwith detailed recording of

user behaviour (clicks,screenshots) as well as free

feedback and event-triggeredquestionnaires



1

2 3

4

Non-reactivemonitoring of online usage

during several weeks(before receiving task)

3a

Typical StudiesDigital Behaviour Studies





engines, …)



1

2

4



3

Typical StudiesDigital Behavior Studies with Event based Surveys





engines, …)



1

2

4



3

Event based surveys,e.g. having used a

certain function,leaving a certain website,

leaving a certain page

Typical StudiesWeb Efficiency Panel

Internet userin WEP

Browser Add-On sendsURLs

Ad contactsSearch queries

Internet user buys online

Panellist scans purchases

Fusion & analysis

Offliner buys offline

Internet user buys offline

Offlinerin GfK Consumer Scan panel

Generating single source consumer & Internet usage data:

» Purchases(Consumer Scan)

» Internet usage PageImpressions

Visits

AdImpressions

AdClicks

Queries with relevant search engines

» Structural data

Typical StudiesIndustry Benchmarking

» We use the data from the Web Efficiency Panel to calculate benchmarking KPIs for certain industries (e.g. e-commerce, automobile, …)

» Possible analyses: Target group

Cross usage

Previous and following websites

Usage of functions / areas

Conversion rates

…

Typical StudiesTrue Intent Experience Reports

Recruitmentonline (real users)


no installation needed

Initial survey(e.g. reason for

the visit, …)

Final survey(e.g. rating of website, satisfaction, reason for

leaving, …)

1

2

4

Non-reactivemonitoring of website usage

during this sessionoptional: free feedback

3

The Importance of Measurements

Quantitative studies are an ideal enhancement of qualitative studies

Qualitative Approach Quantitative Approach

Focus on “usability” Focus on “user experience”

Derive recommendations Measure key performance indicators

Focus on product management / UX departments

Focus on market research / general management

Sometimes “artificial” lab situation, no “real” behavior can be observed

“True intent” studies possible, observation of “real behavior”

Combination with qualitative marketing research possible

Combination with marketing KPIs possible

Testing of “offline” applications possible

Test object must be “online”

Why is quantitative measurement important?

» Users tend to misjudge their behaviour Social request, known brands are over-rated, problem to correctly quantify their own behavior

Example: User were asked: What kind of websites did you use while looking for a product? Answer: 88% Google. In the actual measurement phase, only 68% really used Google.

» “User Experience” gets more and more important “Usability” still is an important factor of the overall user experience, but e.g. “joy of use” and

“design” come into the focus of testing – also in the early development phase.

With this, classical market research methodologies and “real measurement” become central for the usability testing practice – or else market research companies will gain bigger parts of the market.

» “Usability” and Marketing come closer together It is nowadays not only important to get as much traffic as possible to a website (online

marketing) and to optimize the conversion rate (usability), it is also important to combine both views so that the whole shopper experience can be optimized.

Automobile Industry Monitor 2009

23WebWerte

Methods

Internet Tracking

Surveys» Surveys completed by the WEP panelists regarding the

topics automotive ownership, automotive purchase, brand affinity and advertising awareness.

To satisfactorily address the questions at hand, we have employed all of the following methods:

» Permanent data collection regarding all surfing behaviour by means of the Web Efficiency Panel (WEP) designed by GfK during Q1 2009 (01/01 - 31/03).

» Quantitative evaluation of internet use with regard to relevant automotive websites.

» Determination of indicators and modelling of navigation behaviour.

Market Analysis» Expert analysis of the most important manufacturer

websites with regard to user experience. Identification of best practices.

In Q1 2009, one-fifth of internet users visited atleast one automotive manufacturer website.

Internet users in Germany: 42,540,000

Total users of "Auto-Websites": 16,380,824

Users of manufacturers' websites: 8,308,536

Users of sales portals: 7,692,277

Users info portals: 5,423,271

Users of auto club websites: 4,153,947

Users of online community websites: 2,678,939

Users of media websites: 2,352,151

Users of auto group websites: 223,034 1%

Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

In the first quarter, manufacturer websites werevisited around 41 million times, and on average,9.5 PIs were generated.

(in mil.)

9.5 PIs per Visit11.1 5.0 8.9 5.9 3.8 4.6


The VW website has by far the most users,followed by Opel, Ford and Audi.


Users ofmanufacturerwebsites(in thousands)

Page 1 of 2

The Mercedes and VW websites have the highestrate of repeat visits.

1.9MercedesVW

OpelAudi

SkodaFiat

CitroënToyota

MitsubishiMazda

SeatPorscheRenault

JaguarSuzukiSmartHonda

DaciaFord

PeugeotBMW

NissanAlfa Romeo

DaihatsuHyundai

SubaruKia

VolvoChevrolet

Visits per user of the givenmanufacturer websites (mean)

Data base: Internet users inWEP designed by GfKbetween 01/01.–31/03/2009

Overview of the most significant indicators formanufacturer websites

Unique Users(in thousands)

PIs per Visit

Visits per User

User Proportionfor Showroom

use

User Proportionfor

Configuratoruse

Proportion ofended

configurations

User Proportionfor Financing

pages

User ProportionFinancingCalculator

User Proportionfor Dealer

Search

Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%

Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%

BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%

Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%

Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%

Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%

Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%

Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%

Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%

Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%

Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%

Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%

Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%

Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%

Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%

Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%

VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%

Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

* Refers to the 16 manufacturers represented

Showrooms, configurators and information aboutfinancing are of particular interest to users who intend to purchase – the dealer search is somewhat less used by those interested in purchasing.


Info: In the calculation of the mean,only those manufacturer websitesare represented for which use dataare available for the given area.

Fewer than half of manufacturer website usersvisit showrooms – if one is visited, then others arelikely to be also.

Showroom users (Q1): 3,680,249

1 Showroom visited: 1,373,005

2 Showrooms visited: 626,044




More than 5 Showrooms visited: 720,963

Number of showrooms visited for allusers of manufacturer websites (mean):

1.0

Number of showrooms visited for allusers of showrooms (mean):

4.2


Overview of the most significant indicators formanufacturer websites

Unique Users(in thousands)

PIs per Visit

Visits per User

User Proportionfor Showroom

use

User Proportionfor

Configuratoruse

Proportion ofended

configurations

User Proportionfor Financing

pages

User ProportionFinancingCalculator

User Proportionfor Dealer

Search

Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%

Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%

BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%

Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%

Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%

Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%

Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%

Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%

Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%

Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%

Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%

Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%

Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%

Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%

Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%

Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%

VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%

Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

* Refers to the 16 manufacturers represented

The showrooms on the websites of Dacia, Toyotaand Chevrolet are visited especially often.

Proportion of total users whovisited the showroom on anymanufacturer website (mean):

52%

Number of users of a website who visited theshowroom on thatwebsite.


Images of the models displayed on the start pagespur users on to visit the showrooms.

» The first-ranked websites of Dacia, Toyotaand Chevrolet feature links to the showroom directly on the start page, including images of the models.

» Websites that only offer a text link (e.g. Peugeot, VW) create less interestfor showrooms. If the links to the models are not prominently placed,

the showrooms are used significantly less often (see Honda).

Showrooms with a high degree of multimedia concentration often may notexploit their potential because they are less able to be located.

DaciaToyota

Honda

eCommerce Branchenmonitor 2008

35WebWerte

Analysis question

» Why does a website sells good – and better than the competition?

» Why do customers decide for one special online-shop?

» Where (also online/offline) and how do customers inform themselves?

» Which products are bought out of what reasons?

Focus: Consuming goods in low to middle pricesegment, which are bought regularly

Method- Mixing

1. LEOtrace® Behaviour Tracking of online-shopper

» Multi-Method Study for wholistic mapping and analysis of the onlineshopping-process, from wording the shoppingintention until delivery.

» Monitoring of users during productresearch, measuring of detailled data-characteristics and different interviews before, during and after the monitoring phase.

2. Expert assessment by the SirValUse E-Commerce-Team

» Focus: Best Practice-Analysis.

Fact sheetLEOtrace® Behaviour Tracking

Method» Several weeks long Add-On-Study for Internet Explorer and

Firefox.

Sample

» 440 Users were recruited via an Online-Access-Panel for the study.

– All participants planned to purchase online-products during field time.

– 59% Women, 41% Men.

– 52% under 30 years, 26% 30-39 years, 22% 40 years & older.

Field time » From Juli 1st until September 8th 2008.

Tested websites

» Testing of all websites which could be important for research-and shoppingprocess.

– Focus on onlineshops.

– Furthermore: price comparing-websites, searchengines and portals, manufacturer-websites and community-websites.

Expiry of Behaviour Trackings for Users

First survey:

Screening

Installation of AddOn

Questions to online-shopping

Field time:

Handle the shoppinglist

Monitoring of surf behaviour

Event-Interviews

Final survey:

Afterexploration

Imagemeasurement of onlineshops

Deinstallation of AddOn

2 weeks 7 weeks 1 week

TechnologieBrowser Extension LEOtrace® AddOn

The Shoppinglist

» Origin- and endpoint of all productresearch.

» The Users were told to keep following aspects up to date:

– Productcategory

– Productname

– (contemplated) price of product

– State of research

» Furthermore they were told before a research session, what product had to been researched.

Devolution of a research-session

Shoppinglist: Instruction forproduct to beresearched for

Shoppinglist: If so matching ofprice andresearch-status

Post survey: Evaluation ofresearch-session

Eventinterview "general"

Eventinterview"detailled product

website

Eventinterview"searchfunction

Eventinterview "ordering process"

Research-session

Example: Research- & Shoppingsession (1)

User: 244756

male, 29 years

Product for which exists a buying interest:

Nintendo Wii

Research-sessions:

July 20th (21:12 - 21:57)

July 21st (18:20 - 18:41)

On July 20th following websites were selected:

Geizkragen.de

Search for "Nintendo Wii"

cyberport.com

discount24.de

amazon.de

Example: Research- & Shoppingsession (2)

User: 244756

male, 29 years

On July 21st amazon.de was accessed:

search for "Wii"

Game console put into shopping basket

Search for "Mark Medlock"

CD "Cloud Dancer" by Mark Medlock put into shopping basket

Analysis of research:

Both products were purchased online at amazon.de

Comment about order:"Great that I could bought the CD here as well"

Research-Status at end of field time

Cases QuotaCases

per user

Still in research-phase 800 36% 1,8

Bought online 920 42% 2,1

Bought offline 189 9% 0,4

No shopping interest anymore

289 13% 0,7

Total 2.198 5,0

Basis: Total (N=440).

Used Onlineshops during field time

Basic: Total (N=440).

Rating of Onlineshops

Question: How helpful were following Onlineshops during research and purchase of the varied products?

Basic = User of each website

Rating on a scale of 6: 1=Very helpful 6=Not helpful at all.

Only a section of the answering scale is shown .

Amazon (N=342)

eBay (N=281)

Bonprix (N=50)

Weltbild (N=57)

Neckermann (N=53)

Tchibo (N=65)

Buecher.de (N=46)

Esprit (N=35)

Ikea (N=33)

Quelle (N=80)

Otto (N=81)

Buch.de (N=42)

H&M (N=36)

Conrad (N=64)

Used shops per product following productcategories

Computerhardware / -equipment

Toys

Books

housewares / -tools

Fitments

Consumer electronics

CDs / DVDs

Other products

Telecommunicationproducts

Fashion / Shoes

Videogames

Sports goods / -tools

Jewellery/Watches/Accessoiries

Healthproducts

Perfume / Cosmetiques

Tickets

2,3

Basic: products being researched for (N=1.766).

Rating of experiences during research-phase

Incomplete delivery Couldn´t find anything fitting

Unsufficient productdescription/-mapping

No wanted payment options

no response on E-Mail-request Complicated navigation

Delivery problems pricing not transparent

Product not available unsufficient searchfunction Oversupply

Speedtransfer Too expensive

Und zwar:

How do you rate the experiences you made during the last 6 weeks according to research and shopping in internet?

Tag Cloud View

Basic: Total (N=440).

True Intent Study

50WebWerte

Question 10040: Did you succeed in what you wanted to do on the website today?

in percent

Area: Shop – 'Handys & Shop'Success

37

52

63

48

successful not successful

Buy something (N=50)

Buy something online (N=50)

Results generated from log data.Participants who started from the page "Handys & Shop": N=100. Only navigation steps with N > 1.

NavigationCancel of remote-session

"Handys & Datengeräte"

"Aktions-Angebote"

"Vertrags-Verlängerung"

8 28 12 6

"Prepaid-Angebote"

10

Login area("https")

5

"Tarife & Optionen" *

Handsetdetails

8

24

30

27

9 24

Area: Shop – 'Handys & Shop'Structure of navigation

* First page of the area (Overview page)

in percent

Results generated from log data.Participants with buying intention: N=100. Only navigation steps with N > 3.

NavigationCancel of visit

Area: Shop – 'Handys & Shop' Structure of navigation – Buyers

"Handys & Shop" *

"Aktions-angebote"

25

37 Homepage23

"Handys & Datengeräte"

Handsetdetails

35

18

20

9

11

Login-area("https")

20

23

Pricepop-up

18

20

20

* First page of the area (Overview page)

in percent

Area: Shop – 'Handys & Shop'General

The homepage of the section 'Handys & Shop' was praised as clearly arranged and pleasantly designed.

The range of devices in 'Handys & Shop' was well liked. The choice of mobile phones was stated as complete and up-to-date. Particularly the phone offers for 1€ were liked by the participants.

"Great, that there are mobile phones for one Euro, which look o.k. and have many features."

If the preferred phone model was not offered on the website, the range of products was often criticized as too small. In this case the choice was also described as out of date. Participants always expect that the latest models are available on the website.

"I searched for the Samsung E 840, thinking that XXX as market leader always offers the most up-to-date phone models, even before other providers offer them."

" 'Top-Angebote' (top-offers) are on offer - latest mobile phones are missing."

Area: Shop – 'Handys & Shop'Mobile phones detail-page

Understanding the meaning of the several "function-icons" was a problem.

"Icons completely inoperative!!! They don't work for a comparison and are not understandable without explanations. Very, very bad."

The icons did not suggest that further explanations were available and participants could therefore not find them.

The offered alt-tags were often not clearly understandable (e.g. Voice, Orga).

The explanations were sometimes very long and it was necessary to read carefully through them in order to understand which function was described. Most users only wanted to know which function was included, not its

purpose.

? (radio)

Time for Questions

56WebWerte

SirValUse Consulting GmbH | Schlossstraße 8g | 22041 Hamburg

Tel. +49 (40) 68 28 27-0 | Fax +49 (40) 68 28 27-20

www.sirvaluse.de

WebWerte

http://www.nurago.com/

Measuring UX by Tim Bosenick, SirValUse

Technology

Transcript of Measuring UX by Tim Bosenick, SirValUse