Empirically Validated Web Page Design Metrics Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst UC...

Empirically Validated Web Page Design Metrics

Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst

UC BerkeleyCHI 2001

Empirically Validated Web Page Design Metrics2

The Usability Gap

196M new Web sites in the next 5 years [Nielsen99]

~20,000 user interface professionals [Nielson99]


The Usability Gap

Most sites have inadequate usability [Forrester, Spool, Hurst]

(users can’t find what they want 39-66% of the time)

196M new Web sites in the next 5 years [Nielsen99]

A shortage of user interface professionals [Nielson99]


The Usability Problem

NON-professionals need to create websites Guidelines are helpful, but

– There are MANY usability guidelines Survey of 21 web guidelines found little overlap [Ratner et al. 96]

Why?– One idea: because they are not empirically validated

– Sometimes imprecise– Sometimes conflict


Possible Solutions: Tools to Help Non-Professional Designers

Examples:– A “grammar checker” to assess guideline

conformance Imperfect Only suggestions – not dogma

– Automatic comparison to highly usable pages/sites– Automatic template suggestions

How to create these?


Current Design Analysis Tools

Some tools report on easy-to-measure attributes

– Compare measures to thresholds Stein (Rating Game), Theng &

Marsden, Thimbley (Gentler) Not empirically validated

– Guideline conformance CAST (Bobby), Scholtz & Laskowski

(WebSAT), Lift Online

– Perceptually based heuristics Faraday (Design Advisor)

Small subset of features [Brajnik00]

– Simplistic– Not empirically validated


The WebTANGO Approach

Models of good design by looking at existing designs– Empirical foundation for easy-to-measure attributes– Focus on information-centric sites

First work to take a large set of sites and analyze them


The Investigation

Using quantitative measures to predict Web site ratings– Followup investigation [HFW00]

Given– 1898 pages from 400+ sites– 11 quantitative measures to assess various Web page aspects

Questions– Which features distinguish well-designed web pages?– Can metrics predict ratings?– Are there differences for categories of pages?


Webby Awards 2000

2000 sites initially – 27 topical categories

We studied sites from information-centric categories– Finance, education, community, living, health, services

100 judges– International Academy of Digital Arts & Sciences– 3 rounds of judging


Webby Awards 2000

6 criteria– Content– Structure & navigation– Visual design– Functionality– Interactivity– Overall experience

Scale: 1-10 (highest) Nearly normally distributed What are judgments about?


Webby Awards 2000

Content criterion is best predictor

Visual design criterion is worst predictor

User study of 57 sites– Ratings reflect usability


Quantitative Measures:Aspects Impacting Usability

Identified 42 attributes from the literature Roughly characterized:

– Page Composition words, links, images, …

– Page Formatting fonts, lists, colors, …

– Overall Page Characteristics information & layout quality, download speed, …


Quantitative Measures: Word Count


Quantitative Measures: Body Text %


Quantitative Measures: Emphasized Body Text %


Quantitative Measures: Text Positioning Count


Quantitative Measures: Text Cluster Count


Quantitative Measures: Link Count


Quantitative Measures: Page Size (Bytes)


Quantitative Measures: Graphic %


Quantitative Measures: Graphic Count


Quantitative Measures: Color Count


Quantitative Measures: Font Count


Characterizing Measures:A View of Web Site Structure [Newman et al. DIS00]

Information design– structure, categories of information

Navigation design– interaction with information

structure

Graphic design– visual presentation

Courtesy of Mark Newman

Characterizing Measures: Web Site Structure Assessed

Metric Information Design

Navigation Design

Graphic Design

Word Count Body Text % Emp. Body Text % Text Positioning Count Text Cluster Count Link Count Page Size Graphic % Graphics Count Color Count Font Count


Study Method

The Webby factor– Principle components analysis of the 6 criteria

Accounted for 91% of the variance

Two comparisons– Model 1: Highly rated sites (top 33%) vs. the rest

Using the overall Webby score

– Model 2: Highly rated sites vs. bottom 33% Using the Webby factor


Findings

We can accurately classify web pages– Linear discriminant analysis – Model 1: For highly-rated sites vs. rest

67% correct when not considering content categories 73% correct when taking content categories into account

– Model 2: For highly-rated sites vs. bottom 65% correct when not considering content categories 80% correct when taking content categories into account


Findings

Top vs. bottom

– Webby factor

Linear discriminant analysis

Better for categories


Deeper Analysis

Which metrics matter?– Linear regression analysis

Backward elimination until adjusted R² reduced

– All metrics played a role Compared small, medium, and large pages

– Across the board (preliminary profiles) Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)


Small pages (66 words on average)

Good small pages have (according to beta coefficients)

– Slightly more content– Smaller page sizes – Fewer graphics – More font variations

Suggests that these pages – Have faster download times

corroborated by a download time metric

– Use different fonts for headings vs. the rest of the text

Examples– Services (Home Pages)

Top Bottom


Medium pages (230 words on average)

Good medium pages – Emphasize less of the body text– Appear to organize text into clusters (e.g., lists and

shaded table areas)– Use colors to distinguish headings from body text

Suggests that these pages– Are easier to scan


Large pages (827 words on average)

Good large pages have – More headings– More links– Are larger but have fewer graphics

Probably attributable to style sheets

Suggests that good large pages– Are easier to scan and facilitate information seeking


Why does this work?

Content is most important predictor BUT there’s predictive power in other aspects

– Visual and navigation design– Verifies preliminary investigation [HFW00]

Possibly: Good design is good design all over Note: we are NOT suggesting we can

characterize:– Aesthetics or subjective preferences


How might we use this?

Web Site Design

•Prediction•Similarities•Differences•Suggestions

Analysis Tool

Profiles

Comparable Designs

Favorite Designs


Future work

Distinguish according to page role– Home page vs. content vs. index …

Better metrics– More aspects of info, navigation, and graphic design

Site level as well as page level Category-based profiles

– Use clustering to create profiles of good and poor sites – These can be used to suggest alternative designs


In Summary

Automated tools should help close the Web Usability Gap

We have a foundation for a new methodology– Empirical, bottom up

We can empirically distinguish good pages– Empirical validation of design guidelines– Can build profiles of good vs. poor sites

Eventually build tools to help users assess designs


More Information

http://webtango.berkeley.edu

Empirically Validated Web Page Design Metrics Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst UC...

Documents

Transcript of Empirically Validated Web Page Design Metrics Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst UC...