Post on 18-Dec-2015
Empirically Validated Web Page Design Metrics
Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst
UC BerkeleyCHI 2001
Empirically Validated Web Page Design Metrics2
The Usability Gap
196M new Web sites in the next 5 years [Nielsen99]
~20,000 user interface professionals [Nielson99]
Empirically Validated Web Page Design Metrics3
The Usability Gap
Most sites have inadequate usability [Forrester, Spool, Hurst]
(users can’t find what they want 39-66% of the time)
196M new Web sites in the next 5 years [Nielsen99]
A shortage of user interface professionals [Nielson99]
Empirically Validated Web Page Design Metrics4
The Usability Problem
NON-professionals need to create websites Guidelines are helpful, but
– There are MANY usability guidelines Survey of 21 web guidelines found little overlap [Ratner et al. 96]
Why?– One idea: because they are not empirically validated
– Sometimes imprecise– Sometimes conflict
Empirically Validated Web Page Design Metrics5
Possible Solutions: Tools to Help Non-Professional Designers
Examples:– A “grammar checker” to assess guideline
conformance Imperfect Only suggestions – not dogma
– Automatic comparison to highly usable pages/sites– Automatic template suggestions
How to create these?
Empirically Validated Web Page Design Metrics6
Current Design Analysis Tools
Some tools report on easy-to-measure attributes
– Compare measures to thresholds Stein (Rating Game), Theng &
Marsden, Thimbley (Gentler) Not empirically validated
– Guideline conformance CAST (Bobby), Scholtz & Laskowski
(WebSAT), Lift Online
– Perceptually based heuristics Faraday (Design Advisor)
Small subset of features [Brajnik00]
– Simplistic– Not empirically validated
Empirically Validated Web Page Design Metrics7
The WebTANGO Approach
Models of good design by looking at existing designs– Empirical foundation for easy-to-measure attributes– Focus on information-centric sites
First work to take a large set of sites and analyze them
Empirically Validated Web Page Design Metrics8
The Investigation
Using quantitative measures to predict Web site ratings– Followup investigation [HFW00]
Given– 1898 pages from 400+ sites– 11 quantitative measures to assess various Web page aspects
Questions– Which features distinguish well-designed web pages?– Can metrics predict ratings?– Are there differences for categories of pages?
Empirically Validated Web Page Design Metrics9
Webby Awards 2000
2000 sites initially – 27 topical categories
We studied sites from information-centric categories– Finance, education, community, living, health, services
100 judges– International Academy of Digital Arts & Sciences– 3 rounds of judging
Empirically Validated Web Page Design Metrics10
Webby Awards 2000
6 criteria– Content– Structure & navigation– Visual design– Functionality– Interactivity– Overall experience
Scale: 1-10 (highest) Nearly normally distributed What are judgments about?
Empirically Validated Web Page Design Metrics11
Webby Awards 2000
Content criterion is best predictor
Visual design criterion is worst predictor
User study of 57 sites– Ratings reflect usability
Empirically Validated Web Page Design Metrics12
Quantitative Measures:Aspects Impacting Usability
Identified 42 attributes from the literature Roughly characterized:
– Page Composition words, links, images, …
– Page Formatting fonts, lists, colors, …
– Overall Page Characteristics information & layout quality, download speed, …
Empirically Validated Web Page Design Metrics13
Quantitative Measures: Word Count
Empirically Validated Web Page Design Metrics14
Quantitative Measures: Body Text %
Empirically Validated Web Page Design Metrics15
Quantitative Measures: Emphasized Body Text %
Empirically Validated Web Page Design Metrics16
Quantitative Measures: Text Positioning Count
Empirically Validated Web Page Design Metrics17
Quantitative Measures: Text Cluster Count
Empirically Validated Web Page Design Metrics18
Quantitative Measures: Link Count
Empirically Validated Web Page Design Metrics19
Quantitative Measures: Page Size (Bytes)
Empirically Validated Web Page Design Metrics20
Quantitative Measures: Graphic %
Empirically Validated Web Page Design Metrics21
Quantitative Measures: Graphic Count
Empirically Validated Web Page Design Metrics22
Quantitative Measures: Color Count
Empirically Validated Web Page Design Metrics23
Quantitative Measures: Font Count
Empirically Validated Web Page Design Metrics24
Characterizing Measures:A View of Web Site Structure [Newman et al. DIS00]
Information design– structure, categories of information
Navigation design– interaction with information
structure
Graphic design– visual presentation
Courtesy of Mark Newman
Characterizing Measures: Web Site Structure Assessed
Metric Information Design
Navigation Design
Graphic Design
Word Count Body Text % Emp. Body Text % Text Positioning Count Text Cluster Count Link Count Page Size Graphic % Graphics Count Color Count Font Count
Empirically Validated Web Page Design Metrics26
Study Method
The Webby factor– Principle components analysis of the 6 criteria
Accounted for 91% of the variance
Two comparisons– Model 1: Highly rated sites (top 33%) vs. the rest
Using the overall Webby score
– Model 2: Highly rated sites vs. bottom 33% Using the Webby factor
Empirically Validated Web Page Design Metrics27
Findings
We can accurately classify web pages– Linear discriminant analysis – Model 1: For highly-rated sites vs. rest
67% correct when not considering content categories 73% correct when taking content categories into account
– Model 2: For highly-rated sites vs. bottom 65% correct when not considering content categories 80% correct when taking content categories into account
Empirically Validated Web Page Design Metrics28
Findings
Top vs. bottom
– Webby factor
Linear discriminant analysis
Better for categories
Empirically Validated Web Page Design Metrics29
Deeper Analysis
Which metrics matter?– Linear regression analysis
Backward elimination until adjusted R² reduced
– All metrics played a role Compared small, medium, and large pages
– Across the board (preliminary profiles) Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)
Empirically Validated Web Page Design Metrics30
Small pages (66 words on average)
Good small pages have (according to beta coefficients)
– Slightly more content– Smaller page sizes – Fewer graphics – More font variations
Suggests that these pages – Have faster download times
corroborated by a download time metric
– Use different fonts for headings vs. the rest of the text
Examples– Services (Home Pages)
Top Bottom
Empirically Validated Web Page Design Metrics31
Medium pages (230 words on average)
Good medium pages – Emphasize less of the body text– Appear to organize text into clusters (e.g., lists and
shaded table areas)– Use colors to distinguish headings from body text
Suggests that these pages– Are easier to scan
Empirically Validated Web Page Design Metrics32
Large pages (827 words on average)
Good large pages have – More headings– More links– Are larger but have fewer graphics
Probably attributable to style sheets
Suggests that good large pages– Are easier to scan and facilitate information seeking
Empirically Validated Web Page Design Metrics33
Why does this work?
Content is most important predictor BUT there’s predictive power in other aspects
– Visual and navigation design– Verifies preliminary investigation [HFW00]
Possibly: Good design is good design all over Note: we are NOT suggesting we can
characterize:– Aesthetics or subjective preferences
Empirically Validated Web Page Design Metrics34
How might we use this?
Web Site Design
•Prediction•Similarities•Differences•Suggestions
Analysis Tool
Profiles
Comparable Designs
Favorite Designs
Empirically Validated Web Page Design Metrics35
Future work
Distinguish according to page role– Home page vs. content vs. index …
Better metrics– More aspects of info, navigation, and graphic design
Site level as well as page level Category-based profiles
– Use clustering to create profiles of good and poor sites – These can be used to suggest alternative designs
Empirically Validated Web Page Design Metrics36
In Summary
Automated tools should help close the Web Usability Gap
We have a foundation for a new methodology– Empirical, bottom up
We can empirically distinguish good pages– Empirical validation of design guidelines– Can build profiles of good vs. poor sites
Eventually build tools to help users assess designs
Empirically Validated Web Page Design Metrics37
More Information
http://webtango.berkeley.edu