Svein ØlnesVestlandsforsking

26.06.2007

Quality Evaluationsand Accessibility

of Public Web Sites

Content• Background• Method• Indicators• Results 2006• Future development

Background• Evaluations started in 2001• Norge.no responsible from 2004• Annual evaluation of all public web sites in

Norway• 2006: 691 web sites evaluated• 2007: In addition to public sector web

sites, also some private web portals will be evaluated

Method• Expert evaluation• Ca. 1 hour per. site• Trained evaluators to reduce variations• Indicators formulated from standards, best

practice and eGovernment objectives

Quality 2006• Three main areas:

– Accessibility (10 indicators – max 21 points)• derived from WCAG priority 1 guidelines (“must”)• possible to transform into measurable indicators

– User orientation (16 indicators – max 38 pts)– Usable services (9 indicators – max 28 pts)

• Relative weight– Accessibility: 24 %– User orientation: 44 %– Usable services: 32 %

• Accessibility is important but must be seen in connection with other requirements– you can have perfect accessibility on a perfectly useless

web site

Accessibility is part of the picture

Peter Morville’s User Experience Honeycomb

Results2006

Indicator AlternativeNumber of sites

% avtotal

1.1 ALT-text No 240 34,8 % Problem

1.1 Partly 251 36,4 %

1.1 Yes 199 28,8 %

1.2 Accessible also without colours? No 103 14,9 %

1.2 Yes 587 85,1 %

1.3 Functionable without CSS instructions? No 68 9,9 %

1.3 Yes 622 90,1 %

1.4 Free from blinking elements etc. No 3 0,4 %

1.4 Yes 687 99,6 %

1.5 Are data tables marked up correctly? No 408 59,1 % Problem

1.5 Partly 166 24,1 %

1.5No data table 58 8,4 %

1.5 Yes 58 8,4 %

Results1.6 Meaningful titles on frames, if frames? No 166 24,1 %

1.6 No frames 511 74,1 %

1.6 Yes 13 1,9 %

1.7 Functionable without scripts and plug-ins? No 98 14,2 %

1.7 Partly 118 17,1 %

1.7 Yes 474 68,7 %

1.8 Contrast between foreground and background colours Poor 90 13,0 %

1.8 Good 600 87,0 %

1.9 Is there a way to jump directly to content? No 586 84,9 % Problem

1.9 Yes 104 15,1 %

1.10 When using frames, is provider clearly marked and No 64 9,3 %

1.10 is the complete frameset loaded? No frames 564 81,7 %

1.10 Yes 62 9,0 %

2006

Total number of evaluated web sites 690

Quality 2006 - results• Accessibility: 75 % of max points• User orientation: 63 % of max points• Usable services: 54 % of max points

• Accessibility problems:– Alternative text (ALT text) to graphic elements– Use of data table– Link to content

Quality evaluation – lessons learned

• The proof of the pudding is in the eating ->– the proof of the indicators is in the testing

• The difficult part is to transform guidelines into operable indicators– difficult to reduce the subjective factor and thus

secure minimal variations between evaluators

• What do we measure?– do we measure what is important for the

users?

Possible future of quality evaluations

• Combination of– expert testing– automatic testing

• User tests for evaluation of the test framework itself

Thank you!

Contact information

Svein Ølnes

email: svein.olnes@vestforsk.no

Web: www.vestforsk.no