Collaborative Heuristic Evaluation, UPA2010
-
Upload
lucy-buykx -
Category
Technology
-
view
1.152 -
download
1
description
Transcript of Collaborative Heuristic Evaluation, UPA2010
![Page 1: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/1.jpg)
Collaborative Heuristic Evaluation:improving the effectiveness of
heuristic evaluation
Helen Petrie, Lucy BuykxHuman Computer Interaction Research Group
Department of Computer ScienceUniversity of York, UK
![Page 2: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/2.jpg)
Standard heuristic evaluation (SHE)
Strengths• Independent evaluator viewpoints• High volume of potential problems
Weaknesses• Laborious, boring, frustrating• Time consuming to combine problems from individual
evaluators• High proportion of trivial problems
![Page 3: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/3.jpg)
Collaborative Heuristic Evaluation (CHE)
Method– Evaluators work in a group– Propose potential usability problems– Create problem list in real time– Independently rate severity
![Page 4: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/4.jpg)
CHE Problem rating sheet
12345
![Page 5: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/5.jpg)
Experiment method
• 4 websites• airline, train, tourist information sites
• 2 groups of 5 usability professionals
• 2 methods: SHE and CHE– each professional performed each method on
two websites
![Page 6: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/6.jpg)
Experiment design
Group 1Group 1 Group 2Group 2
CHECHENational RailVisit Britain
British TownsEasy Jet
SHESHEEasy Jet
British TownsVisit BritainNational Rail
![Page 7: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/7.jpg)
Analysis of data
Made master problem list for SHE– Only included problems found on pages visited
by all evaluators– Relaxed matching of problems
Master list for CHE already there
Analysed how many evaluators found each problem in each method
![Page 8: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/8.jpg)
Results
SHE: 22.9 usability problems per websitePool of 250 problems on pages visited by all
evaluators
CHE: 42.7 usability problems per websitePool of 153 problems
(smaller pool, because of greater overlap between evaluators …)
![Page 9: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/9.jpg)
Inter-evaluator agreement
![Page 10: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/10.jpg)
National Rail website
SHE CHE
![Page 11: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/11.jpg)
Any-two agreement: measure of evaluator effect
SHESHE CHECHE
Agree it’s a Agree it’s a problemproblem
8.9% 91.6%
Agree severity Agree severity ratingrating
36.0%navg = 19
37.5%navg = 38
![Page 12: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/12.jpg)
Conclusion
CHE ….• Exposes evaluators to each other ideas • Creates more agreement about problems• More efficient as a method
- creates master problem list in the session– larger number of problems per hour
• Generates discussion/ideas for solving problems
• More enjoyable for evaluators
![Page 14: Collaborative Heuristic Evaluation, UPA2010](https://reader036.fdocuments.us/reader036/viewer/2022081201/5593eed81a28ab6f3b8b46e0/html5/thumbnails/14.jpg)
Matching process
• “don’t know what overtaking trains means”• “don’t know what ‘include overtaking trains
in journey results’ means”
• “want to be able to select any time of day but drop down only offers 4 hour segments”
• “specified times is well hidden and inflexible - want to go from any time”