Technology for a better society 1

Asbjørn Følstad

1st European workshop on HCI design and evaluation

Limassol, Cyprus, April 8., 2011

Usability evaluation in exclusive domains: How to access domain knowledge

Thoughts in progress!

Conclusions up for discussion

Technology for a better society 2

The two loves of the usability evaluator

Usability testing

Usability inspection

Technology for a better society 3

The two loves of the usability evaluator

Usability testing

Usability inspection considering users

observing users

What do we miss by not accessing the user’s knowledge during our usability evaluations?

Technology for a better society 4

We do not miss much, unless the user …

What do we miss by not accessing the user’s knowledge during our usability evaluations?

… holds domain knowledge we cannot access

Technology for a better society 5

What we may miss if the user knows more of the domain than the usability evaluator:

Review of two studies of usability evaluation methods that allow us to access the users’ knowledge.

Technology for a better society

• Mobile applications for • Medical personnel at hospital wards• Parking wardens• Political advisors

• Evaluated with groups of either• Domain experts• Usability experts

6

Consider: Three cases of Group-based expert walkthrough**) an inspection method allowing domain experts to serve as usability inspectors

12 % overlap in findings between evaluator groups

Domain experts: 59 % domain specific findings

Usability experts: 15 % domain specific findings

Følstad, A., 2007. Work-domain experts as evaluators: usability inspection of domain-specific work-support systems. International Journal of Human– Computer Interaction 22 (3), 217–245.

Technology for a better society

• Mobile applications for • Medical personnel at hospital wards• Parking wardens• Political advisors

• Evaluated with groups of either• Domain experts• Usability experts

7

Consider: Three cases of Group-based expert walkthrough**) an inspection method allowing domain experts to serve as usability inspectors

54 % of domain experts’ findings given high priority by client

27 % of usability experts’ findings given the same priority

Følstad, A., 2007. Work-domain experts as evaluators: usability inspection of domain-specific work-support systems. International Journal of Human– Computer Interaction 22 (3), 217–245.

Technology for a better society

• Mobile applications for • Mobile sales personnel• Medical emergency personnel

• Usability issues identified through • Observation (in the interaction phases)• User dialogue (in the interpretation phases)

8

Consider also: Two cases of cooperative usability testing**) usability testing with phases of interpretative dialogue with the user participant

Findings through user dialogue ...

covered a broader range of themes:-Static design- Interaction design- Needed information- Needed functionality- Req’s for use- Req’s for content- Technical issues

Følstad, A. & Hornbæk, K. (2010) Work-domain knowledge in usability evaluation: Experiences withCooperative Usability Testing. Journal of systems and software, 83, 2019-2030.

Technology for a better society

• Mobile applications for • Mobile sales personnel• Medical emergency personnel

• Usability issues identified through • Observation (in the interaction phases)• User dialogue (in the interpretation phases)

9

Consider also: Two cases of cooperative usability testing**) usability testing with phases of interpretative dialogue with the user participant

Findings through user dialogue ...

had similar impact on subsequent developmentcompared to findings through observation

Følstad, A. & Hornbæk, K. (2010) Work-domain knowledge in usability evaluation: Experiences withCooperative Usability Testing. Journal of systems and software, 83, 2019-2030.

Technology for a better society 10

Why did it pay off to access the users’ knowledge in these cases?

Technology for a better society 11

Domain exclusiveness*

*) The unavailability of domain knowledge to the outsiders of a specific domain

Level of training

Level of specialization

High

High

Low

Low

Specialist(Medical care at hospital wards; medical emergency response)

Generalist(General office work; project leadership)

Popular(eCommerce and eGovernment customership)

Limited training(Professional sales; parking enforcemenet)

Usability testing

Usability inspection

Usabilitytesting

Usabilityinspecti

on

Usability testing

Usability inspection

Technology for a better society 12

In conclusion:

When doing usability evaluation in exclusive domains …

… consider using methods that allow you to access the domain knowledge of users