Thoughts on HCI Requirements Elicitation

Thoughts on HCI Requirements Elicitation

Glenn Fink

31 August 2006

Glenn Fink

31 August 2006

AgendaAgenda

• What is requirements elicitation?

• What I Did

• What Worked

• What Didn’t Work

• What’s Hard Work

• What I Haven’t Tried (Yet)

• Conclusions

• What is requirements elicitation?

• What I Did

• What Worked

• What Didn’t Work

• What’s Hard Work

• What I Haven’t Tried (Yet)

• Conclusions

What is requirements elicitation?What is requirements elicitation?

• Finding out what the users want– Who are they?– What do they do?– Where do they have problems?

• Designing prototype solutions– As simple, cheap, and quick as possible

• Working with the users to improve the prototypes (evolutionary prototyping)

• Finding out what the users want– Who are they?– What do they do?– Where do they have problems?

• Designing prototype solutions– As simple, cheap, and quick as possible

• Working with the users to improve the prototypes (evolutionary prototyping)

What I DidWhat I Did

• Discovering what system administrators need for security

• Finding out how (whether) information visualization could help

• Repeatedly:– Building prototypes– Evaluating the results

• Discovering what system administrators need for security

• Finding out how (whether) information visualization could help

• Repeatedly:– Building prototypes– Evaluating the results

psgraph

Understanding the user community's true needs required evolutionary prototypingUnderstanding the user community's true needs required evolutionary prototyping

User InterviewsPaper Prototypes

Network Pixel MapVISUALNetwork Eye GL

Portall

Host-Network Visualizer (HoNe)

July 2003July 2003

August 2006August 2006

Summative Usability EvaluationSummative Usability Evaluation

• This was the last of a series of evaluations

• I tested 27 system administrators performing intrusion-detection tasks

• My visualization significantly improved scores

• Subjects said that HoNe was better than their existing tools

• This was the last of a series of evaluations

• I tested 27 system administrators performing intrusion-detection tasks

• My visualization significantly improved scores

• Subjects said that HoNe was better than their existing tools

Users preferred and got the most insight from the VC conditionUsers preferred and got the most insight from the VC condition

• Users complained when going from visualization to text• I received many unsolicited positive comments• Users complained when going from visualization to text• I received many unsolicited positive comments

• Statistically significant at 0.001 level (2=135, df=9)• Many users asked for a copy of the visualization

What WorkedWhat Worked

• Semi-structured interviews– HCI Experts– Domain Experts– Expert Users

• Audio recording, taking notes afterward• Paper and PowerPoint prototypes

– Brainstorming sessions with users

• Carefully designed usability evaluations– Rewarding participants

• Semi-structured interviews– HCI Experts– Domain Experts– Expert Users

• Audio recording, taking notes afterward• Paper and PowerPoint prototypes

– Brainstorming sessions with users

• Carefully designed usability evaluations– Rewarding participants

What Didn’t WorkWhat Didn’t Work

• Transcribing audio interviews verbatim• High-fidelity prototypes

– Costly to build, but never enough– Small problems hide core issues

• Changing the interview protocol during the study– May not be avoidable in exploratory studies

• Careless errors in usability evaluation design

• Transcribing audio interviews verbatim• High-fidelity prototypes

– Costly to build, but never enough– Small problems hide core issues

• Changing the interview protocol during the study– May not be avoidable in exploratory studies

• Careless errors in usability evaluation design

What’s Hard WorkWhat’s Hard Work

• Distilling quantifiable facts from semi-structured interviews

• Making sure users understand low-fidelity prototypes the same way you do

• Written interviews (Domain Experts)– Requires follow-up live interviews– You may never get the results

• Solid statistical analysis (get help)

• Distilling quantifiable facts from semi-structured interviews

• Making sure users understand low-fidelity prototypes the same way you do

• Written interviews (Domain Experts)– Requires follow-up live interviews– You may never get the results

• Solid statistical analysis (get help)

What I Haven’t Tried (yet)What I Haven’t Tried (yet)

• Broad Surveys

• Diary Studies

• Field studies (tag-along)

• Broad Surveys

• Diary Studies

• Field studies (tag-along)

ConclusionsConclusions

• Research is always built slowly, brick by painful brick– It’s relatively easy to:

• Schedule interviews• Take notes• Get a good feeling for the subject

– It’s relatively hard to:• Talk on the same wavelength as your subjects• Distill useful facts from interview notes• Write up your findings coherently

• Research is always built slowly, brick by painful brick– It’s relatively easy to:

• Schedule interviews• Take notes• Get a good feeling for the subject

– It’s relatively hard to:• Talk on the same wavelength as your subjects• Distill useful facts from interview notes• Write up your findings coherently