1

Asking users & Asking users & expertsexperts

and Testing & and Testing & modeling usersmodeling users

Ref: Ch. 13-14Ref: Ch. 13-14

2

The aimsThe aims

Discuss the role of interviews & Discuss the role of interviews & questionnaires in evaluation.questionnaires in evaluation.

Teach basic questionnaire design. Teach basic questionnaire design. Describe how to do interviews, Describe how to do interviews,

heuristic evaluation & walkthroughs. heuristic evaluation & walkthroughs. Describe how to collect, analyze & Describe how to collect, analyze &

present data. present data. Discuss strengths & limitations of Discuss strengths & limitations of

these techniquesthese techniques

3

InterviewsInterviews

Unstructured - are not directed by a Unstructured - are not directed by a script. Rich but not replicable. script. Rich but not replicable.

Structured - are tightly scripted, Structured - are tightly scripted, often like a questionnaire. Replicable often like a questionnaire. Replicable but may lack richness.but may lack richness.

Semi-structured - guided by a script Semi-structured - guided by a script but interesting issues can be explored but interesting issues can be explored in more depth. Can provide a good in more depth. Can provide a good balance between richness and balance between richness and replicability.replicability.

4

Basics of interviewingBasics of interviewing

Remember the DECIDE frameworkRemember the DECIDE framework Goals and questions guide all interviewsGoals and questions guide all interviews Two types of questions:Two types of questions:

‘closed questions’ have a predetermined ‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’answer format, e.g., ‘yes’ or ‘no’‘open questions’ do not have a ‘open questions’ do not have a predetermined formatpredetermined format

Closed questions are quicker and easier to Closed questions are quicker and easier to analyzeanalyze

DDetermine the etermine the goalsgoals the evaluation addresses the evaluation addresses..EExplore the specificxplore the specific questions questions to be answered.to be answered.CChoose the hoose the evaluationevaluation paradigmparadigm and and techniquestechniques to answer the questions. to answer the questions.IIdentify the dentify the practical issues.practical issues.DDecide how to deal with the ecide how to deal with the ethical issuesethical issues..EEvaluate, interpret and present thevaluate, interpret and present the data. data.

5

Things to avoid when Things to avoid when preparing interview preparing interview

questionsquestions Long questionsLong questions Compound sentences - split into twoCompound sentences - split into two Jargon & language that the Jargon & language that the

interviewee may not understand interviewee may not understand Leading questions that make Leading questions that make

assumptions e.g., why do you like …?assumptions e.g., why do you like …? Unconscious biases e.g., gender Unconscious biases e.g., gender

stereotypesstereotypes

6

Components of an Components of an interviewinterview

IntroductionIntroduction - introduce yourself, explain the - introduce yourself, explain the goals of the interview, reassure about the goals of the interview, reassure about the ethical issues, ask to record, present an ethical issues, ask to record, present an informed consent form.informed consent form.

Warm-upWarm-up - make first questions easy & non- - make first questions easy & non-threatening. threatening.

Main body – Main body – present questions in apresent questions in a logicallogical orderorder A cool-off period - A cool-off period - includeinclude a few easy questions a few easy questions

to defuse tension at the endto defuse tension at the end Closure - Closure - thank interviewee, signal the end, thank interviewee, signal the end,

e.g, switch recorder off.e.g, switch recorder off.

7

The interview processThe interview process

Use the DECIDE framework for Use the DECIDE framework for guidanceguidance

Dress in a similar way to participantsDress in a similar way to participants Check recording equipment in advanceCheck recording equipment in advance Devise a system for coding names of Devise a system for coding names of

participants to preserve confidentiality.participants to preserve confidentiality. Be pleasantBe pleasant Ask participants to complete an Ask participants to complete an

informed consent forminformed consent form

8

Probes and promptsProbes and prompts

Probes - devices for getting more Probes - devices for getting more information.information.e.g., ‘would you like to add anything?’e.g., ‘would you like to add anything?’

Prompts - devices to help interviewee, Prompts - devices to help interviewee, e.g., help with remembering a namee.g., help with remembering a name

Remember that probing and Remember that probing and prompting should not create bias.prompting should not create bias.

Too much can encourage participants Too much can encourage participants to try to guess the answer.to try to guess the answer.

9

Group interviewsGroup interviews

Also known as ‘focus groups’Also known as ‘focus groups’ Typically 3-10 participantsTypically 3-10 participants Provide a diverse range of Provide a diverse range of

opinionsopinions Need to be managed to:Need to be managed to:

- ensure everyone contributes- ensure everyone contributes- discussion isn’t dominated by - discussion isn’t dominated by one personone person- the agenda of topics is covered- the agenda of topics is covered

10

Analyzing interview Analyzing interview datadata

Depends on the type of interviewDepends on the type of interview Structured interviews can be Structured interviews can be

analyzed like questionnairesanalyzed like questionnaires Unstructured interviews generate Unstructured interviews generate

data like that from participant data like that from participant observationobservation

It is best to analyze unstructured It is best to analyze unstructured interviews as soon as possible to interviews as soon as possible to identify topics and themes from the identify topics and themes from the datadata

11

Asking Users: Asking Users: QuestionnairesQuestionnaires

Questions can be closed or openQuestions can be closed or open Closed questions are easiest to analyze, Closed questions are easiest to analyze,

and may be done by computerand may be done by computer Can be administered to large populationsCan be administered to large populations Paper, email & the web used for Paper, email & the web used for

disseminationdissemination Advantage of electronic questionnaires is Advantage of electronic questionnaires is

that data goes into a data base & is easy to that data goes into a data base & is easy to analyzeanalyze

Sampling can be a problem when the size Sampling can be a problem when the size of a population is unknown as is common of a population is unknown as is common onlineonline

12

Questionnaire styleQuestionnaire style Varies according to goal so use the Varies according to goal so use the

DECIDE framework for guidanceDECIDE framework for guidance Questionnaire format can include:Questionnaire format can include:

- ‘yes’, ‘no’ checkboxes- ‘yes’, ‘no’ checkboxes- checkboxes that offer many options- checkboxes that offer many options- Likert rating scales 1, 2, 3 ,4, 5 - Likert rating scales 1, 2, 3 ,4, 5 - semantic scales- semantic scales- open-ended responses- open-ended responses

Likert scales have a range of pointsLikert scales have a range of points 3, 5, 7 & 9 point scales are common3, 5, 7 & 9 point scales are common Debate about which is bestDebate about which is best

Attractive |___|_X_|___|___|___|UglyClear |___|___|_X_|___|___|ConfusingDull |___|___|___|___|___|ColorfulExciting |___|_X_|___|___|___|BoringAnnoying |___|___|___|___|_X_|PleasingPoor |___|___|___|_X_|___|Well-designed

13

Developing a Developing a questionnairequestionnaire

Provide a clear statement of purpose & Provide a clear statement of purpose & guarantee participants anonymity guarantee participants anonymity

Plan questions - if developing a web-Plan questions - if developing a web-based questionnaire, design off-line based questionnaire, design off-line firstfirst

Decide on whether phrases will all be Decide on whether phrases will all be positive, all negative or mixedpositive, all negative or mixed

Pilot test questions - are they clear, is Pilot test questions - are they clear, is there sufficient space for responsesthere sufficient space for responses

Decide how data will be analyzed & Decide how data will be analyzed & consult a statistician if necessaryconsult a statistician if necessary

14

Encouraging a good Encouraging a good responseresponse

Make sure purpose of study is clearMake sure purpose of study is clear Promise anonymityPromise anonymity Ensure questionnaire is well designedEnsure questionnaire is well designed Offer a short version for those who do not Offer a short version for those who do not

have time to complete a long questionnairehave time to complete a long questionnaire If mailed, include a s.a.e.If mailed, include a s.a.e. Follow-up with emails, phone calls, lettersFollow-up with emails, phone calls, letters Provide an incentiveProvide an incentive 40% response rate is high, 20% is often 40% response rate is high, 20% is often

acceptableacceptable

15

Advantages of online Advantages of online questionnairesquestionnaires

Responses are usually received quicklyResponses are usually received quickly No copying and postage costsNo copying and postage costs Data can be collected in database for Data can be collected in database for

analysisanalysis Time required for data analysis is reducedTime required for data analysis is reduced Errors can be corrected easilyErrors can be corrected easily Disadvantage - sampling problematic if Disadvantage - sampling problematic if

population size unknownpopulation size unknown Disadvantage - preventing individuals Disadvantage - preventing individuals

from responding more than oncefrom responding more than once

16

Problems with online Problems with online questionnairesquestionnaires

Sampling is problematic if Sampling is problematic if population size is unknownpopulation size is unknown

Preventing individuals from Preventing individuals from responding more than onceresponding more than once

17

Developing a web-based Developing a web-based questionnairequestionnaire

Produce an error-free interactive interactive Produce an error-free interactive interactive electronic version from the original paper-electronic version from the original paper-based one.based one.

Make the questionnaire accessible from all Make the questionnaire accessible from all common browsers and readable from different-common browsers and readable from different-size monitors and different network locations.size monitors and different network locations.

Make sure information identifying each Make sure information identifying each respondent will be captured and stored respondent will be captured and stored confidentially because the same person may confidentially because the same person may submit several complete surveys.submit several complete surveys.

User-test the survey with pilot studies before User-test the survey with pilot studies before distributing.distributing.

18

Questionnaire data Questionnaire data analysis & presentationanalysis & presentation

Present results clearly - tables may helpPresent results clearly - tables may help Simple statistics can say a lot, e.g., Simple statistics can say a lot, e.g.,

mean, median, mode, standard mean, median, mode, standard deviationdeviation

Percentages are useful but give Percentages are useful but give population sizepopulation size

Bar graphs show categorical data wellBar graphs show categorical data well More advanced statistics can be used if More advanced statistics can be used if

neededneeded

19

Asking expertsAsking experts

InspectionsInspections Experts use their knowledge of users & Experts use their knowledge of users &

technology to review software usabilitytechnology to review software usability Expert critiques (crits) can be formal Expert critiques (crits) can be formal

or informal reportsor informal reports Heuristic evaluation is a review guided Heuristic evaluation is a review guided

by a set of heuristicsby a set of heuristics Walkthroughs involve stepping through Walkthroughs involve stepping through

a pre-planned scenario noting potential a pre-planned scenario noting potential problemsproblems

20

Heuristic evaluationHeuristic evaluation

Developed Jacob Nielsen in the early Developed Jacob Nielsen in the early 1990s1990s

Based on heuristics distilled from an Based on heuristics distilled from an empirical analysis of 249 usability empirical analysis of 249 usability problemsproblems

These heuristics have been revised for These heuristics have been revised for current technology, e.g., HOMERUN current technology, e.g., HOMERUN for webfor web

Heuristics still needed for mobile Heuristics still needed for mobile devices, wearables, virtual worlds, etc.devices, wearables, virtual worlds, etc.

Design guidelines form a basis for Design guidelines form a basis for developing heuristicsdeveloping heuristics

21

Nielsen’s heuristicsNielsen’s heuristics Visibility of system statusVisibility of system status Match between system and real worldMatch between system and real world User control and freedomUser control and freedom Consistency and standardsConsistency and standards Help users recognize, diagnose, recover from Help users recognize, diagnose, recover from

errorserrors Error prevention Error prevention Recognition rather than recallRecognition rather than recall Flexibility and efficiency of useFlexibility and efficiency of use Aesthetic and minimalist designAesthetic and minimalist design Help and documentationHelp and documentation

22

Discount evaluationDiscount evaluation Heuristic evaluation is referred to Heuristic evaluation is referred to

as discount evaluation when 5 as discount evaluation when 5 evaluators are used.evaluators are used.

Empirical evidence suggests that Empirical evidence suggests that on average 5 evaluators identify on average 5 evaluators identify 75-80% of usability problems.75-80% of usability problems.

23

3 stages for doing 3 stages for doing heuristic evaluationheuristic evaluation

Briefing session to tell experts what to Briefing session to tell experts what to dodo

Evaluation period of 1-2 hours in which:Evaluation period of 1-2 hours in which:- Each expert works separately- Each expert works separately- Take one pass to get a feel for the - Take one pass to get a feel for the productproduct- Take a second pass to focus on specific - Take a second pass to focus on specific

featuresfeatures Debriefing session in which experts work Debriefing session in which experts work

together to prioritize problemstogether to prioritize problems

24

Advantages and Advantages and problemsproblems

Few ethical & practical issues to Few ethical & practical issues to considerconsider

Can be difficult & expensive to find Can be difficult & expensive to find expertsexperts

Best experts have knowledge of Best experts have knowledge of application domain & usersapplication domain & users

Biggest problemsBiggest problems- important problems may get missed- important problems may get missed- many trivial problems are often - many trivial problems are often identifiedidentified

25

Cognitive walkthroughsCognitive walkthroughs

Focus on ease of learningFocus on ease of learning Designer presents an aspect of the Designer presents an aspect of the

design & usage scenariosdesign & usage scenarios One of more experts walk through One of more experts walk through

the design prototype with the the design prototype with the scenarioscenario

Expert is told the assumptions Expert is told the assumptions about user population, context of about user population, context of use, task detailsuse, task details

Experts are guided by 3 questionsExperts are guided by 3 questions

26

The 3 questionsThe 3 questions Will the correct action be sufficiently Will the correct action be sufficiently

evident to the user?evident to the user? Will the user notice that the correct Will the user notice that the correct

action is available? action is available? Will the user associate and interpret Will the user associate and interpret

the response from the action the response from the action correctly? correctly?

As the experts work through the As the experts work through the scenario they note problemsscenario they note problems

27

Pluralistic walkthroughPluralistic walkthrough Variation on the cognitive walkthrough Variation on the cognitive walkthrough

themetheme Performed by a carefully managed Performed by a carefully managed

teamteam The panel of experts begins by The panel of experts begins by

working separatelyworking separately Then there is managed discussion that Then there is managed discussion that

leads to agreed decisionsleads to agreed decisions The approach lends itself well to The approach lends itself well to

participatory designparticipatory design

28

Key pointsKey points Structured, unstructured, semi-structured Structured, unstructured, semi-structured

interviews, focus groups & questionnairesinterviews, focus groups & questionnaires Closed questions are easiest to analyze & Closed questions are easiest to analyze &

can be replicatedcan be replicated Open questions are richerOpen questions are richer Check boxes, Likert & semantic scalesCheck boxes, Likert & semantic scales Expert evaluation: heuristic & walkthroughsExpert evaluation: heuristic & walkthroughs Relatively inexpensive because no usersRelatively inexpensive because no users Heuristic evaluation relatively easy to learnHeuristic evaluation relatively easy to learn May miss key problems & identify false onesMay miss key problems & identify false ones

29

Testing & Testing & modeling usersmodeling users

30

The aimsThe aims

Describe how to do user testing.Describe how to do user testing. Discuss the differences between user testing, Discuss the differences between user testing,

usability testing and research experiments.usability testing and research experiments. Discuss the role of user testing in usability Discuss the role of user testing in usability

testing.testing. Discuss how to design simple experiments.Discuss how to design simple experiments. Describe GOMS, the keystroke level model, Describe GOMS, the keystroke level model,

Fitts’ law and discuss when these techniques Fitts’ law and discuss when these techniques are useful.are useful.

Describe how to do a keystroke level analysis.Describe how to do a keystroke level analysis.

31

Experiments, user Experiments, user testing & usability testing & usability

testingtesting Experiments test hypotheses to discover new Experiments test hypotheses to discover new

knowledge by investigating the relationship knowledge by investigating the relationship between two or more things – i.e., variables. between two or more things – i.e., variables.

User testing is applied experimentation in User testing is applied experimentation in which developers check that the system which developers check that the system being developed is usable by the intended being developed is usable by the intended user population for their tasks.user population for their tasks.

Usability testing uses a combination of Usability testing uses a combination of techniques, including user testing & user techniques, including user testing & user satisfaction questionnaires.satisfaction questionnaires.

32

User testing is not User testing is not researchresearch

User testingUser testing Aim: improve Aim: improve

productsproducts Few participantsFew participants Results inform designResults inform design Not perfectly Not perfectly

replicablereplicable Controlled conditionsControlled conditions Procedure plannedProcedure planned Results reported to Results reported to

developersdevelopers

Research Research experimentsexperiments

Aim: discover Aim: discover knowledgeknowledge

Many participantsMany participants Results validated Results validated

statistically statistically ReplicableReplicable Strongly controlled Strongly controlled

conditionsconditions Experimental designExperimental design Scientific paper reports Scientific paper reports

results to communityresults to community

33

User testingUser testing Goals & questions focus on how well Goals & questions focus on how well

users perform tasks with the productusers perform tasks with the product Comparison of products or prototypes Comparison of products or prototypes

commoncommon Major part of usability testingMajor part of usability testing Focus is on time to complete task & Focus is on time to complete task &

number & type of errorsnumber & type of errors Informed by video & interaction Informed by video & interaction

logginglogging User satisfaction questionnaires User satisfaction questionnaires

provide data about users’ opinionsprovide data about users’ opinions

34

Testing conditionsTesting conditions Usability lab or other controlled spaceUsability lab or other controlled space Major emphasis onMajor emphasis on

- selecting representative users- selecting representative users- developing representative tasks- developing representative tasks

5-10 users typically selected5-10 users typically selected Tasks usually last no more than 30 Tasks usually last no more than 30

minutesminutes The test conditions should be the same The test conditions should be the same

for every participantfor every participant Informed consent form explains ethical Informed consent form explains ethical

issuesissues

35

Type of data Type of data (Wilson & Wixon, (Wilson & Wixon,

‘97)‘97) Time to complete a taskTime to complete a task Time to complete a task after a specified Time to complete a task after a specified

time away from the producttime away from the product Number and type of errors per taskNumber and type of errors per task Number of errors per unit of timeNumber of errors per unit of time Number of navigations to online help or Number of navigations to online help or

manualsmanuals Number of users making a particular Number of users making a particular

errorerror Number of users completing task Number of users completing task

successfullysuccessfully

36

How many participants How many participants is enough for user is enough for user

testing?testing? The number is largely a practical The number is largely a practical

issueissue Depends on:Depends on:

- schedule for testing- schedule for testing- availability of participants- availability of participants- cost of running tests- cost of running tests

Typical 5-10 participants Typical 5-10 participants Some experts argue that testing Some experts argue that testing

should continue until no new insights should continue until no new insights are gainedare gained

37

ExperimentsExperiments Predict the relationship between Predict the relationship between

two or more variablestwo or more variables Independent variable is Independent variable is

manipulated by the researchermanipulated by the researcher Dependent variable depends on Dependent variable depends on

the independent variablethe independent variable Typical experimental designs have Typical experimental designs have

one or two independent variableone or two independent variable

38

Experimental designsExperimental designs Different participants - single group Different participants - single group

of participants is allocated randomly of participants is allocated randomly to the experimental conditionsto the experimental conditions

Same participants - all participants Same participants - all participants appear in all conditionsappear in all conditions

Matched participants - participants Matched participants - participants are matched in pairs, e.g., based on are matched in pairs, e.g., based on expertise, gender expertise, gender

39

Advantages & Advantages & disadvantagesdisadvantages

Design Advantages Disadvantages

Different No order effects Many subjects & individual differences a problem

Same Few individuals, no individual differences

Counter-balancing needed because of ordering effects

Matched Same as different participants but individual differences reduced

Cannot be sure of perfect matching on all differences

40

Predictive modelsPredictive models Provide a way of evaluating Provide a way of evaluating

products or designs without products or designs without directly involving usersdirectly involving users

Psychological models of users are Psychological models of users are used to test designsused to test designs

Less expensive than user testingLess expensive than user testing Usefulness limited to systems with Usefulness limited to systems with

predictable tasks - e.g., telephone predictable tasks - e.g., telephone answering systems, mobiles, etc.answering systems, mobiles, etc.

Based on expert behaviorBased on expert behavior

41

GOMS GOMS (Card et al., 1983)(Card et al., 1983)

Goals - the state the user wants to achieve Goals - the state the user wants to achieve e.g., find a websitee.g., find a website

Operators - the cognitive processes & Operators - the cognitive processes & physical actions performed to attain those physical actions performed to attain those goals, e.g., decide which search engine to goals, e.g., decide which search engine to useuse

Methods - the procedures for accomplishing Methods - the procedures for accomplishing the goals, e.g., drag mouse over field, type the goals, e.g., drag mouse over field, type in keywords, press the go buttonin keywords, press the go button

Selection rules - determine which method to Selection rules - determine which method to select when there is more than one select when there is more than one availableavailable

42

Benefits and limitations Benefits and limitations of GOMSof GOMS

Help make decisions about the Help make decisions about the effectiveness of new productseffectiveness of new products

Allow comparative analysis to be Allow comparative analysis to be performed for different interfacesperformed for different interfaces

Difficult or impossible to predict how Difficult or impossible to predict how an average user will carry out their an average user will carry out their taskstasks

43

Keystroke level modelKeystroke level model

GOMS has also been developed GOMS has also been developed further into a quantitative model - further into a quantitative model - the keystroke level model.the keystroke level model.

This model allows predictions to This model allows predictions to be made about how long it takes be made about how long it takes an expert user to perform a task. an expert user to perform a task.

44

Response times for Response times for keystroke level operatorskeystroke level operators

Operator Description Time (sec)K Pressing a single key or button

Average skilled typist (55 wpm)Average non-skilled typist (40 wpm)Pressing shift or control keyTypist unfamiliar with the keyboard

0.220.280.081.20

P

P1

Pointing with a mouse or other device on adisplay to select an object.This value is derived from Fitts’ Law which isdiscussed below.Clicking the mouse or similar device

0.40

0.20H Bring ‘home’ hands on the keyboard or other

device0.40

M Mentally prepare/respond 1.35R(t) The response time is counted only if it causes

the user to wait.t

0.35

45

Fitts’ Law Fitts’ Law (Paul Fitts 1954)(Paul Fitts 1954)

The law predicts that the time to The law predicts that the time to point at an object using a device is a point at an object using a device is a function of the distance from the function of the distance from the target object & the object’s size. target object & the object’s size.

The further away & the smaller the The further away & the smaller the object, the longer the time to locate it object, the longer the time to locate it and point.and point.

Useful for evaluating systems for Useful for evaluating systems for which the time to locate an object is which the time to locate an object is important such important such as handheld devices like mobile as handheld devices like mobile phonesphones

46

Fitt’s LawFitt’s Law

T = k ln(D/S+0.5) k~100msT = k ln(D/S+0.5) k~100ms T: time to move the hand to a targetT: time to move the hand to a target D: the distance between hand and D: the distance between hand and

targettarget S: size of targetS: size of target

47

Key pointsKey points User testing is a central part of usability testing User testing is a central part of usability testing Testing is done in controlled conditionsTesting is done in controlled conditions User testing is an adapted form of experimentationUser testing is an adapted form of experimentation Experiments aim to test hypotheses by manipulating Experiments aim to test hypotheses by manipulating

certain variables while keeping others constantcertain variables while keeping others constant The experimenter controls the independent variable(s) The experimenter controls the independent variable(s)

but not the dependent variable(s)but not the dependent variable(s) There are three types of experimental design: There are three types of experimental design:

different-participants, same- participants, & matched different-participants, same- participants, & matched participantsparticipants

GOMS, Keystroke level model, & Fitts’ Law predict GOMS, Keystroke level model, & Fitts’ Law predict expert, error-free performanceexpert, error-free performance

Predictive models are used to evaluate systems with Predictive models are used to evaluate systems with predictable tasks such as telephonespredictable tasks such as telephones