Asking Users and ExpertsAsking Users and Experts

Yujia ZHU

Yimeng DOU

Asking UsersAsking Users

Interviews Questionnaires

Asking Users --- InterviewAsking Users --- Interview

Interviews can be thought of as a “conversation with a purpose” (Kahn and Cannell, 1957)

How much the interview is like an ordinary conversation depends on the questions to be answered and the type of interview to be used.

Guideline to develop questionsGuideline to develop questions

Plan to keep the questions short, straightforward and avoid to ask too many

– Avoid long questions– Avoid compound sentences– Avoid using jargon– Avoid leading questions like “Why do you like

it?”– Be alert to unconscious biases

Guideline to plan an interviewGuideline to plan an interviewTry to make the interview as pleasant for interviewees as

possible and make the interviewee feel comfortable

– Introduction– Warmup session– Main session– Cool-off period– Closing session

Conduct InterviewsConduct Interviews

The golden rule is to be professional

– Dress in a similar way to the interviewees as possible, dress neatly and avoid standing out

– Prepare an informed consent form, signature– Make sure your recording equipment works– Record answers exactly

4 types of interviews4 types of interviews

Open-ended or unstructured Structured Semi-structured Group interviews

Unstructured InterviewUnstructured Interview Feature:

– Interviewers have less control on the process

– Open (The format and content of answers is not predefined)

Benefit:– Generate rich data

Disadvantage:– Data is very time-consuming and difficult to analyze

– Impossible to replicate the process

Structured InterviewStructured Interview

Feature– The interviewers have more control

– Typically, the questions are closed To work best:

– Questions need to be short and clearly worded

– Questions should be refined by asking another evaluator to review the questions and run a small pilot study

Choosing type of interviews according to the evaluation goals and questions to ask

Semi-Structured InterviewsSemi-Structured Interviews Feature

– Combine features of structured and unstructured interviews.

– The interviewers start with preplanned questions and then probes the interviewee to say more

Some ways to improve the interview– Neutral Probes are a device for getting more

information– Prompt the person to help him/herself along– Accommodate silences

Probing and prompting should aim to help the interview WITHOUTWITHOUT introducing biases

Group InterviewsGroup Interviews Focus Group: Normally three to 10 people are

involved. Benefit:

– It allow diverse or sensitive issues to be raised

– High validity

– Low-cost, quick results, easily be scaled Disadvantage:

– Need a skillful facilitator

– Difficult to get people together in a suitable location and time

Data analysis and Data analysis and interpretation in Interviewsinterpretation in Interviews

Analysis of unstructured interviews can be time-consuming, though their contents can be rich

Data from structured interviews is usually analyzed quantitatively

Asking Users - QuestionnairesAsking Users - Questionnaires

A well established technique for collecting demographic data and users’ opinion

Questionnaire vs. Interview– Questionnaire can be distributed to a large

number of people– Interview are easy and quick to conduct

Designing questionnairesDesigning questionnaires basic demographic information General, specific questions Advices for designing a questionnaire

– Make question clear and specific– Ask closed questions and offer a range of answers– Include a “no-opinion” option– Ordering of questions– Avoid complex multiple questions– Provide appropriate range– Ordering of scales should be intuitive and consistent– Avoid jargon– Clear instructions– Balance between using white space and the need to keep the

questionnaire as compact as possible

Question and respond formatQuestion and respond format Different types of responses

– Discrete responses (“Yes” or “No”)– Locate the answer within a range– A single preferred opinion

Commonly used formats– Check boxes and ranges

Appropriate range

– Rating scales Likert Semantic differential scales

Likert ScalesLikert Scales Used for measuring opinions, attitudes, and beliefs Widely used for evaluating user satisfaction with

products Example:

The use of color is excellent:Strongly agree agree OK disagree strongly disagree

Designing Likert ScalesDesigning Likert Scales

Gather a pool of short statements about the features of the product that are to be evaluated

Divide the items into groups with about the same number of positive and negative statements in each group

Decide on the scale

Select items for the final questionnaire and reword as necessary to make them clear

Semantic differential scalesSemantic differential scales Explore a range of bipolar attitudes about a

particular item Each pair of attitudes is represented as a pair of

adjectives Example: (evaluation for a homepage)

Attractive UglyClear ConfusingDull ColorfulExciting BoringAnnoying Pleasing…… ……

Administering questionnairesAdministering questionnaires Two important issues

– Reaching a representative sample of participants– Ensuring a reasonable response rate

Some ways to encourage a good response – Ensuring the questionnaire is well designed– Providing a short overview section– Including a stamped, self-addressed envelope for its return– Explaining why you need the questionnaire to be completed

and assuring anonymity– Contacting respondents through a follow-up letter, phone

call or email– Offering incentives such as payments

Online questionnairesOnline questionnaires Advantages:

– Quick responses– Low-cost to copying and postage– Immediate transferring data– Short time to require data for analysis– Easily correct error in questionnaire design

Two types: Email vs. Web-based– Email: target specific users– Web-based: more flexible, can use check boxes,pull-down

and pop-up menus, help screens, and graphics Problem:

– Obtaining a random sample– Response rates may be lower

Developing a web-based Developing a web-based questionnairequestionnaire

3 steps– Designing it on paper, following the general guidelines – Developing strategies for reaching the target population– Turning the paper version into a web-based version

Produce an error-free interactive electronic version from the paper version

Make the questionnaire accessible from all common browsers and readable from different-size monitors and different network locations

Make sure information identifying each respondent will be captured and stored confidentially

User-test the survey with pilot studies before distributing

Analyzing questionnaire dataAnalyzing questionnaire data Identify any trends and patterns

– Use a spreadsheet like excel to hold the data Simple statistics

– Number or percentage of responses in a particular category

Bar charts can also be used to display data graphically

More advanced statistical techniques such as cluster analysis

Asking ExpertsAsking Experts

When users are not accessible, or involving When users are not accessible, or involving them is too expensive,we can ask experts or them is too expensive,we can ask experts or combination of experts and users to provide combination of experts and users to provide feedback.feedback.

Various Inspection Various Inspection TechniquesTechniques

Heuristic evaluations Walkthroughs

Experts inspect the human-computer interface and predict problems users would have when interacting with it.

AdvantagesAdvantages Relatively inexpensive Easy to learn Effective Can be used at any stage of a design project

**Usually when using heuristic evaluation, five evaluators can identify around 75% of the total usability problems.

Heuristic EvaluationHeuristic Evaluation

Developed by Jakob Nielsen and colleagues It’s an informal usability inspection

technique. Experts are guided by a set of usability

principles. These principles are known as heuristics.

Experts evaluate whether user-interface elements conform to those principles.

Heuristics (1)Heuristics (1)

Visibility of system status

Match between system and the real world

User control and freedom

Consistency and standards

Help users recognize, diagnose, and recover from errors

Heuristics (2)Heuristics (2)

Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help and documentation

Different Heuristics For Different Heuristics For Specific PurposesSpecific Purposes

Core heuristics are too general Following you will see an example of a set of

heuristics for website. There are also heuristics for evaluating toys, WAP

devices, online communities, wearable computers, etc.

These heuristics are developed by tailoring Nielson’s heuristics and market research, etc.

HOMERUN—Heuristics For HOMERUN—Heuristics For Commercial WebsitesCommercial Websites

High-quality content Often updated Minimal download time Easy of use Relevant to user’s needs Unique to the online medium Netcentric corporate culture

How To Do Heuristic How To Do Heuristic EvaluationEvaluation

Briefing Session Evaluation Period Debriefing Session

How To Do Heuristic How To Do Heuristic EvaluationEvaluation

Tell experts what to do.Use a Tell experts what to do.Use a prepared script as a guide.prepared script as a guide.

Evaluation Period Debriefing Session

How To Do Heuristic How To Do Heuristic EvaluationEvaluation

Briefing Session Two passes: 1st pass, gain some Two passes: 1st pass, gain some

feeling of the system. 2nd pass, feeling of the system. 2nd pass, focus on specific interface focus on specific interface elements.elements.

Debriefing Session

How To Do Heuristic How To Do Heuristic EvaluationEvaluation

Briefing Session Evaluation Period Experts come together to Experts come together to

discuss their findings and to discuss their findings and to prioritize the problems found prioritize the problems found and suggest solutions.and suggest solutions.

Heuristic EvaluationHeuristic Evaluation

Selecting appropriate heuristics is very important

Because users are not involved, there are fewer practical and ethical issues.

A week is often cited as the time needed to train experts to be evaluators

Dilemma: Problems or Dilemma: Problems or False Alarms?False Alarms?

Different approaches often identify different problems. Sometimes heuristic evaluation misses severe problems.

About 33% reported problems are real usability problems. Heuristic evaluation misses about 21% of user’s problems.

43% are not problems at all.

How to reduce the number of How to reduce the number of false alarms and missed false alarms and missed

problems?problems?

Use complementary user testing techniques along with heuristic evaluation.

Check if experts really have the expertise that they claim.

Have several evaluators to avoid one person’s bias or poor performance.

Heuristic Evaluation Of Heuristic Evaluation Of WebsitesWebsites

In 1999, usability consultant Keith Cogdill was commissioned by NLM to evaluate MEDLINEplus. He identified seven heuristics.

These heuristics are given to 3 experts who independently evaluated MEDLINEplus.

Heuristics used by CogdillHeuristics used by Cogdill

Internal Consistency Simple Dialog Shortcuts Minimizing Memory

Load

Preventing Errors Feedback Internal locus of

control

*What heuristics would we use to analyze ICS website?*What heuristics would we use to analyze ICS website?

Results Of The StudyResults Of The Study

Layout: Uncomplicated Vertical Design; well suited for printing; conservative using of graphics

Internal Consistency: Formatting of pages and logo is consistent across the website.

Arrangement of health topics: Problems should be arranged alphabetically as well as in categories.

Depth of navigation menu: increase the fan-out in navigation menu in the left margin.

Summary of Heuristics Summary of Heuristics for Web Designfor Web Design

Navigation Avoid orphan pages Avoid excessive white

space resulting in long page Provide navigation support Avoid narrow, deep or

hierarchical menus Avoid non-standard link

colors Provide consistent look and

feel for navigation

AccessAvoid complex URLsAvoid long download

times

Information Designcontent comprehension

and aesthetics

Heuristics For…Heuristics For…

For online communitiesSociabilityUsability

For other devices (handhelds, computerized toy)

Another Technique: Another Technique: WalkthroughsWalkthroughs

Cognitive Walkthroughs

Pluralistic Walkthroughs

Cognitive Walkthroughs—Cognitive Walkthroughs—DefinitionDefinition

Simulating user’s problem-solving process at each step in the human-computer dialog

Checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action

They focus on evaluating designs for case of learning

Cognitive Walkthroughs—Steps Cognitive Walkthroughs—Steps (1-3)(1-3)

1. Identify characteristics of typical users; Develop sample tasks; Produce the interface’s prototype, or a description of it; Generate a clear sequence of the actions needed for the users to complete the task

2. A designer and one or more experts begin the analysis

3. Evaluators walk through the sequences for each task, and try to answer the following questions:

Will users know what to do, see how to do it, and understand from feedback whether the action was correct or not?

4. Record critical information, including: Assumptions about what would cause

problems and why Notes about side issues and design changes A summary of the results

5. Revise the design according to the results.

Cognitive Walkthroughs—Steps Cognitive Walkthroughs—Steps (4-5)(4-5)

Example: Finding a book at Example: Finding a book at Amazon.comAmazon.com

Let’s walk through the process of finding a book at

Task: find a book at Typical user: students who use web regularly

Specific Steps, Questions and Answers

Pros and Cons of Cognitive Pros and Cons of Cognitive WalkthroughsWalkthroughs

It takes longer than heuristic evaluation for the same part, because it examines each step of a task.

You may get much more detailed information from the cognitive walkthrough.

It’s useful to examine small part of a system; whereas heuristic evaluation is useful for examining a whole system.

Pluralistic Walkthroughs--Pluralistic Walkthroughs--DefinitionDefinition

Another type of walkthrough in which users, developers and usability experts work together to step through a scenario, discussing usability issues associated with dialog elements involved in the scenario steps.

Pluralistic Walkthroughs--Pluralistic Walkthroughs--StepsSteps

Develop scenarios in the form of a series (usually 1 or 2) of hard-copy screens representing a single path through the interface.

A panelist ask evaluators to write down the sequence of actions they take to move from one screen to another.

Discuss the actions from that round of review. (Users-> Experts-> Designers)

Move on to next round of screens. The process continues until all the scenarios have been evaluated.

Benefits and Constraints of Benefits and Constraints of Pluralistic WalkthroughsPluralistic Walkthroughs

Strong focus on user’s task Performance data is produced and many

designers like the apparent clarity of quantitative data.

Involving a multidisciplinary team Constraint: Limited screens each time, so it

takes relatively long time to complete.

Thanks, that’s all!Thanks, that’s all!