Asking Users and Experts

Bobby Kotzev

Adrian Sugandhi

Outline

Asking Users Interviews Questioners

Asking Experts Inspections Walkthroughs

Interviews, Survey, or Focus Group?

Interview Take significant time and need more resources

Survey Large data set

Focus group Seeking multiple point of views in a shorter period

of time

Preparing for Interviews:Things to be aware

Be alert to unconscious biases Honesty

Social desirability Prestige response bias Sensitive or highly personal topics – survey?

Consider the interviewee as he/she is “Doing you a favor”

Preparing Interviews:Outcome Analyses

Outcome Analyses Plan outcome-based interviews

Associate questions with the goals Capture desired outcomes

Differentiate between outcome and solution Organize the outcomes Rate the outcomes for importance and

satisfactions Importance + (Importance-Satisfaction) = Opportunity

Use the outcomes to jump-start innovation

Preparing Interviews:Pre-interview Identify the objective of the study Select type of interviews Medium of interview (in person/phone) Decide how you will analyze the data Write the questions

Avoid long questions (Brevity) Avoid compound sentences Avoid using jargon Avoid leading questions Avoid Biases Clarity Avoid future prediction questions Avoid inaccessible topics

Test your questions

Preparing Interviews:Players in the Activity

Participants 6-10 of each user type (diversity)

Interviewers Ensure participants understand the questions Must be skilled

The note-taker Interviewers can focus more on body language

The videographer Whenever possible, record the interview session

Conducting Interviews: Sections

Introduction 5-10 Minutes

Warm-up 5-10 Minutes

Main section 85-100 Minutes

Cool off period 5 Minutes

Closing

Preparing for Interviews: Additional Preparations

Run pilot study Be professional

Dress similarly and appropriately Prepare informed consent Check and familiarize yourself with recording

equipment Record answers exactly

Interviews Types

Interview Types Open-ended Unstructured Structured Semi-structured

Unstructured Interviews

A conversation that focus on a particular topic Open questions Can generate rich data which the interviewer

haven’t thought about Generates a lot of unstructured data Impossible to replicate Hard to analyze

Unstructured Interviews: Prepare

Have an agenda Be prepared to follow new lines Pay attention to ethical details Respond with sympathy but make sure to

avoid putting ideas in the user’s head Analyze data as soon as possible after the

interview

Structured Interviews

Predetermined questions, similar to a questionnaire.

Semi-Structured Interviews

Starts as a structured interview but can change direction and inject open-ended questions depending on responses

Avoid preempting answers Probes – “Do you want to tell me anything

else about…”

Interviewer’s Role Do not interrupt Keep on track

Unstructured interview easy goes off-track Silence is golden

“permission” to provide more detail Remain attentive Asking the tough questions

Wait until you develop rapport Using examples Watch for generalities Do not force choices

Interviewer’s Role (cont’d)

Watch for markers Key events to probe for more rich information

Select the right type of probes Know when to move on Reflecting

Summarize, reword, or reflect responses Empathy and antagonism Transitions

Transition smoothly from one topic to another

Monitoring the Relationships with the Interviewee/Participant

Watch participant’s body language Nervous, tenses – go to easier question or restate

purpose/motivation of the study Fighting for control

Ask yourself why one refusing your questions Hold your opinions Dos and Don’ts

Interview: Data Analysis and Interpretation

Must be analyzed shortly after each interview Categorizing Affinity diagram Qualitative analysis tools

Look for patterns

Interview: Communicate the Findings

Over time By topic By participant Vehicles for communicating the results

Summarized poster Identify follow-up activities based on the results Table of recommendations (uncovered issues and

next steps)

Questionnaires

Good, established technique for collecting demographic data and users opinion

Can be Closed and Open. Can be distributed to large number of people

Questionnaires: Designing Start with basic demographic data Make questions clear and specific When possible ask closed questions and offer a range of

answers Include no-opinion option Ordering matters Avoid complex and compound questions Use intuitive, consistent scaling Avoid jargons Provide clear instructions how to complete the questioner

Questionnaires: Question Types

Checkboxes Likert Scales

Semantic differential scales

Gender: MaleFemale

Questionnaires: Administering

Well designed Include stamped self addressed envelope Provide a short version Contact and respondents trough a follow-up,

mail, e-mail phone Offer incentives like payments

Questionnaires: Online Forms Pros:

Reach large audience Response is fast No postage costs Data is already in a electronic format and can be imported

easily in DB Errors can be corrected easily and fast

Cons: Obtaining random sample is difficult – web users represent

only a certain demographic. Participants are self selecting – proclaimed nonscientific

Questionnaires: Online Forms (cont’)

Construct Online questioners based on their paper forms

Run pilot studies

Asking Experts

Experts will be power users but also could be involved with the development of similar, successful products so they have an deeper understanding.

Asking Experts: Inspections

Heuristic evaluations evaluate user-interface against a predetermined guidelines

and principles

Visibility of system statusAre users kept informed about what is going on?Is appropriate feedback provided within reasonable time about a user action?

User Control and freedom Are there ways of allowing users to easily escape from places they unexpectedly find themselves in?

Consistency and standards Are the ways of performing similar actions consistent?

Asking Experts: Inspections (cont’)

Different heuristics are needed for different products, devices and software

Using heuristics and experts can reveal issues fast and inexpensively – 5 experts can reveal 75% of issues - graph

Performing heuristic evaluation Spend 1-2 hours with the product. At least two passes – one to get the feel, second to walk

trough the interface and address all aspects of usability. Have a specific task in mind when evaluating functional

products

Asking Experts: Inspections (cont’)

Heuristic evaluation of websites Internal consistency Simple dialog Shortcuts Minimizing the user memory load Preventing errors Feedback Internal locus of control Layout Internal consistency

Asking Experts: Problems

Reporting issues which are not there – (Bill Bailey 2001)

Experts Problem Reporting

34%

22%

44% problems discovered

problems missed

reported non-problems

Asking Experts: Walkthroughs

Walk through a task and to notice problems “Cognitive walkthroughs” – simulating the user

problem solving process for a task Figure if the users will know what to do, how to do it and

weather the action was correct or not. Record assumptions, side issues and summaries of results

Pluralistic walkthroughs Users developers and usability experts get together to step

through a scenario

Summary

Interviews Fresh input

Questionnaires Reach the masses

Experts Easy way to discover most problems