Alex Cummaudo: Sims 1744070 Usability 27/04/2016 · Evaluation Methods Alex Cummaudo: Sims 1744070...

Evaluation Methods Alex Cummaudo: Sims 1744070 Usability 27/04/2016

Usability Evaluation Methods ▸ Finds problems in your prototypes▸ Prevents you from ‘fixing’ a problem that doesn’t exist

Inspection Methods

▸ Look at UI and predict what it’ll be like▸ Too many evaluators reachers asymptote at around 75%▸ ‘Problems’ may turn into false alarms is just bad as actual user problems (waste res.)▸ Heuristic Evaluation

- Evaluation against a set of guidelines (Heuristics) (consistent, minimisation etc.)- Guidelines interpreted by an ‘experts of usability’- Step 1 - Briefing Session:

▪ Give Experts Design Guidelines, User Tasks, User Profiles, User Goals, UI- Step 2 - Analysis:

▪ Experts analyse the UI against the Heuristics against the Users!- Step 3 - Debriefing Session:

▪ Analyse all the Expert’s opinions to produce a report of problems, severity, solutions etc.

▸ Cognitive Walkthrough ▸ Pluralistic Walkthrough

Empirical/Observational Methods

▸ Done with participants/users▸ Controlled

- In a lab etc. (insert notes)▸ Quick and Dirty ▸ Remote Testing ▸ Field Testing

Data Analysis ▸ Analysis the data from your Usability Analysis determines how you should fix problems

in your analysis▸ Different kinds of data (qualitative and quantitate) require different kinds of analysis

- Objective: Based on measurements in the real world (easy to measure)▪ For efficiency: Time (mins, secs) → a measure of efficiency

Page � of �1 3


- Subjective: Based on personal opinions or judgments (hard to measure)▪ For efficiency: ‘This was quick to do’ → a perception of efficiency▪ E.g. installing mirrors in lifts make perception of time quicker (user distracted) ▪ Subjective data can be assigned numbers with rating systems

- Quantitative: Represented by numbers by an amount or frequency (objective)▪ Rating data is the most frequent kind of quantitate data in Usability

- Qualitative: Represented by words or categories; not by a number (subjective)▪ User’s comments or observations you make

▸ Levels of measurements- Nominal = categorical (apples and oranges cannot be averaged; problem severity)- Ordinal = ranking - not a count or frequency; don’t know the difference between

intervals (1° Favourite Feature, 2° FF (not by that much though), 3° FF (hated it) etc)- Interval = rating scales; differences are equal (interval) (SA [ ] [ ] [ ] [ ] [ ] SD)- Rational = zero-point (baseline); (time, money, weight where 0 is least etc.)

▸ What do I report? Standard usability metrics:- Effectiveness: Unassisted task comp. % (0 or 100), # errors, # assists, goal

achievement % - Efficiency: Mean unassisted task completion time, # clicks- Satisfaction: S.U.S scores, qualitative matrices, open questions- Try to find convergence of data; if effectiveness measures are low, then SUS

scores should be low too (otherwise anomaly)

Limitations

▸ Aids in addressing problems in the evaluation to amend them- Do you find all of the problems or just some?- Is the problem a real problem? Is it worth fixing? Why?

▪ Frequency: Common/Rare?▪ Impact: Easy to overcome/Difficult to overcome?▪ Persistence: Onetime problem/Ongoing problem?▪ Rate your problems on a severity scale:

▹ None: not a usability problem at all▹ Cosmetic: amend only if there’s extra time (navigational error)▹ Minor: fixing this problem needs low priority; unimportant but needs fixing▹ Major: fixing this problem needs high priority; important to fix▹ Catastrophic: imperative to fix this before product is released (data loss)

▪ A catastrophic error is worth more than a cosmetic error (do not average)Page � of �2 3


- Types of limitations:▪ Number of participants

▹ Ideally, should have ~5-6 but ideally 12 (a tradeoff between resources and power)

▹ Your participants may just be the few anomalies if sample sizes are small - 5 users usually find 50% of problems only

▪ Representative Users▹ Would the ‘real’ user be worried by these problems?

▪ Representative Tasks ▹ What tasks are the ‘important’ tasks to test?▹ Test frequent, important and difficult tasks

▪ Ecological validity▹ Is the testing environment the same as the ‘real’ environment?

◦ Physical: A noisy, dully lit street vs. quiet, brightly lit lab would affect efficiency, effectiveness etc.

◦ Technical: different browsers, internet connections, prototype fidelity ▪ Bias: results not due to the interface itself

▹ Hawthorne effect: users know they’re being tested & change their behaviour◦ Make them feel relaxed and comfortable ◦ After a while, they habituate which is good :D

▹ Power relationships: a relationship exists between investigator and participant, independent to the test◦ Friendships, boss > worker etc.

▹ Giving the participant a tour: you’ve given away all the answers▹ Task instructions: do to this task, click this button and slide that up—it’s

given away all the clues

Page � of �3 3

Lecture #1 Alex Cummaudo: Sims 1744070 Defining Usability & Ucd 27/04/2016

Defining Usability ‣ ISO Definition:

- extent to which a product (what) can be used by specific users (who) to achieve specified goals (why) with effectiveness, efficiency and satisfaction (how) in a specified environment (where)

‣ In summary:- WHAT - The Product

▪ E.g. computers, phones, ATMs, public transport, government (eTax)- WHO - The User

▪ Related Users - Just because the user suits A, doesn't mean that it suits B▪ Address commonalities within all the users, then look at the differences while

matching consideration for commonalities - WHY - Their Goals

▪ What the solution achieves for the user- HOW - Effectiveness/Efficiency/Satisfaction

▪ Effectiveness = Accuracy and completeness (binary: yes or no)□ Are goals met without many mistakes?□ “Does the car work?”

▪ Efficiency = Relative resource expenditure (value)□ Are goals met without much effort?□ E.g. time, button clicks, mental effort □ “Does the car drive fast?”

▪ Satisfaction = Comfort and acceptability□ Is the user satisfied with that outcome? Surveys!!□ Is the process to get to that outcome easy/horrible?□ Just because something is effective and efficient ≠ satisfied

➡ Driving game: Effective with a road and moving car, efficient with single straight road and no traffic ≠ satisfying game

- WHERE - Their environment ▪ How does the environment affect the user?

□ A noisy factory that sounds a single beep for the fire alarm

Page � of �1 4


User-centred design (UCD) ‣ Quality Software meets stated and implied needs

- Stated need: "Want a portable toaster"- Implied need: "Want something that won't electrocute me"- (Read up on an area if you're not familiar with it)

‣ Usability is part of 6 quality software attributes- UCD one method to get good usability; others exist

- � - [P] 1: Plan UCD

▪ Management buy-in▪ Budget (resources, cash, equipment)▪ Research▪ Expertise needed

- [C] 2: Understand and specify context of use▪ User profiling▪ Clients▪ Environments (physical, social, technical (see lecture 2))

- [U O R ] 3: Specify User and Organisational Requirements ▪ iPhone app where users travel? Make big buttons so won’t miss button on go

- [D] 4: Produce design solutions▪ Prototyping

- [E] 5: Evaluate design against requirements▪ Test with users in a simulated environment▪ In-vitro environments?

‣ Principles of UCDPage � of �2 4

*


- Actively Involve Users▪ Gather info on users and goals (e.g. focus groups)▪ Let them design what they want (e.g. domain expert)▪ Evaluate with them (e.g. surveys)

- Appropriately allocate the varying functions between user and system▪ Balance between what the user and system can do▪ Users can buy things▪ System can look up things▪ Personalisation solves this problem: if users like system to do something, let

system do something!- Iterate varying designs

▪ Design something → get feedback → use feedback to improve design▪ Try many ideas to find the best one

- Multidisciplinary design teams▪ Domain specialists (where should things go?)▪ Graphic designers (how should things look?)▪ Coders (how should things work?)

‣ Activities within UCD- Plan:

▪ Management buys-in to usability?▪ Where is usability within whole dev?▪ Budget for usability?

- Understand & specify environment▪ Who are the users, their goals, environment?

- Specify usability requirements▪ The effectiveness, efficient, satisfaction

- Produce designs▪ Iterating varying designs

- Evaluate design requirements: ▪ What does the users feedback say?▪ How can we use that to improve our system?

Page � of �3 4


‣ Usability is good value for money:

- � ▪ A change made earlier on costs nothing▪ A change made after deployment costs a lot

- Increases productivity▪ 1 sec saved x 100 000 transactions = a lot of time saved

- Less errors = less stress = increased productiveness = increased profit- Less customer support = less employees needed = decreased expenditure

‣ Final comments:‣ "Usability is not the icing on the cake"

- Make a pumpkin cake for a child. You then find that child likes chocolate. Dumping chocolate icing is not going to fix the problem. Once he takes a bit he’ll spit it out!

Usability prevents costly late design changes.

Usability has the greatest impact in the early stages of development.

Usability is not the icing on the cake.

Intended Learning Outcome 2

HIT2316/6316 Usability

Discuss different methods of researching, analysing and documenting context of use (i.e., users, goals and tasks and environment).

Modules:

‣ 02 Context of use

‣ 03 Data Gathering

Page � of �4 4

Lecture #2 Alex Cummaudo: Sims 1744070 Users, Goals & Tasks & Actions, Environments 27/04/2016

2.1 Know Thy User

2.1.1 Characteristics of Users ‣ Characteristics affect behaviour and hence interaction with UI:

- Visible (Physical): age, gender, height, voice ▪ E.g.: poor eyesight = small fonts hard to read

- Non-visible (Psychological): memories, experiences, cognitive ability▪ E.g.: experience with computers = few mistakes in typing, fast to navigate

‣ 3 Ways to predict how user characteristics affect user behaviour:- Common sense, practitioner experience, psychological research- More in depth in design guidelines and principles later on...

‣ We use constraints and requirements to fix this:

User Characteristic: Fred has poor muscle controlUser Behaviour: Fred can't target the mouse easily on small buttonsDesign Constraint: Make buttons larger with lots of spacing so its easier for him

2.1.2 Documenting User Characteristics ‣ User profiles

- Clear ranges of who may use the product, don't focus on people outside this range

- Tabulated- Good to show data from a survey

‣ User personas - Average out data from your user profiles to produce a fictional generic user- Develop empathy:

▪ Include a head shot to make them seem real▪ A background story / quotes to understand their motivations and attitudes

- State user characteristics to understand design constraints- Conduct interviews and observe user to get good data

‣ Make a poster persona: attractive poster and will remind you who your users are.‣ Make an anti-user

- Who not to develop for- If the anti user likes it, then we're developing for the wrong person!

‣ Make mind maps to summarise everything about your user!

Page � of �1 6


2.1.3 Identifying Your Users ‣ Primary, secondary and tertiary users

- Primary: end user/hands-on user- Secondary: uses product via intermediary or infrequently- Tertiary: doesn't use the product but depends on the secondary user

Marguerite (primary user) uses Excel to make spreadsheets for Jerry, the middle manager.Jerry (secondary user) depends on Excel to work to reports the output to the CEO, Fred.Fred (tertiary user) doesn’t touch Excel but gains information from its output via Jerry.

‣ Clients ≠ users- Don’t treat clients as users- E.g. Healthbook: Aus Government is the client but the primary users are patients!)

‣ Don't treat yourself as the primary user‣ Don't aim for generic users; it's difficult to make everyone happy ‣ 5.2 Cooper et al.:

- When there are differences:▪ Find commonalities in your user base and focus on that first. ▪ Focus on their differences while still considering the commonalties in mind▪ Bring as much together so everyone is satisfied, not just partially satisfied.

When you try to cater for everyone things go badly:

#

# This car would hardly be functional ^

Secondary User Tertiary User

Hands on user Uses the product via intermediary or very infrequently

Have a stake, but usually does not use the product

Primary User

Personal Assistant

Middle ManagerCEO, government,

corporation,

developers...

Traps for the unwary: Do not try to make an interface that will make all users happy‣ it is very difficult to make everyone happy!

Cooper et al Figure 5.2

Copper Figure 5.1

Secondary User Tertiary User

Hands on user Uses the product via intermediary or very infrequently

Have a stake, but usually does not use the product

Primary User

Personal Assistant

Middle ManagerCEO, government,

corporation,

developers...

Traps for the unwary: Do not try to make an interface that will make all users happy‣ it is very difficult to make everyone happy!

Cooper et al Figure 5.2

Copper Figure 5.1

Page � of �2 6


2.2 Goals, tasks and actions ‣ Goals: the motivation behind users using the product (either long-term or short)‣ Tasks: things to do in order to get the goals done‣ Sub-tasks: a decomposed task; processes within the task‣ Actions: automated processes within the sub task that do not need a decision

Goals: Get a HD in Usability (ST) / Become successful (LT)Tasks: Attend Lectures and TutorialsSub-Tasks: Commute to station, listen to lecturer, attempt tutorial questionsActions: Listen, read, write, walk, breathe

‣ Users are stupid:

- Remember, a user sometimes doesn't know what they want- Satisfy their implied needs

▪ Users can sometimes become too focused on the tasks at hand▪ Take a step back and try and determine the goals at the right level (we want

lower level goals):

Stated Need: Fred wants to use FTP to on DropboxImplied Need: Fred really wants an easy way to upload filesAs a result: Dropbox instead develops drag-and-drop for browsers for easy upload

Page � of �3 6


2.2.1 Documenting Tasks ‣ Hierarchal task descriptions

- Displays a goal with tasks and suntans in order to complete that goal

- � ‣ Story board

- Shows the bigger picture: context of use

- � ‣ Essential use case

- � - Links up what the user and system is responsible for- Fulfils UCD principle (determining the functionality of the system versus the user)- Identifies the underlying structure of interaction - Don't make the user do what the system could- Don't insert lots of detail in regards to implementation

Show photos to

friend

Take photo Print photo Display photo

Take file to photo

shop

Photo shop prints

photoPay for photo Take photo home

Take money from

walletPay cashier

Budget BudgieOn-Line Photo Shop

Hierarchical Task Description: Print images at shop


Story Board: Print images from website




user purpose system responsibility

transfer image files to system

receive image files

print images

pay

request payment

receive payment

request delivery of prints deliver prints

receive prints

select images to print

Essential Use Case: Print images

Show photos to

friend


Take file to photo

shop

Photo shop prints


Take money from

walletPay cashier










receive image files

print images

pay

request payment

receive payment


receive prints



Show photos to

friend


Take file to photo

shop

Photo shop prints


Take money from

walletPay cashier










receive image files

print images

pay

request payment

receive payment


receive prints



Page � of �4 6


2.2.2 Characteristics of Tasks ‣ Tasks have characteristics

- Frequency, consistency, cooperation and communication, complexity‣ These characteristics can lead to user behaviour; implement constraints /

requirements as a result

Task Characteristic: Task is infrequentUser Behaviour: user forgets how to complete taskDesign Constraint: make the interface easy to relearn

Page � of �5 6


2.3 Environments ‣ Any interface has its own particular environment

- Physical - noise, lighting, space, temperature- Technical - hardware, software, Internet, data storage- Social and organisational - Management structure, privacy, training

‣ These characteristics lead to user behaviour: hence implement design constraints to accompany their environment

Environmental Characteristic: refrigerated warehouse (very cold)User Behaviour: user can't remove their gloves to operate systemDesign Constraint: must be operable with gloves on

Page � of �6 6

Lecture #3 Alex Cummaudo: Sims 1744070 Data Gathering 27/04/2016

3.01 Background Research ▸ Research you can do without users.

Why do background research?

▸ Answers the questions you want to know about users and their domain - Who are the users? What goals do they want to achieve? Tasks?

▸ Prepares for user research (see 3.02)

- Become familiar with the context of use ▪ This makes talking to users easier later on

- Know the gaps of knowledge of the domain you will need to fill▪ Collect as much generic information from textbooks, internet▪ Removes generic questions to ask: get straight into specifics of domain

What do you and your team already know? ▸ Brainstorm the domain with your team: identify what you will need to learn▸ Speak to customer support or HR:

- Know the demographics/statistics from HR- Learn what is currently being complained about

What do publications already show? ▸ Search through:

- user manuals, procedural data▪ BUT, user manuals may become outdated: We never do it like that anymore!!

- ABS for statistics▪ Saves you from having to research your own stats.

- Academic journals▪ Use prior research to save you from researching your own (for user reqs...)

What do competitive analyses show?

▸ What are the things competitors:- Do right or wrong?- Don’t do?

▸ What do their customer’s feedback look like?▸ How can we make it better than them?

Page � of �1 7


3.02 User Research ▸ Research done with primary users; sampling who they are.

Why do User Research? ▸ Understand context of use of your users▸ Make your background research more accurate or complete

Data Gathering Methods

▸ Interviews - A conversation with a purpose (to develop software!!) - To understand the interviewee’s perspective on the issue

▪ BUT Get multiple points of view though, not just one▪ SO Chose the commonalities between all people interviewed…

- Gets stuff that survey questions can't answer▪ Open questioning allows for unexpected answers

- But interviews are time-consuming▪ No. interviews held dependent on how big the user group is▪ Rule of Thumb: When you're not learning anything new, you stop.▪ If too diverse, will have to do tradeoffs:

"Okay we’ve done our 5 interviews, still learning new things, but we don't have the resources to continue”

Hence, the budget constrains the development from getting more users to interview.

- SET UP! ▪ R e c r u i t p o t e n t i a l i n t e r v i e w e e s

▹ Target group: the people you’re

developing for▹ Accessible group: target people who

are available to interview▹ Sample group: the people that can be

handled by your resources◦ Ensure for diverse (gender, culture)

cohort within a sample group▪ Get permission: this isn’t just a nice chat!

Page � of �2 7

Interviews:

‣ a conversation with a purpose (i.e., to develop software)

‣ provide in depth understanding of an individuals perspective

‣ good to do more than one to obtain multiple view points

‣ sample (chose participants) carefully

‣ rich source of information, time consuming Focus group: intervie

w

with multip

le participants.

Interviews:

‣ set up

‣ interview plan

‣ interview questions

‣ guidelines for good interviews

‣ analysis

Recruit participants Get permission

Arrange location (preferably at users site)

Schedule and confirm arrangements

Prepare data recording equipment (e.g., notes or audio/video recordings)

Get permissionGet permission

General Population

Target Population

Accessible Population

Sample Population

Introduction: Who yo

u are, the

purpose of the intervie

w, ethical

issues, perm

ission to m

ake recordings

Warm-up: Easy non-

threatening questions

Body: Logical sequence of

questions, with harder

questions the end

Wrap up: Thank the intervie

wee,

switching off of re

cording

equipment to show intervie

w has

ended

Interviews:

‣ set up

‣ interview plan

‣ interview questions

‣ guidelines for good interviews

‣ analysis


▹ You will be recording their thoughts and using it as data▪ Arrange locations at the interviewee’s natural environment

▹ So that they feel comfortable and give accurate answers▪ Prepare the equipment

▹ Always have a backup: after interview, realise that you forgot to press rec.! - CREATE INTERVIEW PLANS

▪ Introduction▹ Let interviewee know: purpose, permission to use, who you are, ethics

▹ Warm-up: Small talk to relax interviewee▹ Body: Logical sequence of questions; harder questions at the end▹ Wrap-up: Say thanks and switch off recording equipment (to show end)

- MAKE GOOD INTERVIEW QUESTIONS▪ Open Questions = GOOD

▹ No predetermined answers▹ Covers range of issues of the goal▹ Unexpected answers will come up

◦ Follow a new sequence of questioning if so to extract more info on that…

▪ Don’t write:▹ Long Questions▹ Compound Questions (too many ideas in one)▹ Questions with technical jargon▹ Judgemental Questions (“Don’t you think this feature would look better?”)▹ Questions that don’t predict future beh. (“Is it usable compared to now?”)

- HAVE SOME INTERVIEW GUIDELINES ▪ Be professional▪ Dress similarly to interviewee▪ Make out that you’re enjoying the interview▪ Give silences for answers to follow▪ Give probes to get more out of interviewee (“Anything else?”) ▪ Give prompts to remind users of things (screenshots, pics)

- THEN ANALYSE YOUR INTERVIEW ▪ TRANSCRIBE the audio▪ Extract any qualitative data

▹ Create requirements based on that qualitative data▪ Use affinity diagrams:

Page � of �3 7

Analysis:

‣ sort and categorise comments/ideas (look for themes)

‣ affinity diagram

‣ count frequency of different categories of comments

‣ look for common concerns between participants

Random Ideas Affinity Diagram

Theme A Theme B Theme C

http://www.baran-systems.com/Products/Affinity%20Diagram%20for%20Excel/index_concept.htm


▹ Sort comments into themes, count the frequency of these concerns▸ Surveys/Questionnaires

- Efficient way to obtain lots of statistics (background info.) from lots of people- Good when you know what you’re looking for: use closed questioning - BUT: Usually low response rates (20-30%)- SET UP:

▪ Prepare and test questions to see if okay▪ Decide Delivery depending on user groups:

▹ Phone—More social with quality control (can’t give bad answers). Good for the elderly.

▹ Web—Low cost and can reach mass audiences. Good for techies.▹ Paper—Same as paper except less social and bad answers…

▪ Find participants; by submission they consent- CLOSED QUESTIONING

▪ Answer is limited by interviewer▪ Focus is provided by questions

- TYPES OF QUESTIONS: ▪ Checkbox and Ranges

▹ Answers should be very explicit (Very often vs. More than 2 times a day)▹ Good for demographics; quick and easy

▪ Semantic Differential ▹ Simple question format with antonyms▹ Get’s people’s feelings about system

▪ Likert Scale ▹ Rating of agreement with statements▹ Get’s people’s feelings about system▹ Don’t give questions with two statements

◦ (I use net to do X AND Y) ▹ Remove poor data by:

◦ Giving 2 of the same questions, see if they answer the same for both- ANALYSIS:

▪ TRANSCRIBE the audio▪ Extract quantitative data

▹ Count the frequency of answers▹ Plot them using bar charts or pie charts to get point out easily▹ Use a table to explain if number of participants is low

Page � of �4 7

How often do you check email?❏ Very Often❏ Often❏ Sometimes❏ Rarely❏ Never

Checkbox and Ranges

• good for demographic/factual information

• make it quick and easy to fill out

Trap: ambiguous options (e.g.,

often m

ay mean diffe

rent things to

different p

eople)

How often do you check email (please check option that best describes you)?❏ More than 2 times a day❏ once a day❏ 2-3 times a week❏ 2-3 times a month❏ Never

Range

Please circle the number that describes your experience using YourEmail in relation to the following adjectives:

hard 1 2 3 4 5 easyconfusing 1 2 3 4 5 understandableexiting 1 2 3 4 5 boringstressful 1 2 3 4 5 relaxing

Semantic Differential

• simple question format

• good for getting peoples feelings about a system

use antonyms

Please circle the number that best represents how you feel about the following statements.I enjoy working with computers.Strongly 1 2 3 4 5 StronglyDisagree Agree

Learning to use new software is stressful.Strongly 1 2 3 4 5 StronglyDisagree Agree

Likert Scale

• rating of strength of agreement with statements

• good for measuring opinions and feelings

Trap: avoid ‘d

ouble barreled’

statements (ie statements th

at

have two or m

ore ideas ‘I use th

e

internet to contact fr

iends and

research my in

terests.’)

Survey/Questionnaires:

‣ set up and planing

‣ question types and guidelines

‣ analysis


Checkbox and Ranges




often m

ay mean diffe

rent things to

different p

eople)


Range






use antonyms



Likert Scale



Trap: avoid ‘d

ouble barreled’


at

have two or m


e


iends and

research my in

terests.’)




‣ analysis


Checkbox and Ranges




often m

ay mean diffe

rent things to

different p

eople)


Range






use antonyms



Likert Scale



Trap: avoid ‘d

ouble barreled’


at

have two or m


e


iends and

research my in

terests.’)




‣ analysis


▸▸ Observation of users in the field

- Find out what people actually do rather than what they say they do▪ Hawthorne effect: changes in behaviour due to being aware of observed▪ Habituation: after certain time, forget you’re there!

- BUT: Events of interest infrequent, rapidly changing, time consuming, lots of data- GENERAL GUIDELINES:

▪ Be clear as to what you want to study▪ Decide how to record information▪ Be prepared to change your focus if you learn more▪ Identify boundaries of observation▪ Confer notes with participants

- ANALYSIS: ▪ Use diagrams and photos to display the environment▪ Similar to interviews: Look similar themes in the environment in diff’nt people.

▸ Card sort - Good to find out how people think information should be organised- Prepare:

▪ Set of cards with objects of interest▪ Shuffle cards▪ Ask users to sort into piles and then group▪ Make a cart sort cluster tree to display all information

▸ Ultimately which one to choose?- What data do you want?- You may need a combination of activities to help you find the qualitative

[interviews] or quantitative [card sort/survey] data…- Remember:

▪ Your budget & resources—time, money, skills, equipment▪ The stage of the project—early requirement stage vs. mature requirements

Page � of �5 7


3.03 Ethical Research

Informed Consent ▸ Applies primarily with interviews and observations only

- Participation is voluntary- Participants must understand what’s involved, the risks and benefits

▪ Outline this in an information sheet: ▹ Purpose of study? what they’re required to do? how long? voice

recordings? procedures to protect privacy? where to go to complain?▹ For the participant to keep

▪ Legalise this in a consent sheet: ▹ “I, ___, understand that…” ▹ Outline who will see their data:

◦ Me: Alex Cummaudo◦ Teammates: Brenton Thomas, Ryan Cooke, Amit Pandey◦ Tutor: Mr. Leonard Hoon◦ Convenor: Dr. Karola Von Baggo

▹ Outline that their data is de-identified◦ Consent forms will be separately attached to all anonymised

participant data▹ Permission to record…▹ For the investigator to keep

Freedom to Withdraw ▸ Participants need to understand that they can leave at any time.

- No prejudice: if you’ve promised them Candy you must give it!▸ If you can see that participants look distressed, offer to bring the session to an end

Assess Risks and how you mitigate them

▸ Physical harm: any pain or injury▸ Psychological harm: angered, frustrated, worthlessness, fear▸ Social harm: harm to social networks/employment▸ Economic & legal harm: costs to participants, participants breaking laws/

Page � of �6 7


▸ Remember: it’s just a prototype; it’s never their fault but your fault!

Page � of �7 7

Lecture #4 Alex Cummaudo: Sims 1744070 Usability Requirements 27/04/2016

Defining Usability Requirements ▸ What Are They?

- Specifies all aspects of products- Motivates designs (designs that meet requirements)- Bad requirements → designs that don’t meet users needs- Developmental resources get allocated to items via requirements

▸ Tradeoffs and Constraints from Requirements- Resources available- Technology available- Stakeholder politics- You negotiate with stakeholders to get what you want to provide and what they

desire

Three Types of Requirements ▸ Business Requirements▸ Technical Requirements (e.g. OS, RAM etc.)▸ User Requirements

- Functional▪ The system shall do this functional requirement

- Nonfunctional▪ The system shall be this nonfunctional requirement▪ Usually qualitative or quantitative

▸ Need context of use to find requirements- Users, Tasks, Environments- Be specific; use user profiles, affinity diagrams, task descriptions

Page � of �1 3


Usability Metrics

Defining Usability Metrics

▸ Use Usability Metrics to Determine User Requirements- User Requirements must be quantifiable- Effectiveness (if >0, effectiveness = 0%)

▪ always within the boundaries of one task▹ # errors: any deviations from the perfect task path▹ # assists: any help from developers▹ completeness: anything within the task that does not get done▹ accuracy: how good the results from the tasks are

▸ Efficiency- always within the boundaries of one task

▪ average assisted time taken and▪ average unassisted time taken

▸ Satisfaction- within the boundaries of all tasks- use S.U.S (System Usability Scores) scores

▪ they reflect the satisfaction of the entire system▪ industry standard; better SUS = better product▪ aim for S.U.S scores of ~85%; they are the most commercially successful▪ looks for user’s first reactions to statements▪ negatively reversed scores help spot out bad feedback

▸ Quesenbery’s Five E’s expand Usability Metrics- Effectiveness- Efficiency- Engaging (satisfaction)- Error Tolerant- Easy To Learn

Measuring success in Usability Metrics

▸ Set achievable targets▸ Measure product performance against those targets▸ Identify the benchmarks▸ Measured in three cases: business, competitor and expert

- Business Cases: Has to be done within 5 secs, else costs us money!Page � of �2 3


- Competitor Cases: They can do it in 3 mins, so we have to be just as good as that!

- Expert Performance: If the design team can do it in 30s, then users probably can do it 50s.▪ Note: Expert Performance exists where no competitors don’t exist/allow you

to see their benchmarks.

Measuring importance in Usability Metrics

▸ Not all metrics have equal importance (see below:)

(Below) Prioritising in terms of the 5 E’s

� �

Usability Metric Doctors Train Tracker Games

EffectivenessCare that they get the right meds

Care that it gives us right information

Somewhat care how complete the game is

EfficiencySomewhat care how quick it takes them

Care how quick it tells us information

Don’t care how quickly it takes them

Satisfaction

Don’t care how satisfied it makes them

Don’t care how satisfied it makes them

Care how satisfied it makes them

2.2 What Type of Data Do I Want to Collect?

As part of your evaluation strategy, you must identify the type of evaluation data thatwill help you explore the usability requirements (see Table 21.2). Two types of dataare obtained during evaluations: quantitative and qualitative.

• Quantitative data are any type of numeric data derived from takingmeasurements. For instance, if during evaluation, you are recording

CHAPTER 21 | Deciding on what you need to evaluate: the strategy

444Part 4

EffectiveEfficientEasy

toLearn

ErrorTolerant

Effective – 20%

Efficient – 5%

Engaging – 40%

Error Tolerant – 5%

Easy to Learn – 30%

Engaging

Figure 21.2 The Five Es for the Online Museum Exhibition: Here we see the high priority placed on“Engaging.” (From Quesenbery, 2003.)

Effective

EfficientEasy

toLearn

ErrorTolerant

Effective – 20%

Efficient – 30%

Engaging – 30%



Engaging

Figure 21.3 The Five Es for the General Museum site. In contrast to the exhibition, the museumsite has more balanced usability requirements. (From Quesenbery, 2003.)

2.2 What Type of Data Do I Want to Collect?

As part of your evaluation strategy, you must identify the type of evaluation data thatwill help you explore the usability requirements (see Table 21.2). Two types of dataare obtained during evaluations: quantitative and qualitative.

• Quantitative data are any type of numeric data derived from takingmeasurements. For instance, if during evaluation, you are recording

CHAPTER 21 | Deciding on what you need to evaluate: the strategy

444Part 4

EffectiveEfficientEasy

toLearn

ErrorTolerant

Effective – 20%

Efficient – 5%

Engaging – 40%



Engaging

Figure 21.2 The Five Es for the Online Museum Exhibition: Here we see the high priority placed on“Engaging.” (From Quesenbery, 2003.)

Effective

EfficientEasy

toLearn

ErrorTolerant

Effective – 20%

Efficient – 30%

Engaging – 30%



Engaging

Figure 21.3 The Five Es for the General Museum site. In contrast to the exhibition, the museumsite has more balanced usability requirements. (From Quesenbery, 2003.)

Page � of �3 3

Lecture #5 Alex Cummaudo: Sims 1744070 Interface Design: Prototyping 27/04/2016

UCD and Prototyping ▸ Prototyping comes underneath ‘iterate different design solutions’

- We’re developing different solutions via prototyping and getting them ‘approved’ by users (i.e. which one they like best)

Why a prototype? ▸ Allows interaction with proposed product▸ Communicates ideas▸ Users can participate in design (esp. low fidelity prototypes!)▸ Can test features in realistic UI▸ Clarifies requirements▸ Helps to negotiate requirements▸ Can indicate potential functionality

- Does the prototype function like the final product should?▪ compare low fidelity prototypes (no functionality) with high fidelity prototypes

(functionality)

Fidelity

Low-Fidelity

▸ Very basic technologies - paper▸ Determines user’s interaction▸ Does not determine functionality

Advantages Disadvantages

• Simple• Cheap• Quick• Flexible• Evaluates multiple designs• Interaction between developer and

stakeholders• Provides proof-of-concept

• Limited error checking• Poor specification to code to• Limited usefulness when requirements are

established• Limited usefulness when usability,

navigation and flow is tested

Page � of �1 2

Lecture #5 Alex Cummaudo: Sims 1744070 Interface Design: Prototyping 27/04/2016

Mid-Fidelity

▸ Simulation of actual product - presentation software▸ No actual code behind the prototype▸ Advantages: Can still make changes relatively quickly▸ Disadvantages: Requires more time to build

High-Fidelity

▸ High level technical applications (could be actual product) - HTML

Development philosophy ▸ You can evolve your prototype into a final product, or just throw it away once you’re

done▸ Evolutionary philosophy

- prototyped code makes it into the final product- requires rigorous code testing

▸ Throwaway philosophy- prototypes used for evaluation- prototype is not used int the final product → thrown away!

Advantages Disadvantages

• Looks similar to final product• Real-time evaluations• Useful for demonstrations

• Costlier (time and money)• Focus is on superficial coding details• Users less willing to change a more

finished looking prototype• Bugs can interfere• Developers resistant to change

Page � of �2 2

Lecture #6 Alex Cummaudo: Sims 1744070 Conceptual Design & Design Guidance 27/04/2016

Conceptual Design

Why?

▸ Use Context of Use to understand the situation of use▸ This allows us to develop a System Description and Requirements▸ From the descriptions and requirements we deduce suitable User Interface (known as

Conceptual Design)▸ Context of Use → System Description & Requirements → Conceptual Design →

Physical Design

What?

▸ conceptual design specifies what the interface should do and how it is organised and structured.

▸ Describes:- the underlying organisation of the UI- functions each screen supports- the system in terms of functions, structure, and content

▸ Clear conceptual designs = easy learning curve▸ Use essential use cases to allocate tasks between user and system▸ Use content diagrams to show sets of functions (containers) and relations between

them (links): insert diagram▸ Content diagrams aren't decision trees; they just map out the basic relations of

functions in the design

Page � of �1 4

• underlying organisation of the user interface

• specifies functions each screen should support

• abstract description of system

• function

• structure

• content

• clear conceptual design

• easy to learn

• focus on tasks

• allocation of tasks to user and system

• essential use cases important

Conceptual design

Main Container Container 1 Container 2 Container 3

Container 4 Container 5

Content diagram

‣shows sets of functions (containers) and relationships between them (links)

Create

Account

Login/

Register

Upload

Images

Order

PrintsPaymentLogout

Manage

Album


Content Diagram

9

10

11


Design Guidance ▸ Conceptual design doesn’t show us how to arrange elements, what colours to use

etc.▸ Use Design Guidance to solve this▸ They come in four flavours:

- Design Principles: ▪ Very broadest level. ▪ But, somewhat up to interpretation. ▪ Describe the following general usable features: [FACTS VS (CRAP)]

▹ Feedback◦ Inform users what’s going on

▫ "Copying complete" → messages ▫ ‘Beeps’ if you've done something wrong ▫ Tactile feedback

▹ Affordance◦ Design the UI so users know that it feels right to do tasks ◦ E.g. shaping and weighting an object that feels wrong if you hold it the

wrong way. (e.g. remote control) → good affordance ◦ E.g pull bar on a push door ≠ good affordance

▹ Consistency◦ Keeping uniformity within the UI◦ This makes it easy to learn◦ Use a base wireframe structure, and then work from there in all UI

pages▹ Tolerance

◦ Design the UI to prevent errors ◦ E.g.: why is the button greyed out for?

▹ Simplicity◦ Keeping the UI as simple as possible◦ Use actions, icons, words that are natural to users◦ E.g. Whitespace, grid patterns, limit choices

▹ Visibility◦ Design the UI so that functionality is visible◦ Let users see what the action is◦ If you can’t see it, it’s not there

▹ StructurePage � of �2 4


◦ Organising the UI in a meaningful way◦ Related features closer together◦ Layout the UI in a way the compliments the user’s task workflow◦ E.g. Toolbars at the top; 3 columns: organiser

at the side, overview of data, detailed data (think of Mail); click ‘get mail’ → select mailbox → select email → read email

▪ Graphic Design Principles [(FACTS VS) CRAP] ▹ Contrast:

◦ Headings, colours (use opposite colours on the colour wheel) for different buttons

◦ E.g. Go button Green, Back Button Red ▹ Repetition

◦ Makes things looks more professional and ordered (else user confused as to where to start?)

◦ Avoids difference→Why is it different?→Meaningless questions▹ Alignment

◦ Centre alignment:▫ non-professional▫ hard to read

◦ Retina workflow = minimise eye-jumping◦ Left Justify

▫ Eyes know where they have to jump to at an e.o.l▫ Easy to scan▫ Keeping things aligned makes things less complex!▫ Reduces cognitive load/processing to make the task easier

▹ Proximity◦ Distance b/n things to create relationships between 2 things◦ E.g. horizontal separators in menu bars

- Design Guidelines: ▪ Specific advice about what to do

- Design Standards: ▪ Very specific in regards to good design that’s recognised internationally▪ ISO have made standards which relate to a wide range of products

▹ ISO Accessibility Standards… - Style Guide

Page � of �3 4


▪ Specific to certain platforms (e.g. Office, OS X, Windows)

Page � of �4 4

Lecture #7 Alex Cummaudo: Sims 1744070 Interaction Styles And Devices 27/04/2016

Interaction Devices ▸ Input devices

- Transform’s users information → electronic signals that machines interpret- Media for entry between users and machines:

▪ Discrete entry▹ alphanumeric keyboards

▪ Continuous entry ▹ Directness

◦ Direct (e.g. touch screen) ▫ Go directly to where you want to go

◦ Indirect (e.g. mouses, trackpads) ▫ Forward and backward maps to upward and downward▫ This mental calculation causes steep learning curve

▹ Gain◦ Absolute gain (e.g. touch screen)

▫ Moving your finger on a screen is absolute to where the pointer is◦ Relative gain (e.g. mouses, trackpads)

▫ Moving mouse 1cm causes 10cm movement in the mouse cursor▸ Output devices

- Transform’s machine information → sensory information humans interpret- Media for output:

▪ Visually (e.g. LCD screens, printers)▪ Auditory (e.g. alarms, speakers)▪ Tactility (e.g. vibration)▪ Olfactory (e.g. smells)

▸ Choosing the right input device:- Accessibility: users have trouble mapping the mouse (mental disability) or using it

(physical disability) use a touchscreen as it’s much more direct with less learning!▸ Choosing the right output devices

- Depends on the amount of attention the users have

Think of humans having 4 ‘buckets’ of attention span: [visual] [audio] [tactile] [olfactory] A system being used in a busy kitchen. There’s lots of smells, visual screens and alarms: [visual] [audio] [tactile] [olfactory] ^ ^ An electric shock [tactility] may work best in this case (although it’s not ethical!)

Page � of �1 2

Lecture #7 Alex Cummaudo: Sims 1744070 Interaction Styles And Devices 27/04/2016

Interaction Styles ▸ How do we issue commands to computers?

- Command Line Interface▪ Direct input by a string of characters, called a ‘command’▪ Discrete input device (keyboard)▪ Pros:

▹ Direct access to functionality (rm /*.* = delete everything on the system) ▹ Very quick and efficient▹ Good for batch processing

▪ Cons:▹ Need to know the syntax very well▹ Steep learning curve

- Menu Interface▪ User chooses commands from a set of options (a restaurant menu metaphor)▪ Discrete (Keyboard) or continuous (Mouse) input ▪ Pros:

▹ All options available (don’t need to remember the commands)

▹ Complex actions broken down▹ Very easy

▪ Cons:▹ Very slow for experts▹ Visual space needed

- Form-fill Interface▪ Gathers information from users in a form (paper form metaphor)▪ Discrete (Keyboard) or continuous (Mouse) input

- Direct Manipulation▪ Interaction with objects as if they are real (e.g. drag and drop)▪ Based on natural understanding of the world▪ Continuous (Mouse) input ▪ Pros:

▹ Easy to remember (based on repetition)▹ Users see if goals are achieved▹ Users feel in control

▪ Cons:▹ Lots of screen space used, slow, other inputs are better (e.g. cmd for find)

Page � of �2 2

Psychology Of Usability Alex Cummaudo 1744070 Usability 27/04/2016

Mental Models ▸ Represent a person’s understanding of a system: functional and structural info▸ Allows people to reason with a system▸ Users develop mental models by interacting with systems

▪ E.g. in Stone Et. Al. is the Mental Model of your home - E.g. restaurant; sit down, wait for waiter, order, eat food, pay, go home.

▸ New systems requires users to create a new mental model of your system- E.g. Paul going from Windows to Mac - “It’s not ‘logical’” - E.g. Nonna using an iPad to read a book and flicks the page like so.

▸ Implementation Models- Ugly stuff that drives the functionality ≠ Mental model- Mental models are a lot simpler than the implementation model

Sending a fax:- Implementation Model is the 010101 that gets sent down the phone line- Mental Model is the piece of paper that gets sent down a long tube

▸ We want UIs that reflect Mental Models (how users think it works) rather than implementation model (how it actually works)

Metaphors ▸ Build bridges between the mental models and implementation model via metaphors▸ E.g. desktop metaphor; files sit on the desktop, calculators sitting on the desktop etc.

The desktop has little floppy disk icons You drag the floppy disk into the trash to delete it People thought that by throwing the disk into the trash, you delete everything on that disk

= Bad Metaphor

Page � of �1 1

Mind Map Usability

C4 data analysis

writing up data summary tables and calculations

interpretation of problem severity

impact

persistence

frequency

central tendencydoes it all lead to one place?

content analysis

types of data

qualitative

qualitative

objective

summative

C3 evaluation procedure

recruiting representative users

ethics

step by step

team roles

biases/generalisation issues

ecological validity is the lab the same as the real?

biasing instructions

accessible population sizes and type

A1 User Centred Design

process

plan the process

understand/research context of use

Organisational and user requirements (usability requirements

prototype alternative designs

evaluation

principles

early focus on users

measurement

iteration

appropriate allocation of tasks

other

software quality

cost benefit

persuasion

usability maturity

C2 evaluation tools

video/screen capture

questionnaire typesdemographic

screening

interpretationsatisfactiontasks

think aloud

observation sheets

one-way mirror

A2 document context of use

users

type

primary

secondary

tertiary

characteristics demographic data

case study domain

documentation profiles

personas

goals and tasks task characteristics

documentation

scenarios

essential use cases

storyboards

hierarchical task descriptionenvironment

story boards

photos

lists/matrix

scenarios

typessocial

physical

technical

A3 research context of use methods

interviews

questionnaires/surveys

observation/field studies

card sorting

competitive analysis

C1 usability evaluation methods

empirical/observation

informal/quick and dirty low budget

formal/lab evaluation

remote testing can't get an accessible population

field testing go into user's environmentinspection/prediction

heuristic evaluation

A4 usability requirements

targetscompetitive analysis/evaluation

business case

metrics

effectiveness

efficiency

satisfaction

B3 psychological principles

cognitionmemory

attention

perception gestalt lawsCRAP

manipulating distance between things etc.mental models

metaphors

B1 design principles

CRAP (graphical design principles)

Contrast

Repetition

Alignment

Proximity

FACTS VS Feedback

Affordance

Consistency

Tolerance

Simplicity

Visibility

StructureB2 interaction controls

interaction styles

radio buttons

sliders

text boxes

date-pickers

checkboxes

sliders

drop down list

interaction controls

command language

menu

form fill

direct manipulation

planning

sampling

recruitment

population

proceduresteps

location

ethics

data recording

materials

questionnaire

interview questions

location

analysis

summary data

content analysis

affinity diagrams

lifecycle

formative

summative

severity ratings

expert case

implementation models

Alex Cummaudo: Sims 1744070 Usability 27/04/2016 · Evaluation Methods Alex Cummaudo: Sims 1744070...

Documents

Transcript of Alex Cummaudo: Sims 1744070 Usability 27/04/2016 · Evaluation Methods Alex Cummaudo: Sims 1744070...