1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

24
1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2
  • date post

    21-Dec-2015
  • Category

    Documents

  • view

    218
  • download

    2

Transcript of 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

Page 1: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

1

CS 430 / INFO 430 Information Retrieval

Lecture 24

Usability 2

Page 2: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

2

Course Administration

Page 3: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

3

Course Administration

Assignment 3

Grades were returned yesterday

Page 4: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

4

The Design/Evaluate Process

Requirements (needs of users

and other stakeholders)

Design (creative application of

design principles)

Implementation (may be prototype)

Evaluation

release

start

Page 5: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

5

Usability Factors in Searching: User Interface

Design of an interface for a simple fielded search.

Interface: Fill in boxes, text string, ... ?Presentation of results ... ?Manipulation of results ... ?

Functions: Specify field(s), content, operators, ... ?Retain results for manipulation ... ?Query options ... ?

Data: Metadata formats ... ?Data structures and file structures ... ?

Systems: Performance ... ?

How do we evaluate various designs?

Page 6: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

6

Usability Factors in Searching: Ordering of results

The order in which the hits are presented to the user:

• Ranked by similarity of match (e.g., term weighting)

• Sorted by a specified field (e.g., date)

• Ranked by importance of document as calculated by some algorithm (e.g., Google PageRank)

• Duplicates shown separately or merged into a single record

• Filters and other user options

What impact do these choices have on the usability?

How do we evaluate various designs?

Page 7: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

7

Evaluation

What is usability?

Usability comprises the following aspects:

• Effectiveness – the accuracy and completeness with which users achieve certain goals Measures: quality of solution, error rates

• Efficiency – the relation between the effectiveness and the resources expended in achieving themMeasures: task completion time, learning time, clicks number

• Satisfaction – the users’ comfort with and positive attitudes towards the use of the systemMeasures: attitude rating scales

From ISO 9241-11

Page 8: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

8

Evaluation

• The process of determining the worth of, or assigning a value to, the usability on the basis of careful examination and judgment.

• Making sure that a system is usable before launching it.

• Iterative improvements after launch.

• Categories of evaluation methods:

– Analytical evaluation: without users

– Empirical evaluation: with users

– Measurements of operational systems

Page 9: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

9

Evaluation without Users

Assessing systems using established theories and methods

Evaluation techniques

• Heuristic Evaluation (Nielsen, 1994)

– Evaluate the design using “rules of the thumb”

• Cognitive Walkthrough (Wharton et al, 1994)

– A formalized way of imagining people’s thoughts and actions when they use the interface for the first time

• Claims Analysis – based on scenario-based analysis– Generating positive and negative claims about the effects of

features on the user

Page 10: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

10

Evaluation with Users

Testing the system, not the users!

Stages of evaluation with users:

Preparation

Sessions conduct

Analysis of results

User testing is time-consuming and expensive.

Page 11: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

11

Evaluation with UsersPreparation

• Determine goals of the usability testing

“The user can find the required information in no more than 2 minutes”

• Write the user tasks

“Answer the question: how hot is the sun?”

• Recruit participants

Use the descriptions of users from the requirements phase to detect potential users

Page 12: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

12

Usability Laboratory

Concept: monitor users while they use system

Evaluators User

one-way mirror

Page 13: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

13

Evaluation with UsersSessions Conduct

• Conduct the session– Usability Lab– Simulated working

environment

• Observe the user– Human observer(s)– Video camera– Audio recording

• Inquire satisfaction data

Page 14: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

14

Evaluation with UsersResults Analysis

• If possible, use statistical summaries

• Pay close attention to areas where users

– were frustrated

– took a long time

– couldn't complete tasks

• Respect the data and users' responses, don't make excuses for designs that failed

• Note designs that worked and make sure they're incorporated in the final product

Page 15: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

15

Measurements on operational systems

Analysis of system logs

• Which user interface options were used?

• When was was the help system used?

• What errors occurred and how often?

• Which hyperlinks were followed (click through data)?

Human feedback

• Complaints and praise

• Bug reports

• Requests made to customer service

Page 16: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

16

The Search Explorer Application: Reconstruct a User Sessions

Page 17: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

17

Refining the design based on evaluation

Designers and evaluators need to work as a team

Designers are poor evaluators of their own work, but know the requirements, constraints, and context of the design:

• Some user problems can be addressed with small changes

• Some user problems require major changes

• Some user requests (e.g., lots of options) are incompatible with other requests (e.g., simplicity)

Do not allow evaluators to become designers and vice versa

Page 18: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

18

Experiment on the Google Interface

Methodology

• 10 information seeking tasks in 2 categories

• Users randomized across tasks

• Click through data to see what the user did

• Eye tracking data to see what the user viewed

• Google results presented with ranks changed or reversed

An example of interdisciplinary information science research by Cornell's Human Computer Interaction Group and Computer Science Department

Page 19: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

19

Evaluation Example: Eye Tracking

Page 20: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

20

Evaluation Example: Eye Tracking

Page 21: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

21

Google EvaluationClick-Through Data

1 2 3 4 5 6 7 8 9 10

rank

0

50

100

150

200

Frequency

rank

Number of users who clicked on link

Rank of hit

Page 22: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

22

Google Evaluation Eye Tracking Data

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10

Number of users who viewed short record before first click

Rank of hit

Page 23: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

23

Google Evaluation: Eye Tracking Data

Title: 17.4%Snippet: 42.1%Category: 1.9%URL: 30.4%Other: 8.2% (includes, cached, similar pages, description)

Part of short record viewed before first click (% of users)

Page 24: 1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.

24

Google Experiment: Click Through Data with Ranks Reversed

1 2 3 4 5 6 8 9 10

ChosenRank

0

20

40

60

80

Percent

Cond: r

ChosenRank

Rank of hit

Percentage of users who clicked on link