ACRL 2011 Data-Driven Library Web Design

28
Data-Driven Library Web Design: Making Usability Testing Work with Collaborative Partnerships Allison Cowgill, Head of Reference Amanda Dinscore, Public Services Librarian Patrick Newell, AUL for Information Technology and Electronic Resources Henry Madden Library California State University, Fresno All documents available at: http://www.slideshare.net/adinscore

description

Presentation from a workshop given at ACRL 2011 conference, Data-Driven Library Web Design: Making Usability Testing Work with Collaborative Partnerships

Transcript of ACRL 2011 Data-Driven Library Web Design

Page 1: ACRL 2011 Data-Driven Library Web Design

Data-Driven Library Web Design: Making Usability Testing Work with

Collaborative Partnerships

Allison Cowgill, Head of ReferenceAmanda Dinscore, Public Services Librarian

Patrick Newell, AUL for Information Technology and Electronic Resources

Henry Madden Library California State University, Fresno

All documents available at: http://www.slideshare.net/adinscore

Page 2: ACRL 2011 Data-Driven Library Web Design

Background

• The Library Study at Fresno State—Ethnographic study conducted by two anthropology professors• Study recommended that the Library’s web site should be should be redesigned

“Draw How You Feel When You Write a Paper.”

Page 3: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 1: Create a Purpose Statement and Objectives

Purpose Statement: Should encapsulate the goals the team hopes to accomplish.

Example: “The purpose of this study is to determine if users can easily accomplish tasks required for research using the library’s web site.”

---------------------------------------------------------------------------Objectives: Use to develop the user tasks. Should

reflect actual user needs.

Example: “Determine the number of study participants who are able to search for and locate a book using the library’s web site.”

Activity: Create a draft

purpose statement and at

least 3 objectives.

Page 4: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 2: Form the Team

Identify librarians and library staff who are sincerely interested in participating and support change.

Find collaborators outside of the library from academic departments such as Anthropology, Business, Computer Science, or Education. Everyone should be fully aware of the time and effort required.

Activity: Brainstorm

potential collaborators from

both within and outside your

library.

Page 5: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 3: Identify User Tasks & Develop Questions

Consult with others (public service librarians, web team,…) • What is easy/difficult for users to do on our web site?• What do you spend time helping users do on our web site?

Create a list of tasks users are expected to perform and relate the tasks to the study objectives

Example tasks from our study: • Find a book title in our library• Find a book title through our patron-initiated borrowing system• Find a newspaper article• Find an article from a scholarly journal

Keep in mind the type of data you will collect & use

Task: An activity that fulfills an

information need

Page 6: ACRL 2011 Data-Driven Library Web Design

“All qualitative data can be coded quantitatively.”

http://www.socialresearchmethods.net/kb/qualdeb.php

Drafting the Research Plan

Step 3: Identify User Tasks & Develop Questions

Types of Data:Independent Variables: Variables you manipulate. Choose these based on your research questions.Dependent Variables (a.k.a. Outcome/Response Variables): Something you measure as the result of (based on the response to) the independent variables.

Quantitative Data: can be counted or expressed numerically Qualitative Data: nonnumeric information such as conversation, text, audio, or video.

Page 7: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 3: Identify User Tasks & Develop Questions

Data Type Common Metrics Statistical Procedures

Nominal (categories) Task success (binary), errors (binary)

Frequencies, crosstabs, Chi-square

Ordinal (ranks) Severity ratings, rankings (designs)

Frequencies, crosstabs, chi-square, Wilcoxon rank sums, Spearman rank correlation

Interval Likert scale data, SUS scores All descriptive statistics, t-tests, ANOVAs, correlation, regression analysis

Ratio Completion time, time (visual attention), average task success (aggregated)

All descriptive statistics, t-tests, ANOVAs, correlation, regression analysis

Page 8: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 3: Identify User Tasks & Develop Questions

Examples:

Using the library web site, find one journal article on swine flu.

Show me where on the web site you can find help using the library.

The library has a page with resources organized by subject. Show me how to find the page with history resources.

Activity: Brainstorm at least 3 tasks based on

the objectives you created.

What data will you use to measure the

tasks?

Page 9: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 4: Determine the Study Population

Who?

How many?

Page 10: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 4: Determine the Study Population

The Size of Your Population or Sub-GroupWhy Sample?• To say something about a population• A statistically valid sample size allows you to generalize to a population from a sample

Confidence Level • Tells you how sure you can be• Represents how often the true percentage of the population who would pick an answer lies within the confidence interval

Confidence Interval • a.k.a. “margin of error”• A range that estimates the true population for a statistic

Page 11: ACRL 2011 Data-Driven Library Web Design

Technique Advantages Disadvantages

Random sampling •Theoretically most accurate.•Influenced only by chance.

Sometimes a list of the entire population is unavailable or practical considerations or prevent random sampling.

Systematic sampling •Similar to random sampling.•Often easier than random sampling.

The system can sometimes be biased.

Quota sampling •Can be used when random sampling is impossible.•Quick to do.

There may still be biases not controlled by the quota system.

Stratified sampling •Ensures large enough sample to subdivide on important variables.•Needed when population is too large to list.•Can be combined with other techniques.

Can be biased if strata are given false weights, unless weighting procedure is used for overall analysis.

Drafting the Research Plan

Step 4: Determine the Study Population

Page 12: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 4: Determine the Study Population

Your Recruitment Strategy

Consider: •Advertising Needs •Recruitment Location(s)

•Incentives

Page 13: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 5: Room/Technology/Data Capture Considerations

Intake/subject data gathering location

Testing location

Equipment/staff to record the data• Hardware• Software• Who configures/operates/troubleshoots?

Privacy/data security considerations• Privacy and personal consent• Data back up and security

Page 14: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 5: Room/Technology/Data Capture Considerations

Back up procedures

Equipment/staff to code the data

Equipment/staff to analyze the data

Activity: Make a list of the

resources available at your

own library. What might you

need?

Page 15: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 6: Develop Scripts/Instructions & Train Moderators

Creating a script and instructions for moderators: • Helps them to clearly explain study procedures to subjects• Ensures uniformity throughout the process

Take advantage of collaborating teaching faculty’s expertise by enlisting their help to train student moderators.

Links: Moderator Instructions & Introduction Scripthttp://www.flickr.com/photos/lwr/4124641930/

sizes/l/in/faves-61999692@N00/

Page 16: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 7: Pre-test & Refine

Pre-test using moderators as subjects and have moderators run each other through the protocols/tasks.

Identify any unclear or skewed questions and revise accordingly

Critical to remember that we are focusing on web site usability, not student ability or experience.

Page 17: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 8:Test

Schedule rooms

Schedule helpers (moderators, recruiters, supervisors, etc.)

Verify all parts of the web site are working (surprise!)

Assure pre-tests and consent forms are present

Assure incentives are present

Back up data nightly (multiple times, on different media, if possible)

Page 18: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 9: Analyze the Data

Prepare forms/spreadsheets for data processing

Code the qualitative data from video/audio• Develop codes beforehand• Review data/develop codes/apply codes

Calculate useful statistics from the data• Time on task (mean)• Completion rate (percentages)

Activity: Practice Coding

Page 19: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Meet with library colleagues, showing them video excerpts and sharing preliminary findings.

Develop clearly understandable graphs and other visuals that show how students navigate the web site and the difficulties they encounter.

Analyze and communicate qualitative data to stakeholders and address issues with an eye towards internal sensitivities.

Page 20: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

• Add graphs here

Page 21: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Page 22: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Page 23: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Page 24: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Page 25: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 10: Communicate the Data

Page 26: ACRL 2011 Data-Driven Library Web Design

Drafting the Research Plan

Step 11: Make Revisions to the Site

Meet with web team to discuss findings

Identify design elements that create information-seeking difficulties

Review other university web sites

Determine how problematic elements should be changed and redesign accordingly

Old site: http://www.csufresno.edu/library/archive/

Development site: http://labs.lib.csufresno.edu/

New site: http://www.csufresno.edu/library/

Page 27: ACRL 2011 Data-Driven Library Web Design

Conclusion

Don’t be discouraged by the time or effort this will take…the results are worth it.

Pay attention to the internal political situation with your library web site development.

Use the collected data to overcome resistance to change.

Subsequent testing allows to validate changes and to identify areas for ongoing improvement.

Page 28: ACRL 2011 Data-Driven Library Web Design

Questions?

Comments?