ACRL 2011 Data-Driven Library Web Design
-
Upload
amanda-dinscore -
Category
Education
-
view
1.079 -
download
1
description
Transcript of ACRL 2011 Data-Driven Library Web Design
Data-Driven Library Web Design: Making Usability Testing Work with
Collaborative Partnerships
Allison Cowgill, Head of ReferenceAmanda Dinscore, Public Services Librarian
Patrick Newell, AUL for Information Technology and Electronic Resources
Henry Madden Library California State University, Fresno
All documents available at: http://www.slideshare.net/adinscore
Background
• The Library Study at Fresno State—Ethnographic study conducted by two anthropology professors• Study recommended that the Library’s web site should be should be redesigned
“Draw How You Feel When You Write a Paper.”
Drafting the Research Plan
Step 1: Create a Purpose Statement and Objectives
Purpose Statement: Should encapsulate the goals the team hopes to accomplish.
Example: “The purpose of this study is to determine if users can easily accomplish tasks required for research using the library’s web site.”
---------------------------------------------------------------------------Objectives: Use to develop the user tasks. Should
reflect actual user needs.
Example: “Determine the number of study participants who are able to search for and locate a book using the library’s web site.”
Activity: Create a draft
purpose statement and at
least 3 objectives.
Drafting the Research Plan
Step 2: Form the Team
Identify librarians and library staff who are sincerely interested in participating and support change.
Find collaborators outside of the library from academic departments such as Anthropology, Business, Computer Science, or Education. Everyone should be fully aware of the time and effort required.
Activity: Brainstorm
potential collaborators from
both within and outside your
library.
Drafting the Research Plan
Step 3: Identify User Tasks & Develop Questions
Consult with others (public service librarians, web team,…) • What is easy/difficult for users to do on our web site?• What do you spend time helping users do on our web site?
Create a list of tasks users are expected to perform and relate the tasks to the study objectives
Example tasks from our study: • Find a book title in our library• Find a book title through our patron-initiated borrowing system• Find a newspaper article• Find an article from a scholarly journal
Keep in mind the type of data you will collect & use
Task: An activity that fulfills an
information need
“All qualitative data can be coded quantitatively.”
http://www.socialresearchmethods.net/kb/qualdeb.php
Drafting the Research Plan
Step 3: Identify User Tasks & Develop Questions
Types of Data:Independent Variables: Variables you manipulate. Choose these based on your research questions.Dependent Variables (a.k.a. Outcome/Response Variables): Something you measure as the result of (based on the response to) the independent variables.
Quantitative Data: can be counted or expressed numerically Qualitative Data: nonnumeric information such as conversation, text, audio, or video.
Drafting the Research Plan
Step 3: Identify User Tasks & Develop Questions
Data Type Common Metrics Statistical Procedures
Nominal (categories) Task success (binary), errors (binary)
Frequencies, crosstabs, Chi-square
Ordinal (ranks) Severity ratings, rankings (designs)
Frequencies, crosstabs, chi-square, Wilcoxon rank sums, Spearman rank correlation
Interval Likert scale data, SUS scores All descriptive statistics, t-tests, ANOVAs, correlation, regression analysis
Ratio Completion time, time (visual attention), average task success (aggregated)
All descriptive statistics, t-tests, ANOVAs, correlation, regression analysis
Drafting the Research Plan
Step 3: Identify User Tasks & Develop Questions
Examples:
Using the library web site, find one journal article on swine flu.
Show me where on the web site you can find help using the library.
The library has a page with resources organized by subject. Show me how to find the page with history resources.
Activity: Brainstorm at least 3 tasks based on
the objectives you created.
What data will you use to measure the
tasks?
Drafting the Research Plan
Step 4: Determine the Study Population
Who?
How many?
Drafting the Research Plan
Step 4: Determine the Study Population
The Size of Your Population or Sub-GroupWhy Sample?• To say something about a population• A statistically valid sample size allows you to generalize to a population from a sample
Confidence Level • Tells you how sure you can be• Represents how often the true percentage of the population who would pick an answer lies within the confidence interval
Confidence Interval • a.k.a. “margin of error”• A range that estimates the true population for a statistic
Technique Advantages Disadvantages
Random sampling •Theoretically most accurate.•Influenced only by chance.
Sometimes a list of the entire population is unavailable or practical considerations or prevent random sampling.
Systematic sampling •Similar to random sampling.•Often easier than random sampling.
The system can sometimes be biased.
Quota sampling •Can be used when random sampling is impossible.•Quick to do.
There may still be biases not controlled by the quota system.
Stratified sampling •Ensures large enough sample to subdivide on important variables.•Needed when population is too large to list.•Can be combined with other techniques.
Can be biased if strata are given false weights, unless weighting procedure is used for overall analysis.
Drafting the Research Plan
Step 4: Determine the Study Population
Drafting the Research Plan
Step 4: Determine the Study Population
Your Recruitment Strategy
Consider: •Advertising Needs •Recruitment Location(s)
•Incentives
Drafting the Research Plan
Step 5: Room/Technology/Data Capture Considerations
Intake/subject data gathering location
Testing location
Equipment/staff to record the data• Hardware• Software• Who configures/operates/troubleshoots?
Privacy/data security considerations• Privacy and personal consent• Data back up and security
Drafting the Research Plan
Step 5: Room/Technology/Data Capture Considerations
Back up procedures
Equipment/staff to code the data
Equipment/staff to analyze the data
Activity: Make a list of the
resources available at your
own library. What might you
need?
Drafting the Research Plan
Step 6: Develop Scripts/Instructions & Train Moderators
Creating a script and instructions for moderators: • Helps them to clearly explain study procedures to subjects• Ensures uniformity throughout the process
Take advantage of collaborating teaching faculty’s expertise by enlisting their help to train student moderators.
Links: Moderator Instructions & Introduction Scripthttp://www.flickr.com/photos/lwr/4124641930/
sizes/l/in/faves-61999692@N00/
Drafting the Research Plan
Step 7: Pre-test & Refine
Pre-test using moderators as subjects and have moderators run each other through the protocols/tasks.
Identify any unclear or skewed questions and revise accordingly
Critical to remember that we are focusing on web site usability, not student ability or experience.
Drafting the Research Plan
Step 8:Test
Schedule rooms
Schedule helpers (moderators, recruiters, supervisors, etc.)
Verify all parts of the web site are working (surprise!)
Assure pre-tests and consent forms are present
Assure incentives are present
Back up data nightly (multiple times, on different media, if possible)
Drafting the Research Plan
Step 9: Analyze the Data
Prepare forms/spreadsheets for data processing
Code the qualitative data from video/audio• Develop codes beforehand• Review data/develop codes/apply codes
Calculate useful statistics from the data• Time on task (mean)• Completion rate (percentages)
Activity: Practice Coding
Drafting the Research Plan
Step 10: Communicate the Data
Meet with library colleagues, showing them video excerpts and sharing preliminary findings.
Develop clearly understandable graphs and other visuals that show how students navigate the web site and the difficulties they encounter.
Analyze and communicate qualitative data to stakeholders and address issues with an eye towards internal sensitivities.
Drafting the Research Plan
Step 10: Communicate the Data
• Add graphs here
Drafting the Research Plan
Step 10: Communicate the Data
Drafting the Research Plan
Step 10: Communicate the Data
Drafting the Research Plan
Step 10: Communicate the Data
Drafting the Research Plan
Step 10: Communicate the Data
Drafting the Research Plan
Step 10: Communicate the Data
Drafting the Research Plan
Step 11: Make Revisions to the Site
Meet with web team to discuss findings
Identify design elements that create information-seeking difficulties
Review other university web sites
Determine how problematic elements should be changed and redesign accordingly
Old site: http://www.csufresno.edu/library/archive/
Development site: http://labs.lib.csufresno.edu/
New site: http://www.csufresno.edu/library/
Conclusion
Don’t be discouraged by the time or effort this will take…the results are worth it.
Pay attention to the internal political situation with your library web site development.
Use the collected data to overcome resistance to change.
Subsequent testing allows to validate changes and to identify areas for ongoing improvement.
Questions?
Comments?