Usability, the User Experience & Interface Design:

The Role of ReferenceJuly 30, 2013

Evaluating user behaviors and experiences

Summon discovery service at Virginia Tech

2

ByTracy Gilmore & Lesley Moyo

3

Introduction

• Continual assessment necessary in providing proactive user support

• At Virginia Tech we endeavor to gather data on user behaviors every spring

• Study of user experiences with discovery services is our current focus

• Summon is our choice of Web scale discovery services

Collecting Data and Observing Behaviors

• Virginia Tech has adopted a mixed methods approach to studying user research behaviors

1. Surveys2. Focus Groups & One-on-one interviews3. Usability task analysis

• This helps triangulate the data and gives us grounds for making strong inferences

4

Unveiling the Data

• Using a robust survey tool is very helpful

• We used Qualtrics Research Suite: www.qualtrics.com

• Qualtrics offers strong data analysis features

• We crunched some of the survey data using SPSS to get more detailed analysis

• NVIVO for analysis of interviews and focus group data

6

What we learned about user behavior and preferences

Survey Results – User Preferences

Preference for Summon:

• Stronger than Google Scholar

• Stronger than the Library Catalog

7Easy to use Moderately easy to use Neither easy nor difficult Moderately difficult use Difficult to useHigh preference Moderate preference Neither high nor low preference Somewhat low preference Low preference

User Preference - 2012

User Preference - 2013

2012 Survey results - Predictors of satisfaction

8

Chi Square

Degrees of freedom

P-value

My rating of Ease-of-use of Summon

Chi Square

Degrees of freedom

P-value

Overall I am very satisfied with Summon

41.40

16

0.00

I always use the facets in Summon to improve my search strategy

Chi Square

Degrees of freedom

P-value

Overall I am very satisfied with Summon

246.68

16

0.00

I can find articles from top ranked journals in my field

Ease of use & Satisfaction Facets use & Satisfaction

Top ranked journals & Satisfaction

Overall I am very satisfied with Summon

229.90

16

0.00

2013 Survey results - Predictors of satisfaction

9

Chi Square

Degrees of freedom

P-value

My rating of Ease-of-use of Summon

Chi Square

Degrees of freedom

P-value

Overall I am very satisfied with Summon

54.90

16

0.00

I always use the facets in Summon to improve my search strategy

Chi Square

Degrees of freedom

P-value

Overall I am very satisfied with Summon

436.74

16

0.00

I can find articles from top ranked journals in my field

Ease of use & Satisfaction Facets use & Satisfaction

Top ranked journals & Satisfaction

Overall I am very satisfied with Summon

263.08

16

0.00

User voices - Open ended questionsComments 2012

• I like that I only have to search in one place

• On several occasions Summon has found some good papers that Google Scholar didn't

• Summon is great if I know the article I’m looking for

• I need more skill at searching

• I get so many hits that are not useful

Comments 2013• The faceted searching (I didn’t

know it was called that) is absolutely essential

• Ability to get many different types of sources with one search

• I found it to be more helpful when looking for books than articles

• Even with facets, Summon casts too broad a net for my needs

10

User voices - Interviews/Focus groups

Comments• Summon is easy to use

• Facets are a great tool

• Too many results!

• Boolean operators please

• Training would be nice

Suggestions• Provide direct linking to

articles

• Provide more instruction on how to use Summon:

• Using facets• Using Subject Terms

11

Task Analysis observations

• Good results achieved with advanced search

• Some participants able to effectively use facets

• Effective use of quotations for known items

Challenges• Page navigation• Evident lack of familiarity with

Summon features• Large result sets still an issue• Mixed success rates in task

performance• Get VText (360 link resolver)

12

Strengths

Enter librarian perspectives

• In spring 2013 - We included librarian voices

• This provided us with comparative perspectives

13

How do we as librarians interpret user behaviors, given our own biases of what research processes should look like?

Librarian voices

14

More students use it as a starting place, so when troubleshooting that's what's involved more often than not

Summon has been especially helpful with interdisciplinary topics even in identifying specific databases to more fully explore

Fast way of gathering extensive information and familiar to the "Google generation”

I am a very happy user of Summon. The interface, the single user box, and the facets are intuitive and easy to use

Changes implemented from findings

• Better integration in the instruction program especially at the freshman level

• Improvements to the interface - better direct linking to full text

• Increase in the content index - emphasis on eBooks and reference sources

• Better visibility on the Library homepage

15

Going forward - Continuing assessment and evaluation

• Implementation of Summon 2.0 (anticipated fall 2013)

• Analysis of usage statistics from Google Analytics added to Summon in November 2012

• Expansion of Mixed Methods study to academic research libraries utilizing the Summon discovery service (fall 2013)

Contact informationLesley MoyoDirector for Research & Instructional ServicesUniversity LibrariesVirginia Tech 560 Drillfield DriveBlacksburg, VA  24061Tel: 540-231-2708Mobile: 814-777-7565E-mail: moyo@vt.eduLibrary Web: www.lib.vt.edu

Tracy GilmoreCollections Assessment Librarian

University LibrariesVirginia Tech 560 Drillfield Drive

Blacksburg, VA  24061Tel: 540-231-4956

Mobile: 540-239-4384E-mail: tgilmore@vt.edu

Library Web: www.lib.vt.edu

17