User-Centered Evaluation of a Discovery Layer System with Google Scholar

User-Centered Evaluation of a Discovery Layer System with Google Scholar

Tao ZhangDigital User Experience SpecialistPurdue University [email protected]

Thursday, July 25, 13

mailto:[email protected]


Library Website is still Important

Breadth of scholarly resourcesCredible, highly relevantGuaranteed full text access


User Expectation

“The problem with the databases is that you can’t content yourself with just searching one place, like you can with Google. It is hard for the students to differentiate between the different databases.”

Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006


Discovery Layer System - Ex Libris Primo®• Uni"ed search interface for (almost) all library resources

• Catalogs, databases, e-journals, digital collections, and repositories

• Increased discoverability

• Easier, faster, and more efficient search interface

• Faceted browsing/"ltering of search results (e.g., by format, date, subject, topic, etc.)

• Record grouping (e.g., multiple editions or versions of the same material)

• Consistent “Get it” option


Study Goals• Compare users’ subjective assessment of search

results from a discovery layer system (Ex Libris Primo) and Google Scholar

• Evaluate the discovery layer system for usability issues


Method• 12 graduate students (mean age = 26.3, SD = 3.2)

• Search tasks on Primo and Google Scholar

• Given topic for "rst three tasks: classroom assessment techniques, food addition, natural language processing

• Participant own keywords for fourth search task

• Rate "rst ten results from 1 (highly irrelevant) to 7 (highly relevant)

• Within-subject design, balanced order of search systems, randomized tasks

• After search tasks: System Usability Scale (SUS) ratings, preference ratings, open questions


Relevancy Ratings

Search 1: Article 9 and 10 signi!cantly different

1"

2"

3"

4"

5"

6"

7"

8"

Ar,cle"1"Ra,ng"

Ar,cle"2"Ra,ng"

Ar,cle"3"Ra,ng"

Ar,cle"4"Ra,ng"

Ar,cle"5"Ra,ng"

Ar,cle"6"Ra,ng"

Ar,cle"7"Ra,ng"

Ar,cle"8"Ra,ng"

Ar,cle"9"Ra,ng"

Ar,cle"10"Ra,ng"

Google"Scholar"

Primo"


Relevancy Ratings

Search 2: Article 10 signi!cantly different

1"

2"

3"

4"

5"

6"

7"

8"

Ar,cle"1"Ra,ng"

Ar,cle"2"Ra,ng"

Ar,cle"3"Ra,ng"

Ar,cle"4"Ra,ng"

Ar,cle"5"Ra,ng"

Ar,cle"6"Ra,ng"

Ar,cle"7"Ra,ng"

Ar,cle"8"Ra,ng"

Ar,cle"9"Ra,ng"

Ar,cle"10"Ra,ng"

Google"Scholar"

Primo"


Relevancy Ratings


1"

2"

3"

4"

5"

6"

7"

8"

9"

Ar-cle"1"Ra-ng"

Ar-cle"2"Ra-ng"

Ar-cle"3"Ra-ng"

Ar-cle"4"Ra-ng"

Ar-cle"5"Ra-ng"

Ar-cle"6"Ra-ng"

Ar-cle"7"Ra-ng"

Ar-cle"8"Ra-ng"

Ar-cle"9"Ra-ng"

Ar-cle"10"Ra-ng"

Google"Scholar"

Primo"


Relevancy Ratings


1"

2"

3"

4"

5"

6"

7"

8"

Ar,cle"1"Ra,ng"

Ar,cle"2"Ra,ng"

Ar,cle"3"Ra,ng"

Ar,cle"4"Ra,ng"

Ar,cle"5"Ra,ng"

Ar,cle"6"Ra,ng"

Ar,cle"7"Ra,ng"

Ar,cle"8"Ra,ng"

Ar,cle"9"Ra,ng"

Ar,cle"10"Ra,ng"

Google"Scholar"

Primo"


Results• No signi!cant difference of relevancy ratings between

given and participant chosen searches

• Usability ratings and preference of Google Scholar signi!cantly higher than Primo

• SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43)

• Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38)

• Usability correlated to preference ratings (r = 0.59, p < 0.01)


Usability Issues of Primo - Work$owSearch Results List

Identify Article

Link Resolver

Article Page


Usability Issues of Primo

!


Usability Issues - Consistency


ConclusionPrimo’s search relevancy may be on par with Google Scholar, but usability issues have affected users’ perceived usability and preference.

User interface of discovery layer systems needs to address work$ow and consistency issues.


What’s More• “Blind” test on relevancy ratings

• Test large sample of search keywords

• Server log analysis on keywords, facets, session length, etc.

• Better (Google Scholar-like?) link resolver

• UI improvement


[email protected]




User-Centered Evaluation of a Discovery Layer System with Google Scholar

Design

Transcript of User-Centered Evaluation of a Discovery Layer System with Google Scholar