Discovery or Displacement? A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online

Journal Usage

July 23, 2014

Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California

Jason Price, SCELC Consortium

Why Do We Use Discovery Services?

• Too many sources of information– Specialized– Confusing– What’s the right specialized database for my

subject?

• Why do you search in one place for an article, another place for a different article, another for a book?

Terminology

Discovery Tool = Discovery Layer = Discovery Service = Discovery System

Web-scale discovery services

• Single source for finding information– Books– Articles– Local content

• Metadata and/or full text

• Content is pre-indexed and/or pre-harvested

• Single fast search

ILS

HathiTrust

MLA Bibliograph

y

Institutional Repository

Publisher Metadata

Discovery Service

Does implementation of a discovery service impact usage of publisher-

hosted journal content?

What did we measure?

• Whether there is an effect

• NOT why that effect exists (that’s a future study!)

Publisher-hosted journals are only part of the picture

eBooks, pBooks, newspaper articles, aggregator journal content, etc.

publisher journal content

The six publishers in this study

52.3

12.41

2.8

32.5

Open web searchLibrary referralsSocial mediaAcademicN/A

Journals Traffic Sources (SAGE, Conrad ALPSP 2013)

An assumption

• At any given institution, given a relatively stable user base, the total search effort will remain roughly the same.

Discovery services Will take up an increasing amount of a finite

time for searching

Will draw users from other (more or less efficient) search tools

Will alter the overall productivity of searches (users will find more or less)

Will alter the overall efficiency of users (users will access more or less full-text)

Dataset• 33 Libraries– 28 US, 2 CA, 1 each from UK, AUS, NZ

– WorldCat book holdings>Average: 1,114,193 ; Range: ~300k to ~2.6mil

– 4 discovery groups, of 6 libraries each

– 1 control group, 9 libraries• Implementation dates (Discovery Libraries):

>2010 (3), 2011 (19), 2012 (2)

• 6 Publishers• 9,206 Journals• 163,545 Usable Observations

Methodology

Compared COUNTER JR1 total full text article views for the

12 months before vs 12 months after implementation date

June

201

0St

art

Impl

emen

tatio

nM

ay 2

011

May

201

2En

d

Year 1 Year 2

Included implementation month in Year 1 to ensure that both periods included an entire academic year

Examine Data for Outliers

Analyzing Usage Change: % vs Total

Use 12 months before

Use 12 months

after% Change Total

Change

Journal A 500 600 20% 100

Journal B 5 15 200% 10

Which is the better measure?

Is it the same for publisher- & journal-level data?

Observations by Publisher

Journals by Library & Service

EDS Primo Summon WorldCat Control

Average Journal Usage by Library

Testable Effects

• Discovery Tool– Implemented by multiple libraries– Used to find content from all publishers

• Publisher– Accessible in all discovery tools– Accessible across multiple libraries

• Library– Uses content from multiple publishers– Uses only one discovery tool (so only within

DT)

Full Model

Including Discovery Service, Publisher, and Library

Including Discovery Service, Publisher, and Library

Nested ANOVA Model Results

How does usage change differ across discovery services?

ABB

C

D

Letters indicate statistically significant differences (Tukey multiple comparisons, p < .05)

How does usage change differ across publishers?

Publisher (sorted by Mean Change)

C

Letters indicate statistically significant differences (Tukey multiple comparisons, p < .01)

D

BBB

A

How does usage change differ across publishers?

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Usage Change Per Institution: All Journals

Control EDS Primo Summon WCL

Usage Change By Institution: Pub 16.32

Primo WCLSummonEDSControl

Usage Change By Institution: Pub 2

Control EDS Primo Summon WCL

Usage Change By Institution: Pub 3

WCLEDS PrimoControl Summon

Usage Change By Institution: Pub 4

WCLEDSControl Primo Summon

Usage Change Per Institution: Pub 5

Control EDS Primo Summon WCL

Usage Change by Institution: Pub 6

WCLSummonPrimoEDSControl

Publisher 1 by Discovery System

Publisher 2 by Discovery System

Publisher 3 by Discovery System

Publisher 4 by Discovery System

Publisher 5 by Discovery System

Publisher 6 by Discovery System

Next Steps• Design & test for effects of:

–Aggregator full text availability–Journal age (archive vs current)–Journal Subject–Overall usage trends–Configuration options in Discovery services

• Expand pool of libraries• Perhaps explore WHY

Sharing Data• With participating libraries

–Customized reports for each library

• With participating publishers–Customized reports for each publisher–Presentations as requested

• With discovery vendors–Presentations as requested

• In publications and presentations–Maintaining anonymity of data

Presentations• Ithaka Sustainable Scholarship Conference (October 2013)• Charleston Conference (Nov 2013) http://sched.co/17A3Kun

• ER&L/Library Journal Webinar (December 2013)• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)• SCELC Colloquium (March 2014) http://goo.gl/WmJoIw

• ER&L (Mar 2014) http://bit.ly/discovery-impact-erl2014 • UKSG (April 2014) https://www.youtube.com/watch?v=d_2ycMk_9fA

• Presentations to three publishers (Spring-Summer 2014)–More to come