Discovery  or  Displacement?  A  Major  Longitudinal  Study  of  the  Effect  of  Web-­‐Scale  Discovery  Services  on  

Online  (Journal)  Usage  

SCELC  Colloquium  March  5,  2014  

 Michael  Levine-­‐Clark,  University  of  Denver  

John  McDonald,  University  of  Southern  California  Jason  Price,  SCELC  ConsorNum  

“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/

h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html  

Vendor  marke5ng  

Does  implementa4on  of  a  discovery  service  impact  usage  of  publisher-­‐

hosted  journal  content?  

What  did  we  measure?  

•  Whether  there  is  an  effect  

•  NOT  why  that  effect  exists  (that’s  a  future  study!)  

•  “Society  will  need  to  shed  some  of  its  obsession  for  causality  in  exchange  for  simple  correla5ons:  not  knowing  why,  but  only  what”    

•  Cukier  &  Mayer-­‐Schonberger.  2013.  Big  data:  A  revolu4on  that  will  transform  how  we  live,  work,  and  think.  

Data  collec5on  •  List  of  libraries  with  discovery  services  

> Searched  on  lib-­‐web-­‐cats  •  Surveyed  Libraries  

> Discovery  service  Implemented  > ImplementaNon  Date  (month/year)  > Search  box  locaNon  > MarkeNng  effort  

•  149  Libraries  Gave  Approval  > 33  libraries  selected  for  this  phase  > 6  for  each  of  the  4  major  discovery  services  and  a  group  of  9  libraries  with  no  service  

Dataset  •  33  Libraries  – 28  US,  2  CA,  1  each  from  UK,  AUS,  NZ  

– WorldCat  book  holdings  >  Average:  1,114,193  ;  Range:  ~300k  to  ~2.6mil  

•  ImplementaNon  dates  (Discovery  Libraries):    >  2010  (3),  2011  (19),  2012  (2)  

•  6  Publishers  •  9,206  Journals  •  163,545  Usable  ObservaNons    

Methodology  Compared  COUNTER  JR1  total  full  text  arNcle  views  for  the  12  months  before  vs  12  months  aeer  implementaNon  date  

             

June

 2010  

Start  

Implem

entaNo

n  May  2011  

May  2012  

End  

Year  1   Year  2  

Included  implementaNon  month  in  Year  1  to  ensure  that  both  periods  included  an  enNre  academic  year  

Examine  Data  for  Outliers  

Observa5ons  by  Library  &  Service  

Observa5ons  by  Publisher  

Average  Usage  Change  By  Discovery  &  Publisher  

Analyzing  Usage  Change:  %  vs  Total    

Use  12  months  before  

Use  12  months  aRer  

%  Change   Total  Change  

Journal  A   500   600   20%   100  Journal  B   5   15   200%   10  

Which  is  the  beier  measure?    

Is  it  the  same  for  publisher-­‐  &  journal-­‐level  data?      

Reducing  varia5on  due  to  ins5tu5on  size  Currently  converNng  to  change  per  FTE      Values  are  shown  as  x  1,000  to  bring  the  change  metric  back  per  journal-­‐library  combinaNon  to  a  minimum  of  0.1    2013  JISC  Discovery  study  took  a  similar  approach  

Average  Usage  Change  By  Discovery  &  Publisher  

Per  Journal  &  Per  10,000  FTE  

Full  Model  

Including  Discovery  Service,  Publisher,  and  Library  

Including  Discovery  Service,  Publisher,  and  Library  

Does the effect of discovery service differ across libraries?

Library  10-­‐15   Library  16-­‐21   Library  22-­‐27   Library  28-­‐33  Library  1-­‐9  

Nested  ANOVA  Model    

[all  three  factors  –  preliminary  results]  

Does  usage  change  vary  across  libraries?    

Institution (sorted by Mean Change)

Does  usage  change  vary  across  publishers?    

Publisher (sorted by Mean Change)

Does  usage  change  vary  across  discovery  services?  

Publisher

Does  the  effect  of  discovery  service  differ  across  publishers?    

Results  Can  we  detect  differences  between  Discovery  Services,  Publishers,  and/or  Libraries  and/or  their  interac4ons?    •  Library  –  Yes  •  Publisher  –  No  •  Discovery  Service  –  Yes  

•  DifferenNal  discovery  service  effect  by  publisher  –  Yes  

Next  Steps  •  Design  &  test  for  effects  of:    

–  Aggregator  full  text  availability  –  Publisher  Size  –  Journal  Subject  –  Overall  usage  trends  (Requires  Disc  Srvc  ‘control’)  –  ConfiguraNon  opNons  in  Discovery  services  

•  Expand  pool  of  libraries  •  Perhaps  explore  WHY    

   

Sharing  Data  •  With  par5cipa5ng  libraries  

–  Customized  reports  for  each  library  

•  With  par5cipa5ng  publishers  –  Customized  reports  for  each  publisher  –  PresentaNons  as  requested  

•  With  discovery  vendors  –  PresentaNons  as  requested  

•  In  publica5ons  and  presenta5ons  –  Maintaining  anonymity  of  data  

   

Doing  “Resarch”,  SCELC  Style!  

•  Why  SCELC?  •  SCELC  Funding  

–  staNsNcs  consultant    –  research  &  wriNng  retreats  –  See  hip://bit.ly/1dNMDL3  for  more  detail  

•  SCELC  libraries  encouraged  to  parNcipate  in  next  round  –  Survey:  hip://bit.ly/DSparNcipaNon