ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL

ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL

Association of College and Research LibrariesACRL Conference 2005

MinneapolisApril 7, 2005

Toni OlshenYork University

www.arl.org/stats/

MINES for Libraries

www.minesforlibraries.org

Ontario Council of University Libraries

• OCUL is a consortium of twenty university libraries in the province of Ontario

• The member libraries cooperate to enhance information services through: resource sharing, collective purchasing, document delivery and many other similar activities.


OCUL Members – Those in Green are ARL Libraries

Brock University

Carleton University

University of Guelph

Lakehead University

Laurentian University

McMaster University

Nipissing University

Ontario College of Art & Design

University of Ontario Institute of Technology

University of Ottawa

Queen's University

Royal Military College of Canada

Ryerson Polytechnic University

University of Toronto

Trent University

University of Waterloo

University of Western Ontario

Wilfrid Laurier University

University of Windsor

York University


Member Institution Enrolments (2003)- Total Undergraduate and graduate students

• University of Ontario Institute of Technology 936

• Royal Military College of Canada 1,941

• Ontario College of Art and Design 3,062

• Nipissing University 5,478• Lakehead University 7,304• Trent University 7,388• Laurentian University 8,751• Wilfrid Laurier University 12,426• Brock University 15,527

• University of Windsor 16,266• University of Guelph 19,096• Queen's University 20,034• McMaster University 22,064• Carleton University 22,535• University of Waterloo 25,029• Ryerson University 27,221• University of Ottawa 30,948• University of Western Ontario 32,784 • York University 46,794 • University of Toronto 68,290

• Total 391,933 • 90% undergrads 10% graduate• 12,500 faculty


Scholars Portal – What is it?

• A unique set of shared information resources and services

• Resources acquired and managed through OCUL with funding support from a 5-year grant from the Ontario Innovation Trust (OII), a provincial funding body

• Resources are made available to researchers and students in Ontario through their own university libraries


Scholars PortalOntario Council of University Libraries (OCUL)

• Ontario Information Infrastructure (OII) funded by the Ontario Innovation Trust in 2001 for five years

• Consortia-purchased electronic resources offered through the Ontario Scholars Portal

• March 2004, we began the evaluation phase of $7.6 million dollar OII project


Scholarly Information Resources

• As of the end of March, contains 7,547,904 full text articles from 6,783 full text journals published by 12 academic publishers

• Coverage of most disciplines but concentration in

sciences

• Current and historic coverage

• One of the largest collections of electronic

journals available to researchers anywhere


Scholars Portal Resources

• Academic Press, • American Psychological Association, • American Chemical Society, • Berkeley Electronic Press, • Cambridge University Press, • Emerald Publishing, • Elsevier Science (Elsevier Science, Harcourt Health Sciences), • Kluwer (Kluwer Academic Publishers, Kluwer Law International and

Kluwer/Plenum), • Oxford University Press, • Project MUSE, • Springer-Verlag, and • John Wiley & Sons.

http://scholarsportal.info/cgi-bin/publisher_list.cgi?pub_name=academicpress

http://scholarsportal.info/cgi-bin/publisher_list.cgi?pub_name=apa


Scholars Portal – Project Goals

• Centrally mount and deliver information resources acquired through OCUL consortia purchases to ensure rapid and reliable access

• Provide for the long term, secure archiving of resources to ensure continued availability


Scholars Portal – Project Goals

• Ensure that the resources and services provided meet the needs of faculty, students and staff.

• Ensure that resources and services can be seamlessly integrated to the local library and information systems


Measuring Success

• OCUL provides a sophisticated statistical report mechanism. (see next slide). Download statistics are a rough measure of value but we need more to properly assess impacts.

• Need to measure also the significance forresearch of access to e-journals

• Employing ARL MINES Survey methodology to capture information on how resources are being used (from where, by whom, and for what purposes)


SP Statistics and Report Generator


Why Evaluation?

• Feedback to OII and University funders

• Understand who, where, and why the digital resources are used

• Supplement usage numbers to answer the key question:

What is the impact of Portal content on research at Ontario academic libraries?


Evaluating Success

• Evaluating Scholars Portal from user and staff points of view

• Use a mix of quantitative and qualitative tools for a richer assessment: MINES, focus groups, staff survey

• Are OII projects improving research services?

• Does Scholars Portal meet OCUL user and staff expectations?


MINES (Measuring the Impact of Networked Electronic Services)

• MINES survey is one of a new breed of assessment tools that did not exist before because services were not digital.


MINES (Measuring the Impact of Networked Electronic Services)-Desired Outcomes

• To capture in-library and remote web usage of the Scholars Portal in a sound representative sample using MINES methodology;

• To identify the demographic differences between in-house library users as compared to remote users by status of user;


MINES (Measuring the Impact of Networked Electronic Services)-Desired Outcomes

• To identify users’ purposes for accessing Scholars Portal electronic services (funded research, non-funded research, instruction/education use, student research papers and course work);

• To assist with the evaluation of the project as well as to capture information for OCUL about indirect research costs; and

• To develop an infrastructure to make studies of patron usage of networked electronic resources routine, robust and integrated into the decision-making process.


MINES Methodology

• What user groups use SP?• What specific resources are used?• From where?• How do users learn about SP?• Are there differences in the use of digital

resources based on the user's location?• Why use SP? (sponsored research?

Instruction? patient care?)• Does use differ by discipline? user group?

location?


MINES Methodology

• Web-based surveys conducted over the course of a year for each institution

• Activated during randomly selected 2-hour survey periods each month as users access one of SP’s journals

• Mandatory, short, and anonymous


ARL/MINES – Jan. ’04-Dec. ‘05

• ARL developed random schedule of two-hour sessions per month

• OCUL designed local questions, mounted survey, collects and sends data to ARL

• ARL compiles survey results for all sites

• ARL reports findings on a semi-annual basis

• ARL presents findings and final report to project participants on an aggregated and individual institution basis


Development of survey form

• Finding balance between simplicity, ease and richness of data elements

• Bilingual – University of Ottawa, Laurentian University, Glendon College at York University

• Ultimately a change in focus to the creation of a unique data set


MINES Survey Form – Five Questions and a Comment Box


Survey Form

• Survey form determined :– users’ status– Discipline (affiliation)– location or where accessed from– purpose of use (sponsored research,

instruction, patient care, course work) – how the resource was identified

(bibliography, colleague, librarian, important journal in field etc.)


OCUL Definition of Usage for MINES

• A successful search connecting the user to an article of interest for viewing, printing or downloading

• Unique to Scholars Portal because of consortia server setup and archiving of all journals


MINES Methodology

• Random sampling plan and the mandatory nature of the questions are both required to create a statistically sound study

• If the survey is not mandatory, the group of non-respondents is likely to be different from the group of respondents, and we will not know what that difference is

• One of the strengths and innovations of this survey technique is that it is based upon actual use, not on predicted, intended, or remembered use


OCUL Implementation of MINES

• Once the survey is completed, the respondent's browser is forwarded to the desired networked electronic resource

• If more than one search is carried out, the survey form is auto-populated with user’s responses as defaults which only have to change if response is different


Informed Consent

• Because this is a Web-based survey, the respondents consent to participate by electing to fill out the survey questionnaire

• It is the participating library’s responsibility to provide an explanation of the survey and information pertaining to its confidentiality


Confidentiality of Data

• Institutional data are confidential. Individual institutions and/or their specific data will not be identified.

• Individual data are anonymous. The respondent’s privacy is protected because only very indirect information is captured, which would be difficult to trace back to an individual.


Ethics Review

• A major step was contacting research ethics officers and/or Ethics Review Boards to get approval, where necessary, to run the survey

• Purpose of ethics reviews for human subjects is to prevent putting subjects at risk

• Officers/Boards on 16 OCUL campuses accepted that no physical or psychological harm would come to library users who are asked to fill out a brief mandatory anonymous survey before they are connected to the title of their choice.


Ethics Review

• Reference to interesting opinion piece by J. Paul Grayson. “How Ethics Committees are Killing Survey Research on Canadian Students”. University Affairs, January 2004.

http://www.universityaffairs.ca/issues/2004/jan/print/opinion.html


Mandatory Survey

• If individuals chose to avoid filling out the brief anonymous survey, they might be inconvenienced for a maximum of a two-hour period, but they would not be harmed

• We needed to balance good data for making decisions and the inconvenience caused to the user.


Ethics Review – Issues and Problems

• Mandatory nature of the survey required discussion on some campuses

• Several campuses did not require approval because the survey fell into quality assurance guidelines and was seen as a library management tool (8)

• Several schools received approval after an application process (8)

• One Library and Review Board did not support the mandatory nature of the methodology so that school dropped out of the project.


Pre-testing and False start – January – March 2004

• ARL prepared a schedule for the random two-hour monthly runs.

• A test run was planned at York and Wilfrid Laurier in January with the real survey commencing at the end of February.

• The pilot in January failed at York and highlighted

the need for all institutions to be using a link resolver URL when connecting to SP journals from their catalogues or eResources databases.

• Each site reviewed their configuration and

necessary changes were made.


Pre-testing and False start – January – March 2004

• Survey form and the explanatory material were translated into French for bilingual Ottawa, Laurentian, and Glendon College at York.

• February run highlighted concerns about the data collection.

• The technical infrastructure was capturing only access through library catalogues or eresource databases, but not from the use of the SP directly.

• There were some technical problems with the February and March runs and the validity of the data was under question. The data-collection programming was revisited.


Lessons Learned

• Early runs taught us a great deal about the different ways OCUL libraries access the SP

• We needed to reflect that in the data gathering


Lessons Learned

• As originally planned, we now capture as much usage as possible that comes from :

– local eresource databases

– library catalogues

– Scholars Portal browse and search functions.


New Definition of Usage for MINES

• A successful search is now defined as connecting the user to an article of interest for viewing, downloading or printing

• Definition is unique to Scholars Portal because of consortial server setup and archiving of content

• We cancelled the April 20 run and reset the dates of the survey from May 2004 through April 2005, considering the February and March runs as tests.


New Definition of Usage for MINES - Innovation

• We continue to build on the unique opportunity we have to gather useful data that is not open to other types of library groups. By the end of March about 22,500 surveys have been completed. One more month to go!

• By implementing the MINES survey, OCUL is ahead of other projects in that we are not held "hostage" to the limitations and inconsistencies of vendor statistics

• We have opportunities to disseminate research on measurement of networked resources through conferences and publications


MINES Very Preliminary Output: MAY –AUGUST 2004

5223 respondents


Very Preliminary Findings – 4 months of data -Subject Affiliation

• Applied Sciences• Business • Education• Environmental Studies• Fine Arts• Humanities• Law• Medical Health• Sciences • Social Sciences• Other

• 804 17.5%• 146 3.2• 176 3.8• 160 3.5• 22 .5• 93 2.0• 21 .5• 1341 29.2 • 1031 22.4• 673 14.6• 129 2.8


Very Preliminary Findings – 4 months of dataUser Status

• Faculty

• Graduate/Professional

• Undergraduate

• Library Staff

• Staff

• Other

• 764 16.6%

• 2068 45.0

• 1039 22.6

• 47 1.0

• 427 9.3

• 251 5.5


Very Preliminary Findings –4 months of data - Location

• Library

• Off-Campus

• On-Campus ( but not in the library)

• 578 12.6%

• 1978 43.6

• 2040 44.4


Very Preliminary Findings-4 months of data - Purpose of Use

• Sponsored research• Other non-

sponsored research• Teaching• Course work• Patient care• Other activities

• 2189 47.6%

• 919 20.0• 278 6.0• 686 14.9• 143 3.1• 381 8.3


Cross Tabulations

• Purpose of use by affiliation, user status, location, why

• Location by affiliation, user status, purpose of use, why

• Why by affiliation, user status, location, purpose of use

• Which titles used by which users for which purposes


Location and Purpose of Use


Additional Qualitative Data

• MINES Survey respondent comments

• Staff Survey: What does the range of institutional experiences reveal?

• Focus Groups: What anecdotal data can faculty and students add to the development of the Scholars Portal?


Thank you for your attention!

• Questions?

• Toni Olshen: [email protected]

ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL

Documents

Transcript of ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL