Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for...

Measuring the Impact of Networked Electronic Services: Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for Libraries, Developing Assessment Infrastructure for Libraries,

State, and Other Types of ConsortiaState, and Other Types of Consortia

Presented By:Terry Plum, Simmons GSLIS

Brinley Franklin, University of ConnecticutMartha Kyrillidou, ARL

Gary Roebuck, ARLRaynna Bowlby, ARL Consultant & Simmons GSLIS

MaShana Davis, ARL Kristina Justh, ARL

Library Assessment Conference 2008University of Washington

Seattle, WA

http://www.minesforlibraries.org/

ARL New Measures Toolkit: ARL New Measures Toolkit: StatsQUALStatsQUAL®®

LibQUAL+® is a rigorously tested Web-

based survey that libraries use to solicit, track, understand, and

act upon users‘ opinions of service

quality.

LibQUAL+®

DigiQUAL®

The DigiQUAL® online survey designed for

users of digital libraries that measures reliability and trustworthiness of

Web sites. DigiQUAL® is an adaptation of

LibQUAL+® in the digital environment.

MINES for Libraries®

Measuring the Impact of Networked Electronic

Resources (MINES) is an online transaction- based survey that

collects data on the purpose of use of

electronic resources and the demographics

of users.

ARL Statistics™

ARL Statistics™ is a series of annual publications that

describe the collections, expenditures, staffing,

and service activities for Association of Research Libraries (ARL) member

libraries.

ClimateQUAL™

ClimateQUAL™: Organizational Climate

and Diversity Assessment is an online

survey that measures staff perceptions about:

(a) the library's commitment to the

principles of diversity, (b) organizational

policies and procedures, and (c) staff attitudes.


What is MINES?What is MINES?

Action Research Historically rooted in indirect cost studies Set of recommendations for research design Set of recommendations for web survey

presentation Set of recommendations for information

architecture in libraries Plan for continual assessment of networked

electronic resources An opportunity to benchmark across libraries


What is MINES? What is MINES? (cont’d)(cont’d)

Measuring the Impact of Networked Electronic Services (MINES)

MINES is a research methodology that measures the usage of networked electronic resources of a library or consortium by a specific category of the patron population.

MINES is a Web-based survey form consisting of 5 questions that is administered at the time of transaction.

MINES measures: User status and discipline/affiliation (who) Physical location (where) Primary purpose and reason of use (why)

MINES a part of Association of Research Libraries’ New Measures & Assessment Initiatives (since 2003).

MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+®).

MaShana Davis

Discussion randomization here?


Early Data Collection ActivitiesEarly Data Collection Activities

Academic Medical Libraries U of Connecticut Health

Center U of North Carolina U of Texas Medical Branch U of Texas Southwestern U of Utah U of Virginia Washington U

Main Universities U of Colorado U of Connecticut U of North Carolina Oregon State U U of Utah U of Virginia Washington U

Data was collected at seven main campus libraries and seven academic health sciences libraries in the U.S. between 2003 and 2005.


Early Data Collection ActivitiesEarly Data Collection Activities

More than 45,000 networked electronic services uses were surveyed.

At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.


Recent Data Collection Recent Data Collection Activities via ARLActivities via ARL

Ontario Council of University Libraries (OCUL)

University of Iowa Libraries University of Macedonia


Questions AddressedQuestions Addressed

How extensively do sponsored researchers use the new digital information environment?

Are researchers more likely to use networked electronic resources from inside or outside the library?

Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

Are particular network configurations more conducive to studies of digital libraries patron use?


MINES for LibrariesMINES for Libraries®® Survey Form Survey FormFive Questions and a Comments BoxFive Questions and a Comments Box

Methodological Considerations:Methodological Considerations:Experience with the MINES SurveyExperience with the MINES Survey

Terry PlumAssistant DeanSimmons GSLIS


Seattle, WA


Issues with Web surveysIssues with Web surveys

Research design Coverage error

Unequal access to the Internet Internet users are different than non-users

Response rate Response representativeness

Random sampling and inference Non-respondents

Data security


MINES StrategyMINES Strategy

A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.

Random moment/web-based surveys are employed at each site.

Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time. IRB waiver or approval

Libraries with database-to-web gateways or proxy re-writers offer a comprehensive networking solution for surveying all networked services users during survey periods.


MINES StrategyMINES Strategy

Placement Point of use Not remembered, predicted or critical incident

Usage rather than user What about multiple usages Time out ? Cookie or other mechanism with auto-population or

more recently counting invisibly with a time out. Distinguish patron association with libraries.

For example, medical library v. main library. But what if the resources are purchased across

campus for all. Then how to get patron affiliation?


Web Survey Design GuidelinesWeb Survey Design Guidelines

Web survey design guidelines that MINES followed: Presentation

Simple text for different browsers – no graphics Different browsers render web pages differently

Few questions per screen or simply few questions Easy to navigate Short and plain No scrolling Clear and encouraging error or warning messages Every question answered in a similar way - consistent

Radio buttons, drop downs ADA compliant Introduction page or paragraph Easy to read

Must see definitions of sponsored research. Can present questions in response to answers – for example if

sponsored research was chosen, could present another survey


Quality ChecksQuality Checks

Target population is the population frame – surveyed the patrons who were supposed to be surveyed - except in libraries with outstanding open digital collections.

Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.

Alter order of questions and answers, particularly sponsored and instruction.

Spot check IP against self-identified location Spot check undergraduates choosing sponsored research –

measurement error Check self-identified grant information against actual grants Content validity – discussed with librarians and pre-tested. Turn-aways – number who elected not to fill out the survey Library information architecture -- Gateway v. HTML pages –

there is a substantial difference in results.

Documenting the Purpose and Use of Documenting the Purpose and Use of Electronic Resources:Electronic Resources:

Experience with the MINES SurveyExperience with the MINES SurveyBrinley Franklin

Vice Provost for University LibrariesUniversity of Connecticut


Seattle, WA


““It is useless to tell the acquisitions librarian that half the It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify monographs ordered will never be used, unless we can specify

which 50% to avoid buying.” (Galvin and Kent, 1977)which 50% to avoid buying.” (Galvin and Kent, 1977)


Reliance on Vendor StatisticsReliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.


The most popular current method of measuring usage of electronic resources by libraries is not through web-based usage surveys, but through vendor supplied data of library patron usage or transaction usage.

Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs.

Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries®)” IFLA Journal 32 (1) March, 2006

Measuring Digital Content UseMeasuring Digital Content Use


A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan.

MINES for Libraries® has been administered at 40 North American universities in the last four years. More than 100,000 North American networked services users have been surveyed using a standard protocol.

Measuring Digital Content UseMeasuring Digital Content Use


Library User SurveyLibrary User Survey


Library User SurveyPatron Status


Library User SurveyAffiliation


Library User SurveyLocation


Library User SurveyPurpose


Sample Survey Data File GeneratedSample Survey Data File GeneratedOther UConnInstruction/Education/Departmental (Non-Funded)

Researchhttp://newfirstsearch.oclc.org/done=referer;dbname=WorldCat;autho=100122319;FSIP12:36:5012/3/2004Off CampusUConn Faculty12.101.104.84

Family StudiesInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/cgi-bin/jstor/gensearch12:37:4312/3/2004Off CampusUConn Undergraduate Student12.101.104.84Non-UConnOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:08:4112/3/2004Off CampusNon-UConn12.18.36.40

Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html13:31:2912/3/2004Off CampusNon-UConn12.76.131.246

Non-UConnOther Activitieshttp://magic.lib.uconn.edu/index_real.html12:11:0612/3/2004Off CampusNon-UConn130.132.86.156Agriculture & Natural ResourcesInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html12:33:5712/3/2004

Off CampusNon-UConn131.128.89.54EducationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.euromonitor.com/womdas/12:57:4412/3/2004

Off CampusNon-UConn134.241.135.70Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004

Off CampusNon-UConn134.74.42.82Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty137.99.1.122

Liberal Arts & SciencesOther Activitieshttp://www.siam.org/journals/simax/simax.htm12:52:1712/3/2004On Campus - StorrsUConn Graduate Student137.99.1.122

EngineeringInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:04:3112/3/2004On Campus - StorrsUConn Graduate Student137.99.1.219

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/pqdweb?RQT=31812:16:3312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/journals/07322399.html12:16:5212/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:29:5312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:48:4112/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/login?COPT=SU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD&clientId=4899612:04:2312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175


24%

25%

8%

43%

Graduate Students

Faculty, Staff,Research Fellows

UndergraduateStudents

All Other Users

40%

31%

25%

4%

28%

11%

21%

40%

On Campus, Not in the Libraryn = 6,391

Inside the Libraryn = 9,172

Off-Campusn = 4,953

MINES for LibrariesMINES for Libraries®®

Demographics by Location of UserU.S. Main Libraries


14%

14%

69%

3%

Graduate Students

Faculty, Staff,Research Fellows

UndergraduateStudents

All Other Users

Inside the Libraryn = 4,047

On Campus, Not in the Libraryn = 7,090

45%

24%

29%

2%

31%

16%

48%

5%


Demographics by Location of UserOntario Council of University Libraries

Off-Campusn = 9,163


Purpose of UsePurpose of Use

Are users engaged in coursework, funded (or unfunded) research, public service, patient care, or other activities?


26%3%

5%

SponsoredResearch

Instruction

Other

Other SponsoredActivities

*72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus.

63%

14%2%

21%

11%

62%

25%2%

6%

58%

34%2%

On-Campus, Not in the Libraryn = 9,460

In the Libraryn = 9,733

Off-Campusn = 7,790

Overall Usen = 26,983

66%


Purpose of Use by LocationU.S. Main Campus Libraries 2003-05


8%6%2%

16%

26% 42% Sponsored Research

Coursework

Other Research

Other Activities

Teaching

Patient Care


Purpose of UseOCUL Scholars Portal Users

In a sample of 20,300 electronic resources uses at OCUL Libraries, there were four uses outside the library for each use in the library.


AnalysisAnalysis

Web deliverables: Cross-tabulations in HTML for all institutional

data Interactive crosstabs for all institutional

Print deliverables: Summary tables Final report


Web InterfaceWeb Interface


OCUL Scholars Portal Usage OCUL Scholars Portal Usage AffiliationAffiliation


OCUL Scholars Portal Usage OCUL Scholars Portal Usage User StatusUser Status


OCUL Scholars Portal UsageOCUL Scholars Portal UsageLocationLocation


OCUL Scholars Portal UsageOCUL Scholars Portal UsagePurpose of UsePurpose of Use


Cross Tabulations Affiliation by Purpose of Use

100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL

100.0%4.2%13.6%0.7%14.4%4.5%62.6%Social Sciences

100.0%2.4%31.8%0.4%11.1%9.7%44.6%Sciences

100.0%5.0%7.4%2.1%10.9%22.8%51.9%Other

100.0%5.7%32.0%8.6%18.4%5.5%29.7%Medical Health

100.0%9.4%2.6%0.9%12.8%6.8%67.5%Law

100.0%6.7%9.5%0.5%21.0%10.8%51.5%Humanities

100.0%9.4%5.6%1.3%20.6%6.9%56.3%Fine Arts

100.0%6.3%23.3%0.3%24.0%2.5%43.5%Environmental Studies

100.0%24.0%11.8%0.8%17.1%5.4%40.9%Education

100.0%16.0%10.8%0.9%30.0%7.6%34.8%Business

100.0%3.7%46.3%0.6%17.7%7.6%24.0%Applied Sciences

TotalTeachingSponsoredPatient

CareOther

ResearchOther

ActivitiesCourseworkAFFILIATION

PURPOSE OF USE


100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL

100.0%1.9%5.9%0.9%7.7%7.8%75.8%Undergraduate

100.0%12.7%51.6%2.1%20.6%9.5%3.5%Staff

100.0%2.5%26.8%8.7%20.8%35.2%6.0%Other

100.0%5.2%17.7%16.5%13.1%24.1%23.5%Library Staff

100.0%3.2%45.4%2.5%25.5%3.9%19.5%Graduate Professional

100.0%25.6%42.6%4.4%21.2%4.7%1.5%Faculty


CareOther

ResearchOther

ActivitiesCourseworkUSER STATUS

PURPOSE OF USE

Cross Tabulations User Status by Purpose of Use


Cross TabulationsCross TabulationsLocation by Purpose of UseLocation by Purpose of Use

100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL

100.0%5.7%42.2%0.9%17.9%4.0%29.2%On-campus

100.0%4.6%19.9%4.1%17.3%7.0%47.2%Off-campus

100.0%7.9%12.3%1.2%10.8%14.9%52.8%Library


CareOther

ResearchOther

ActivitiesCourseworkLOCATION

PURPOSE OF USE


4,388Other

925Course Reading

620Recommended Librarian

6,090Reference/Citation

2,436Recommended Colleague

10,219Important Journal

FrequencyReason for Use

(n=20293)

21.6%Other

4.6%Course Reading

3.1%Recommended Librarian

30.0%Reference/Citation

12.0%Recommended Colleague

50.4%Important Journal

PercentReason for Use

(n=20293)

OCUL Scholars Portal Usage OCUL Scholars Portal Usage Reason for UseReason for Use


ResourcesResources

Web interfacehttp://minesforlibraries.org

ARL New Measures and Assessment Initiatives: MINES for Libraries®

http://www.arl.org/stats/initiatives/mines/index.shtmlplace

Articles & Presentationshttp://www.arl.org/stats/initiatives/mines/minesresources.shtml

http://minesforlibraries.org/

http://www.arl.org/stats/initiatives/mines/index.shtmlplace

http://www.arl.org/stats/initiatives/mines/minesresources.shtml


Contact UsContact Us

ARL Staff: MaShana Davis, Technical Communications Liaison Martha Kyrillidou, Director, ARL Statistics and Service

Quality Programs Gary Roebuck, Technical Operations Manager

Project Management: Brinley Franklin, Vice Provost for University Libraries,

University of Connecticut Libraries Toni Olshen, Business Librarian, Peter F. Bronfman

Business Library, York University Terry Plum, Assistant Dean for Technology and Director,

Simmons Graduate School of Library and Information Sciences


Questions

Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for...

Documents

Transcript of Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for...