Scholars Portal :An Evaluation Overview

Mary Cavanagh, doctoral candidateFaculty of Information Studies

FIS 1325S

Adapted with permission of A. Darnell, OCUL Scholars Portal

Three sets of data

Usage data available through ScholarsPortal (SP) reports

Ontario Council of University Libraries(OCUL) Librarian & User Survey

Assocation of Research Libraries (ARL)– MINES data

Article Downloads

july 2002 - june 2005

toronto 3,567,865 32.69%

waterloo 1,053,530 9.65%

guelph 846,129 7.75%

ottawa 776,910 7.12%

western 712,242 6.53%

york 646,119 5.92%

trent 493,234 4.52%

brock 462,398 4.24%

carleton 440,817 4.04%

mcmaster 437,115 4.01%

queens 411,584 3.77%

windsor 352,208 3.23%

ryerson 323,293 2.96%

laurier 169,920 1.56%

laurentian 122,639 1.12%

lakehead 62,093 0.57%

nipissing 18,882 0.17%

uoit 15,633 0.14%

total 10,912,611 100.00%

Top Ten Downloads

Downloads Title(Publisher)

87877 Lancet, The(Elsevier Science)

85393 Social Science and Medicine(Elsevier Science)

81374 Phytochemistry(Elsevier Science)

79992

Journal of Personality and Social Psychology(American Psychological

Association)

56511 Animal Behaviour(Elsevier Science)

50223 Personality and Individual Differences(Elsevier Science)

49551 Sex Roles(Kluwer Academic/Plenum Publishers)

43775 Developmental Psychology(American Psychological Association)

36867

International Journal of Eating Disorders(Wiley Subscription

Services, Inc., A Wiley Company)

36699 Child Abuse and Neglect(Elsevier Science)

Un-met Demand

RefWorks Accounts

Citations Captured

RefWorks Users

Reason for Use

User Satisfaction

Scholars Portal Search Working GroupReport of Findings

November 2005Sandra Langlands, TorontoZita Murphy, RyersonPatti Ryan, YorkHarriet Rykse, WesternMartha Whitehead (chair), Queen’sDeborah Wills, Wilfrid LaurierPeter Zimmerman, Windsor

Working Group Mandate

Solicit feedback from librarians andusers on Illumina functionality

Conduct research on the “single search”approach to searching multipledatabases

Make recommendations to OCUL’sScholarly Information Resources TaskGroup regarding Illumina promotionand instruction

User Study Objectives

Learn about students’ and researchers’satisfaction with aggregated search results

Draw conclusions regarding proposed accessroutes in Scholars Portal Search

Methodology

Usability study principle - 5 usersAt our 7 institutions - range of sizes and

programs23 users in total - 10 undergrad, 1 staff

(novice), 5 graduate students, 7 facultyRange of disciplines

Methodology

Users were asked to comment on threedifferent sets of search results inScholars Portal Search, from:– A search of all databases– A search of one of subject area groupings

of databases, chosen from the drop-downmenu

– A search of a single database

Observations

Which results did users prefer?Inconsistent, but half of the faculty and

graduate students preferred the resultsfrom the search of all databases

Subject Area groupings are problematicSingle database search results preferred

by some novice users

Observations

Relevance, reaction to large result setsPoorer than algorithms that use full-text;

date sort more useful than relevance sortFew users looked at number of hits right

away, and high not always a deterrentJudged results by first few citations and

rarely went past first screenFew noticed descriptors

Observations

Interface and search functionalitySome observations similar to OCUL

staff:– Shouldn’t have to select databases– Quick search assumed ‘and’, not phrase– Tabs weren’t noticed– A “did you mean” search suggestion

would be good

Anywhere search most common

What Are Other StudiesSaying?

Faculty and grad students often findsignificant, previously unknown citations

Don’t make users choose or read beforesearching

Groupings eliminate “those long listsreally make me feel stupid”

But subject profile choices are difficult,and many choose ‘multidisciplinary’

SELECTED RESOURCES ONUSER TESTING

Augustine, S. and Greene, C. (2002). Discovering how students search a library web site: a usability casestudy. College & Research Libraries 63(4): 354-3. - Good example of a usability test.

Crowley, G.H. et al (2002). User perceptions of the library's web pages: a focus group study at TexasA&M University. Journal of Academic Librarianship 28(4): 205-210. - Describes the focus groupprocess.

Dickstein, R. and Mills, V. (2002). Usability testing at the University of Arizona Library: how to let theusers in on the design. Information Technology and Libraries 19(3): 144-51. Step-by-step descriptionof user-centred design.

Fichter, D. (2004). Heuristic and cognitive walk-through evaluations. Online 28(3): 53-56. Comparesand contrasts the two methodologies.

Head, A.J. (1999). Web redemption and the promise of usability Online 23(6): 21-32. clear, succinctdiscussion of different types of tests.

McGillis, L. and Toms, E.G. (2001). Usability of the academic library web site implications for design.College & Research Libraries 62(4): 355-368. Includes useful description of methodology.

McMullen, S. (2001). Usability testing in a library web site redesign project. Reference Services Review29(1): 7-22. Includes sample questions.

Mach, T. et al (2004). Designing for experts: how scholars approach an academic library web site.Information Technology and Libraries 23(1): 16-22. Describes how usability sessions were conducte

Spool, J.M. et al (2004). Designing for the Scent of Information. UIE Report. **Highly recommended forwebsite design

Usability Research Lab, NCSU Libraries User Interface Engineeringhttp://www.lib.ncsu.edu/dli/usability/library_usability.htmlhttp://www.uie.comUsability Resources, compiled by Darlene Fichter useit.com (Jakob Nielsen)

http://library.usask.ca/~fichter/usability/http://www.useit.com/Prepared by Deborah Wills, Wilfrid Laurier University Library, December 2004

Demonstrating Value and Creating Value:Evidence-Based Library Management

Martha KyrillidouDirector of the ARL Statistics and Measurement Program

Association of Research Libraries

Brinley FranklinVice Provost for University Libraries

University of Connecticut

Scholars Portal Forum, Ontario Council of University LibrariesFebruary 1 2006

Toronto

www.arl.org/stats/

Bangor University considers removing librarians posted byBlake on Thursday January 27, @07:30AM -753 hits

Ms Information writes "News from the University of Wales Bangorin the UK. senior management no longer feel that subject librarians /academic liaison librarians are needed in the modern academic library.They have made restructuring proposals which include removing allbar one of the subject librarians and a tier of the library management,including the Head of Bibliographic Services. The universitymanagement thinks that technology has 'deskilled' literaturesearching. As far as I know, this proposal is unprecedented in theUnited Kingdom.In essence, there will remain 4 professional librarians serving a'research-led' university of 8,000 plus FTEs and with 8 library sites.These will be the university librarian, cataloguing librarian,acquisitions librarian and Law librarian.Has anything like this happened anywhere that you know of? If so,what have been the effects?

Libraries Remain aLibraries Remain aCredible Resource in 21Credible Resource in 21stst

CenturyCentury

98% agree with statement98% agree with statement, , ““My My …… library librarycontains information from credible andcontains information from credible andknown sources.known sources.””

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

Changing BehaviorsChanging Behaviors

Recent Survey:Recent Survey:Only Only 15.7% agreed with the statement15.7% agreed with the statement ““TheTheInternet has not changed the way I use theInternet has not changed the way I use thelibrary.library.””

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

Total Circulation

400,000

450,000

500,000

550,000

600,000

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

Total Circulation

Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.

Reference TransactionsReference Transactions

90,000

100,000

110,000

120,000

130,000

140,000

150,000

160,000

170,000

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.

ARL Overall

MINES for LibrariesTM

Measuring the Impact of Networked Electronic InformationServices

MINES is a transaction-based research methodologyconsisting of a web-based survey form and a randommoments sampling plan.

MINES typically measures who is using electronic resources,where users are located at the time of use, and their purposeof use.

MINES was adopted by the Association of ResearchLibraries (ARL) as part of the “New Measures” toolkit inMay, 2003.

MINES is different from other electronic resource usagemeasures that quantify total usage (e.g., Project COUNTER,E-Metrics) or measure how well a library makes electronicresources accessible (LibQual+TM).

• How extensively do sponsored researchers useOCUL’s Scholars Portal? How much usage is for non-funded research,instruction/education, student research papers, and course work?

• Are researchers more likely to use the Scholars Portal from inside oroutside the library? What about other classifications of users?

• Are there differences in Scholars Portal based on the user’s location(e.g., in the library; on-campus, but not in the library; or off-campus)?

• Could MINES, combined with usage counts, provide an infrastructure tomake Scholars Portal usage studies routine, robust, and easily integratedinto OCUL’s administrative decision-making process for assessingnetworked electronic resources?

Questions Addressed ByMINES for Libraries™ for the

OCUL Scholars Portal

OCUL/MINES MethodologicalConsiderations

The sampling plan was determined at the outset. Surveys were

conducted once a month for two hours a month between May, 2004

and April, 2005

The selection of the monthly survey periods were weighted based on

usage counts by time of day and were chosen randomly.

Participation was mandatory, negating on-respondent bias, was based

on actual use in real-time, and was brief (to minimize user

inconvenience).

OCUL designed the local questions, mounted the survey, collected

data and sent it to ARL for tabulation in aggregate and by individual

institution.

If more than one search was conducted by a user, the survey form was

auto-populated with initial responses as the default.

OCUL/MINES MethodologicalConsiderations (continued)

Each participating library explained the survey and its confidentialityprovisions to their local constituency.

Research ethics officers and/or Ethics Review Boards, where necessary,reviewed and approved the survey instrument and methodology.

OCUL determined that individual institutions and their institution-specific data collected during the survey periods would not bedisclosed. Individual data was anonymous.

The mandatory nature of the survey required discussion on somecampuses and caused one OCUL member library to withdraw from thestudy.

Two institutions pre-tested the survey in January, 2004. Data collectionprogramming and configurations/links had to be revised in Februaryand March, 2004.

After completing the survey, users were connected to their desiredScholars Portal networked electronic resource.

OCUL Scholars Portal UsageAffiliation

Reason for Use

21.6%Other4388Other

4.6%Course Reading925Course Reading

3.1%Recommended Librarian620Recommended Librarian

30.0%Reference/Citation6090Reference/Citation

12.0%Recommended Colleague2436Recommended Colleague

50.4%Important Journal10219Important Journal

PercentReason for Use (n=20293)FrequencyReason for Use (n=20293)

How extensively do sponsored researchersuse OCUL’s Scholars Portal? How muchusage is for non-funded research,instruction/education, student researchpapers, and course work?

MINES for Libraries™shows that theScholars Portal resources are heavily usedby faculty and students in all OCUL. Themajority of the use is from the sciencesand the medical field and particularly inthose fields the majority of the use is forsponsored research purposes.

Are researchers more likely to use theScholars Portal from inside or outside thelibrary? What about other classificationsof users?

Most faculty, graduate professionals andundergraduates uses of the ScholarsPortal are from outside the librarybuilding. Undergraduates though do showmany uses of the Scholars Portal fromwithin the library as they are probablybecoming more exposed to these resourcesby having more physical contact with thelibrary.

Are there differences in Scholars Portalbased on the user’s location (e.g., in thelibrary; on-campus, but not in the library;or off-campus)?

Most of the faculty and graduateprofessionals use Scholars Portal eitherfrom on-campus locations outside thelibrary or from other off-campus locations.Most of the uses from these locationsoutside the library are for sponsoredresearch purposes.

Could MINES, combined with usagecounts, provide an infrastructure tomake Scholars Portal usage studiesroutine, robust, and easily integratedinto OCUL’s administrative decision-making process for assessingnetworked electronic resources?

Yes

Issues with web surveys:brief bibliography

Cook, Colleen; Heath, Fred; and Russell L. Thompson. 2000 (December). “A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys.” Educational andPsychological Measurement 60(6): 821-836.

Couper, Mick P.; Traugott, Michael W.; and Lamias, Mark J. 2001. "Web Survey Design andAdministration," Public Opinion Quarterly, 65 (2): 230-253.

Covey, Denise Troll. . 2002. Usage and Usability Assessment: Library Practices and Concerns.CLIR Publication 105. Washington DC: Council on Library and Information Resources.

– http://www.clir.org/pubs/reports/pub105/contents.html Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method. 2nd

Ed. New York: John Wiley & Sons. Gunn, Holly. 2002. “Web-based Surveys: Changing the Survey Process.” FirstMonday

7(12).– http://firstmonday.org/issues/issue7_12/gunn/index.html

LIBQUAL+ ™ Spring 2004 Survey. 2004. Cook, Colleen, and others.– http://www.libqual.org/documents/admin/ARL_Notebook2004.pdf

Schonlau, Matthias; Fricker Jr., Ronald D.; and Elliott, Marc N. 2002. Conducting ResearchSurveys via E-Mail and the Web. Santa Monica, CA: RAND.

Tenopir, Carol, with the assistance of Brenda Hitchcock and Ashley Pillow. 2003 (August).Use and Users of Electronic Library Resources: An Overview and Analysis of Recent ResearchStudies. Washington DC: Council on Library and Information Resources.

– http://www.clir.org/pubs/reports/pub120/contents.htmls Thomas, Susan J. 2004. Using Web and Paper Questionnaires for Data-Based Decision Making:

From Design to Interpretation of the Results. Thousand Oaks, Corwin Press. Thompson, Bruce.; Cook, Colleen.; Thompson, Russell L. 2002. Reliability and Structure of

LibQUAL+™ scores: Measuring Perceived Library Service Quality. portal: Libraries and theAcademy.3-12.