Project 1 Off Campus Services Benchmarking Project 2001/02 ... · Project 1: Off Campus Services As...

Project 1: Off Campus Services

As Others See Us: Benchmarking in Practice 1

Project 1

Off Campus ServicesBenchmarking Project 2001/02

Project Co-ordinator:

Sue White ([email protected])

Project Members

Sue White, Chris Beevers, (University of Huddersfield)David Potter, Carolyn Barton (University of Derby)Greg Allen, (Staffordshire University)Julie Brett, James Fisher (Leeds Metropolitan University)

Project Contributors

Dave BreezeNatalie GudgeonRose McDonaghLinda O’SheaJanet WeaverAnn WestJill Wisniewski

Our thanks are also due to all those library staff who participated in the project in variouscapacities.


2 As Others See Us: Benchmarking in Practice

ContentsIntroduction ................................................................................................................................... 3

Aims and objectives........................................................................................................................................... 3Methodology....................................................................................................................................................... 3Key areas ........................................................................................................................................................... 3

Mapping of Off Campus Services .................................................................................................. 4

Usage of Postal Loan Services...................................................................................................... 4

Methodology....................................................................................................................................................... 4Results ............................................................................................................................................................... 5Library staff observations .................................................................................................................................. 5Publicity and promotion ..................................................................................................................................... 6

Mystery Shopper Exercise to Evaluate Ease of Access to Web Based Informationfrom Off Campus........................................................................................................................... 7

Aims.................................................................................................................................................................... 7Methodology....................................................................................................................................................... 7Key issues.......................................................................................................................................................... 8

Analysis of the questions .............................................................................................................................. 8The tables...................................................................................................................................................... 8

Findings .................................................................................................................................... 9From the summary charts............................................................................................................................. 9Answers to individual questions ................................................................................................................. 10Mystery Shoppers’ Comments ................................................................................................................... 11

Limitations............................................................................................................................... 13Conclusions and outcomes...................................................................................................... 13Evaluation of the Mystery Shopper exercise............................................................................. 15

Usage of Electronic Resources.................................................................................................... 16

Methodology..................................................................................................................................................... 16Results ............................................................................................................................................................. 16Conclusion/recommendations:........................................................................................................................ 18

Summary of Key Findings............................................................................................................ 19

Postal loan services......................................................................................................................................... 19Off Campus access to web based resources................................................................................................. 19Usage of electronic resources......................................................................................................................... 19Conclusions...................................................................................................................................................... 20

Appendix 1.1 Benchmarking Off Campus Services Template ...................................................... 21

Appendix 1.2 Mystery Shopper Questionnaire ............................................................................ 22

Appendix 1.3 Mystery Shopper Biographies................................................................................ 25

Appendix 1.4 Results of Mystery Shopper exercise..................................................................... 27

Appendix 1.5 Historical development of the websites .................................................................. 34



Introduction

Aims and objectives

This project aimed to investigate andcompare provision of Off Campus servicesfor students between the four participatinguniversities.

Specifically the objectives were

1. To identify the various Off Campusservices provided at each institution

2. To distinguish between servicesavailable remotely to all library users,and those targeted at selected groups

3. To measure usage of agreed OffCampus services

4. To identify methods of measuring usageof agreed electronic resources

5. To evaluate the ease of access to agreedOff Campus services

6. To identify the costs of providingagreed Off Campus services, includingconsideration of charging mechanisms

7. To identify any problems experiencedby library staff with delivery of OffCampus services

8. To consider current publicity andpromotion of Off Campus services

Methodology

Given the diverse nature of this project itwas not feasible to produce a single modelof best practice. Our focus was rather togather management information and to shareexperiences of this developing area ofservice provision.

A range of data was collected and analysed:

• Environmental data from institutions onall services currently provided whichcould be deemed ‘Off Campus’

• Usage statistics of postal loan servicesSeptember – December 2001

• Costs (where available) of providingpostal loan services

• Usage statistics of electronic resources

In addition:

• A mystery shopper exercise wasundertaken to assess ease of access toweb-based information.

• Feedback from front-line library staffwas considered.

• Examples of publicity materials wereevaluated.

For the purposes of anonymity, the fourinstitutions here are referred to as Ashton,Burton, Carlton and Denton Universities.

Key areas

The report focuses on the four main areascovered by the project

• Mapping of Off Campus services

• Usage of postal loan services

• Mystery shopper exercise to assess easeof access to web-based information

• Usage statistics of electronic resources



Mapping of Off Campus Services

To establish the range of Off Campusservices provided by each institution, a proforma (Appendix 1.1, page 21) wascompleted. This served to identify servicesavailable according to user category.

• Access to a website and web opac wasavailable to all users at all fourinstitutions.

• Access to a CD-Rom network, digitisedcourse resources (e.g. electronic exampapers, HERON readings) and internetdatabases was provided to most users(franchise and some overseas studentsbeing the exception).

• Provision of an email account wasavailable to most users at three of thefour institutions; at the fourth only staffand special needs students were affordedthis service.

• Only one institution had a dedicatedphone line for Off Campus services.

• Two institutions offered an electronicenquiry desk service.

• All institutions were members of UKLibraries Plus, and most also operatedlocal co-operative schemes.

• All institutions provided web access tostudy skills materials to registered users .

• All institutions offered a postal loansservice to some categories of user.

Usage of Postal Loan Services

The environmental data collected in the firstphase of the project allowed us todistinguish between specific Off Campusservices which were offered to particularuser groups (e.g. postal loans) and thoseservices available to all users (e.g. onlinecatalogue).

A postal loans service was common to allfour institutions, though each servicediffered in its operation. Nonetheless we felta comparison would provide useful data andthe opportunities to learn from each other interms of good practice.

Methodology

Data on the usage of the postal loans servicewere collected for the Autumn Term 2001,from 17 September to 21 December. Inaddition, approximate costs of providing theservice were calculated, though in the eventthese proved problematic given that coststended to be subsumed into the generallibrary budget. The results are tabulated inTable 1a

Originally we had hoped to be able tocalculate the percentage take up of theservice per institution, based on number ofusers as a proportion of the total eligible.However due to the difficulty of estimatingthe numbers eligible we were obliged toabandon this objective.



Results

Table 1a Postal loan services. Usage statistics and service costs 17.09.01 – 21.12.01

University Description ofservice

Eligible touse theservice

Numberusing theservice

Number ofbook

requests

Number ofother

requests

Service costs(includingstaffing)

Enquiries

Ashton Postal loan servicecommenced Sept2001. Registration

fee £5.00(contributes

towards postal loancosts). Studentspay £1.50 per

photocopied article,£4 per postal bookloan, plus return

postage

All part-timestudents

including 150distancelearners

(approx. 6,500in all)

21 2 72 journals £3.56 perphotocopy

journal fromstock £8.02 perphotocopy viaInter Library

Loan

£6.41 per book

Notrecorded

separatelyto mainlibrary

enquiries

Burton Off Campusservice pilot

commenced Sept2001. Registration

(free) required.Return postagepaid by student

All studentsand staff whoare unable to

physicallyaccess the

library. Maincriterion is

distance fromcampus

86registered

5 actualusers

3 requestsfor a total of

7 books

2 requestsfor

photocopies(22 pages,

13 using ILLand 9 fromown stock)

Non staffingcosts of £60.80.Project staff timehas been costedat £20,000 per

annum

Notrecorded

separately

Carlton Pilot commencedin 2000/2001 but

developmentdelayed due to

staff illness. Returnpostage paid by

student

DistanceLearnerswithin theSchool of

Health andCommunity

Studies

63registered

(i.e. alleligible

students)

2 requestsfor a total of3 books (1)

47photo-copiesof journals(24 in stockand 23 ILL)

£20,500 annualstaff costs.

Postage perbook £2.80.

Photocopies 5pper page

26

Denton Commenced 1999.Off Campus

learning supportservices providesenquiry service

during office hours,via post, phone, fax

or email. Nocharge for posting

books out, butstudents pay return

postage. Up to 5journal article

photocopies free ofcharge; thereafter

£4 each

Students oncourses whereattendance oncampus is 3

times asemester or

less (41courses,

approx. 1,750students). All

registeredresearch

students onnon-taughtcourses notliving locally

44 32 requestsfor a total of113 books

5 requestsfor

photo-copies(20 articles,2 in stock

and 18 ILL)

£2.80 to post abook. Use

existing teamtherefore noadditional

staffing costs

53

(1) These figures not representative. 74 requests were made between May 2000 and December 2001

Denton was the only institution with a wellestablished unit. The other three were allstarting up or piloting new services, and thisaccounts for the relatively low take upduring the data collection period. On thepositive side, however, there was anecdotalevidence to suggest that those students whodid make use of the service found itextremely valuable, and tended to use it on aregular basis.

Ashton was the only institution to chargestudents for postal loans, and these charges –and particularly the requirement to pay anup-front registration fee – probably inhibitedtake up of the service.

Library staff observations

Library staff involved in delivering thepostal loans service were invited to commenton their experiences to date. These



experiences were specific to each institution(see Table 1b) but a number of genericissues emerged:

• Problems with identifying studentseligible to use the service. Good liaisonwith Registry and School Administratorsis extremely important.

• Effective publicity and promotion isrequired (see below).

• Development and co-ordination of theservice is time consuming. This factorneeds to be included in evaluation of thepilots.

Publicity and promotion

Effective publicity and promotion of postalloan services were felt to be critical inensuring all students were at least aware ofwhat was available. Although the group

recognised that good promotion does notnecessarily lead to good take-up, a numberof ideas for promoting the service werediscussed:

• Glossy leaflet, preferably producedprofessionally

• Web pages

• Inductions, ideally co-presented withacademic staff

• Email to tutors

• Mailshot to distance learners

• Direct contact with SchoolAdministrators (in order to identifyeligible courses)

• School newsletters

• Posters in Schools and Library

• Display in library foyer/communal areas

Table 1b Postal loans services : staff observations December 2001

University Comments/observations from the Off Campus staff about the service to date

Ashton • Staff enjoy providing a more personal service and entering into a dialogue with user

• Remembering to check the email account for incoming on-line requests can be a problem

• References supplied can be poor (in some cases because of incomplete references providedby lecturers)

• Many articles that should be in stock are not – leading to disappointment and embarrassment

• Administration of the service is cumbersome and bitty

• Emphasis on speed of reply (same day service) can place staff under pressure

• There has been one instance where copyright was breached in that it was noticed that threeusers on the same module had requested exactly the same three articles

• The on-line registration forms are sometimes awkward to decipher when they come throughto the email account

Burton • New initiative which will take time to establish

• Gradual increase in the number of students registering but few have gone beyond the initialcontact stage. Reasons may be that distance learning courses may provide all the materialsthe students need, the growth of valuable and free material on the web etc.

• Need to look at how we promote the service emphasising our flexibility and responsivenessto student’s needs

Carlton • Developing/co-ordinating the service – takes a lot of time

• Enrolment/registry issues – time consuming for staff to check enrolment with School/StudentOffice

Denton • Problems can arise when obtaining items from other sites: adds a delay and items can gomissing

• Offsite courses often have complications with the enrolment procedures. Offsite staff have tonegotiate with students who are anxious to use our services but aren’t yet property enrolled

• Decisions on which courses are eligible can be sensitive and political

• Students perception of what is a distance learner can be different to our criteria



Mystery Shopper Exercise to Evaluate Ease of Access to Web BasedInformation from Off Campus

The Mystery Shopper exercise waseffectively a self-contained project and iswritten up accordingly, in some detail.

Aims

The main purpose of the Mystery Shopperexercise was to evaluate the ease with whichOff Campus students could access andobtain information from theLibrary/Learning Centre web pages of thefour universities. The Shoppers would beable to highlight some of the difficulties OffCampus students experience. Theirsuggestions on how to rectify some of theseproblems could also give a useful insightinto how the participating institutions couldwork to improve their web pages.

Methodology

Questions (Appendix 1.2, page 22) wereconstructed and broken down into variousparts so that three variables could beassessed:

• Success in finding the requiredinformation

• Ease of access

• Time taken

Two volunteers from each institution wereselected. It was felt that one volunteershould have moderate IT skills and oneshould be more experienced to give arepresentative picture of Off Campusstudents. Appendix 1.3 (page 25) outlinesthe IT background of the Shoppers.

Passwords were distributed to allow theShoppers from each institution access to thefour library web pages.

Nine of the ten questions were the same forall four sites. The last question was differentfor each site to test ease of access to alocally mounted information resource.

It was decided that the exercise should beconducted in the morning if possible whenInternet speed would be less of a problem.

The two Shoppers from each institution wereasked to access the sites in a predeterminedorder. This was intended to eliminate anybias towards later sites accessed whichmight be caused by growing familiarity withthe questions

• Carlton – Carlton, Denton, Burton,Ashton

• Denton – Denton, Burton, Ashton,Carlton

• Burton – Burton, Ashton, Carlton,Denton

• Ashton – Ashton, Carlton, Denton,Burton

Each Mystery Shopper was given theexercise and asked to record the date and thestart/finish time for each site

The exercise was carried out between the18th and 25th February 2002.

Each university drew up a set of modelanswers to the questions.

Results from each university were tabulated(Appendix 1.4, page 27). These tables werethen further processed into three summarytables in an attempt to address precisely thethree variables listed:

• Success in finding the requiredinformation

• Ease of access

• Time taken

Staff from the four Universities interviewedtheir Mystery Shoppers after they had



completed the exercise to obtain feedback ontheir likes and dislikes of each site.

They also provided background informationabout the creation of the various websites asnon-library staff may have been involved inproduction. This was to determine whetherthe design had any bearing on the ease ofinformation retrieval or site navigation.(Appendix 1.5, page 34).

Finally, each University reported onmeasures it planned to take, as a result of theproject, to improve its website.

Key issues

Analysis of the questionsStudents working from within their homeUniversity learning centres may be able tobegin their information search from theLearning Centre home page. However, OffCampus students are likely to start theirsearch from the University home page.

Question 1 was designed to test how easythis initial navigation is.

Question 2 tests the OPAC: how easy is it toaccess and find a specific catalogued item.

Questions 3, 4 and 6 test how easy it is tofind specific information contained in theLearning Centre web pages.

The remaining questions (except forQuestion 10) address the accessibility ofInternet resources using the Learning Centrewebsite as a gateway. Question 9 tests therouting to an outside-recommended website.Questions 5, 7 and 8 require renewedauthentication through Athens or otherusername and password. Question 5 ArtIndex/Abstracts is either an Internet or alocally mounted CD-ROM database.

The final question allows each University totest the access it has provided to a locallymounted information resource.

The tablesA primary concern was whether ourShoppers succeeded in discovering therequisite information and whether there wasany difference in the success rate betweenthe four Learning Centre websites(Table A1.1, page 27).

Ease of access is explored in various ways inTables A1.2 and A1.3 (pages 28-29). Threequestions (Q1, Q5 and Q7) invite Shoppersto rate how easily they found the requiredanswers. The scale ranged from 0 (verydifficult) to 5 (very easy). Five questions(Q5, Q7, Q8, Q9 and Q10) ask Shoppers toindicate if they had any problems with thediscovery process. These were tabulated asYes/No answers in Table A1.2 and thenumber of “No’s” (indicating no problems)summed. Thus throughout Table A1.2higher numbers indicate better results.

A further measure of ease of access wasthought to be how long each Shopper took toanswer each question. This is tabulated inTable A1.3 (page 29). Where Shoppers wereunable to answer a question, they were nothowever clearly asked to indicate how longthey spent on their search. They were askedto give the total time spent on each exerciseand this figure is used to derive the amount(called “Extra time”) of time spent onunsolved questions. (A minus figure in the“Extra time” column indicates that the sumof the timings of individual questions isgreater than the reported time, thuspresumably indicating some inaccuratereporting by our Shoppers).

The raw data in Tables A1.4 to A1.7(pages 30-33) provide useful means ofcomparing the experience of individualShoppers accessing the four websites.



Findings

From the summary chartsFig 1c indicates clearly that Shoppers hadconsiderably greater success overall infinding the required answers on the Burtonwebsite. The other three attracted broadlysimilar results with Ashton marginally moresuccessful than Carlton and Denton.

Fig 1c Questions answered successfully

585859

67

0

10

20

30

40

50

60

70

Burton Ashton Carlton Denton

Num

ber

Fig 1d shows that questions where theanswers were to be found directly on theLearning Centre websites were generallyanswered successfully (though answers toQ6 did not always accord - particularly atBurton - with those expected from the modelanswers). However, whereusername/password authentication wasneeded, the success rate fell to 75% or below(falling to a little over 25% on Q8 requiringa search of Cinahl).

The ratings in Fig 1d compare success rateswith questions requiring basic navigationaround the Learning Centre websites (Q3,Q4, Q6) and those requiring access toresources via usernames and passwords (Q5,Q7, Q8). In general, higher success rateswere achieved where the answers were to befound directly from the website, e.g. Q3 onweekend opening hours achieved a 100%success rate, Q4 on locating a subjectcontact for law achieved 84%. In contrast,the average success rate for Q8, whichrequired a search of Cinahl, was only 31%(see Table A1.1, page 27)

Fig 1d Website versus password-required answers

0 10 20 30 40 50 60 70 80

Q3

Q4

Q6

All

Q5

Q7

Q8

All

Web

site

Pas

swor

d

Number of Successes

Ashton Burton Carlton DentonAshton



Fig 1e Time spent at each website

0

20

40

60

80

100

120

140M

inut

es

Ashton Burton Carlton DentonAshton

Fig 1e, which suggests that some Shoppersspent substantial amounts of time in trying,unsuccessfully, to answer some of thequestions, confirms the need forimprovement. We anticipated that 60minutes would be the absolute maximumtime needed to complete an exercise. Resultsindicate that no fewer than ten out of thethirty-two exercises took longer than this.Shoppers spent distinctly less time onaccessing the Ashton site than any of theother three, which may suggest that theAshton site is the best organised.

Burton gained the highest score overall interms of ease of access, followed by Denton,Ashton and Carlton (Table A1.2, page 28).The results indicate that there is still muchwork to do by all of us to improve ease ofaccess to our resources.

Answers to individual questions

Q1 From University home page how easy is it

to find library web pages?Most found the Library/Learning Centrepages easily. Two Shoppers quoted theon-line catalogue URL rather than theLibrary URL. Ashton was the easiest toaccess, Burton the most difficult. The timetaken ranged between 10 seconds and 6minutes. (One of the Carlton Shoppers couldnot access the Burton site initially because a

lower specification PC was being usedwhich could not load the University homepage. However, since the exercise wasprimarily about the Learning Centre site, theshopper was asked to try again using adifferent PC).

Q2 Check library catalogue for particular titleAll found the class number for Charles Handy“Understanding Organizations” except the twoAshton shoppers who could not access theirown institution’s catalogue on two separateoccasions, and a Denton shopper who couldnot find the information at Carlton. Time takenwas between ten seconds and four minutes.

Q3 What are weekend opening hours?All shoppers found the information. Noproblems at any sites. Time taken: 15 secondsto 18 minutes.

Q4 Who would you contact for subject

information on law?All shoppers found the information at Burtonand Ashton. At the Carlton site one shoppergave the name of the Head of Division for Lawand the other found that the server was busy.Most problems occurred at the Denton site.Shoppers may have taken the subject route tothe law pages which will have caused problemsas contact names are not currently given onsubject pages. The question specifically asked



for the name of the ‘librarian’. The subject staffat Denton have a different job title which didcause problems for one Shopper who haddifficulty in distinguishing if the name givenwas the librarian. Time taken: 30 seconds to 15minutes.

Q5 Search Art Index / Art Abstracts. How

easy is it to get to search screen?This was the first of the username/passwordquestions, which caused problems at allsites. The main problem was passwords notbeing accepted after accessing the entrypage. One Shopper stated that accessing ArtAbstracts at the Ashton site was a longprocedure, as a vast amount of informationhad to be read. Confusion may have arisen atDenton as Art Index is not available to OffCampus students and one shopper stated thatthere was no clear pathway. Burton attractedthe best results, though one shopper spent 30minutes on this question. Burton was ratedthe easiest, Ashton the most difficult.

Time taken: shoppers at Carlton reportedtaking the least time and at Ashton the mosttime on this question.

Q6 If you were having problems accessing

e-resources how would you get help?Most shoppers found an answer to thisquestion without spending a lot of time on it.However in some cases the answer givenwas different from the model answer (forinstance, some shoppers stated help could befound at the IT help desk rather than asstated in the web pages). Some of the results(particularly from Burton) have thereforebeen adjusted from the first draft of thereport. Time taken: 20 seconds to 12minutes.

Q7 Do we have issues of (two) journals

electronically in full text?This question was username/passwordcontrolled but did not present as manyproblems as Q5. Shoppers at two sites hadproblems with access.

Q8 Using Cinahl do a keyword search on the

Abortion Act 1967This question proved to be the most difficultfor the majority of shoppers with a very lowsuccess rate and problems related to access.Shoppers could not find Cinahl; they feltthat there was little explanation of Cinahland if they managed to access the databasethey could not find the article. Whilst someof the difficulties experienced with thisquestion may be due to the unfamiliarity ofthe shoppers with Cinahl, there were alsoproblems with navigation. This highlightsthe need for all Universities to provide amore “user friendly” route to this site. Allshoppers at the Ashton site and sevenshoppers at Denton site reported problems.

Q9 How best could you access the National

Institute for Social Work website?Shoppers managed to answer this questionwithout reporting too many problems. Therewas a wide variation in the number of stages(from 1 to 9) which indicates shoppersadopting many different search strategies.Several shoppers used “Google” as a searchengine and not the library web pages.Shoppers visiting the Carlton site managedto find the information using fewer stagesthan the other sites.

Q10This question was different for each site.Burton attracted the best result with Dentonshoppers reporting the least problems. TheCarlton result was poor probably because itsquestion required a non-obvious navigationplus employment of a username/password.

Mystery Shoppers’ Comments

CarltonIn general Shoppers felt that the links fromthe University home page to the LearningCentre home page were easy to find. Thelinks at the bottom of the page were the mostuseful but they were only seen when therequired information was not found on theleft-hand bar. Finding information about thelibrary was clear and straightforward.



The Learning Centre home page was thoughtto be bland and “clinical” and required morecolours. Problems arose with log on screensrequiring passwords with no explanation orvisible help particularly CD-ROMs. OneShopper commented that the i symbol in theA-Z list of e-journals did not have aninformation link. (This has now beenremoved.)

To access exam papers Off Campus studentswould be required to have some priorknowledge of their place in the Website.Shoppers felt that this was not obvious andsuggested that some navigation informationshould be given on an opening screen.

Shoppers suggested more buttons with dropdown menus linking to further conciseexplanations of various resources.

AshtonThis site was found to have an aestheticallypleasing Learning Centre home page.Overall Shoppers felt that this was awell-presented site with obvious, conciseand understandable links. The straight linkto the Learning Centre home page from theUniversity home page was appreciated andeasy to locate. Opening hours were clear andthere were very good links to subjectinformation. There were obvious links to thecatalogue but too many “clicks” to reach therelevant search page and no “back button”facility.

Shoppers commented on the difficulty ofaccessing Art Index/Abstracts. This requiredthe volunteers to read a vast amount ofinformation. One shopper tried theelectronic resource route but could not findthe database. Shoppers suggested cross-referencing, navigating users to a moreeffective route.

Accessing exam papers was long-winded asAcrobat reader had to be installed whichmay be difficult for Off Campus users.

Cinahl was not obvious and the link from the“Alphabetical list of information databasesand e-journals” was out of date.

Shoppers suggested Subject Librariansshould be named and links to other websitesshould be included on an A-Z list further upthe hierarchy.

DentonThe Learning Centre link from theUniversity home page was not obvious atfirst glance and Shoppers had to scroll downthe page to find the button. Some felt thatthis could be made more eye catching. TheLearning Centre page first impression wasthat it was bland, dull and plain. The linkswere confusing and not clear either for thenovice or the more experienced user.

Several Shoppers liked the student cardexamples explaining log on procedures forthe catalogue and resources online.However, the catalogue link was not obviousfrom the Learning Centre page as all linkshad to be read before reaching the requiredinformation.

The A-Z of services was thought to be “toowordy and bunched up” and some userscould be overwhelmed by the amount oftext. Some frustration was experienced overpassword failure. Help with passwords waslocated under “frequently asked questions.”It may be more useful to display this in themain menu.

There was some confusion over terminologybecause subject staff at Denton do not have‘Librarian’ in their job titles.

Skills for learning was thought to be easy tofind and well presented.

BurtonOverall the Library home page wasconsidered to be straightforward and easy tonavigate. The layout was clear and logicalwith adequate explanations of Athensauthentication. The OPAC wasuncomplicated with use of the “back arrow”



for navigation. Subject contacts andelectronic services had minimal links withgood descriptions and Off Campus resourceswere prominently displayed.

Shoppers did not like the black screen ofBurton’s University home page or the factthat the Library was not given priority fromthis page. It was felt that the library shouldbe given a higher profile. (One of theCarlton Shoppers could not access theBurton site initially because a lowerspecification PC was being used whichcould not load the University home page).

Limitations

Limitations of the study were as follows:

• Problems with down time. This wasexperienced by Ashton Shoppers tryingto access their own site on the days theycompleted the exercise. This can causefrustration for users but highlightscomputer problems rather thannavigation problems. All users couldencounter this problem at any timetherefore the exercise can be equated to“real life” experience.

• Problems with authentication – usernameand passwords not being accepted orlinks not working.

• Trying to select volunteers to completethe exercise was demanding. It wasnecessary to select Mystery Shopperswith various levels of IT skills to reflectOff Campus students (Appendix 1.3).Selection was limited to members ofstaff available at the time the exercisewas to be carried out. The experience ofthe shoppers was difficult to quantify asmost used the Internet as part of theirdaily routine. Selecting novice Internetusers may be a consideration for furtherinvestigation (Carlton initially tried touse a novice Shopper – a non-studentShelver – and provided an hour’straining in logging on and using InternetExplorer. However they were unable tocarry out the Exercise without continualrecourse to IT Help staff for assistance,

which – though interesting – invalidatesthe methodology).

• Problems with individual questions – forexample Q5 asked the Shopper to accessthe Art Abstract search screen. MostShoppers stated that they completed thetask but they were not asked to open anydocuments. The question may needrefining so that proof of reaching thedesired answer is clear. Most Shoppershad problems with Q8, using Cinahl. Thequestion assumes that the shoppers knowCinahl is a nursing database. Some triedkeyword searches, which slowed downthe searching process.

• Several Shoppers used “Google” ratherthan the Learning Centre home pageswhen searching for the answer to Q9although the question instructed theShopper to start from the LC home page.This could reflect Off Campus studentswho may be familiar with “searchengines” and keyword searches. Thismay be less obvious to novice Internetusers.

• Some limitations have been seen in thepressure Shoppers may have experiencedtrying to complete the task whilstcarrying out their other library duties.

Conclusions and outcomes

Overall the Mystery Shopper exercise hasproved to be useful to the four participatinginstitutions. Each university has addressedthe suggestions highlighted by the Shoppersin various ways.

In general it was found that there was a needfor Off Campus access to electronicresources to be streamlined for example:

• Providing a subject based approach

• Providing seamless access to e-resources

• Having general search facilities,incorporating a search engine

• Designing more straightforwardnavigation



• Reducing multiple passwords, aiming forsingle authentication of user providingunrestricted access to eligible resources.

Each University has responded with a list ofimprovements to its website planned as aresult of the Mystery Shopper exercise, andthese are given below:

CarltonCurrently we are in the process of reviewingthe website. The results of this exercise haveclearly identified navigation as being aparticular weakness of our site. At themoment we are restructuring thearchitecture of the site in order to reduce thenumber of “clicks” before reaching thedesired target. We are undertaking thechallenge of making the site more intuitiveand also looking to offer a subject basedapproach to finding information as analternative to a media/medium approach. Aswell as this structural development we aredeveloping other tools to add to the sitewhich should be of benefit to the end user.

AshtonOur new Web Developer will be completelyre-launching the Learning Centre Website intime for the beginning of the next session.

The Mystery Shopper exercise demonstrated(fairly conclusively) what was alreadyknown, i.e. that the procedures for remoteusers to access some of our subscribedelectronic resources were far toocumbersome. Thus, though not specificallyresulting from the findings of the MysteryShopper exercise but supported by them, it isintended to make access (both on and OffCampus) to electronic resources much morestraightforward.

There are four strands being investigatedwhich singly or in conjunction will achievethis:

• Use a search engine approach to directusers, via their subject pages, to theresource most suitable to their needs.The search will return not only to the

most relevant resource but will alsoprovide easier instructions about how toauthenticate.

• Using an Authentication tool toauthenticate registered users to accessthe web catalogue and thereafter provideunimpeded access to e-resources towhich they are eligible.

• Beta testing a third party product whichwould allow users to tailor their ownone-stop shop gateway to our electronicresources via one log-in.

• Use of an aggregator’s search enginewhich has the potential to provide asearch facility across all our electronicholdings as well as just one level ofauthentication.

Meanwhile the debate about having aseparate A-Z list of e-journals on the webpages continues. There are still significantnumbers of individual titles provided by theaggregating services, which are not listed onthe catalogue. However at the moment thecase is still argued that it would be better todedicate resources to achieve bettercatalogue coverage and to go with thesearch engine approach for tracing titles onthe web pages.

DentonVarious issues clearly need to be addressedin the light of the Mystery Shopper exercise.

Questions which required a passwordcreated the most difficulty. Two practicalsteps can be taken quickly.

• A button stating “Not available OffCampus” has been already added to thelinks for all resources which can only beaccessed from campus.

• Against each resource there is currentlya “Connect” and “Password” button.The order of the “Connect” and“Password” buttons will be switchedand the text on the buttons made moremeaningful. An “Info” button will becreated for each resource which includes



instructions on accessing that specificitem. There may, for instance, be abutton saying “Info” then “GetPassword” and finally a button saying“Connect to Resource”

In the long term Denton is endeavouring tomake access to all resources as seamless aspossible. Various projects are underwayincluding WebCT and Campus Pipelinewhich facilitate this.

Question 4, which asked Mystery Shoppersto identify the appropriate librarian for lawcaused problems on the Denton site. Thisinformation is currently only accessible viathe “Help and Contacts” link. Contactdetails for subject staff will be added to theSubject pages as this is where most MysteryShoppers expected to find it. Some furtherthought should be given to the wording andposition of the “Help and Contacts” buttonas users do not quickly identify this facility.

Generally a search facility which could betargeted at the Learning Centre pages wouldhave helped with some of the issues such asnot being able to locate CINAHL. The Webdevelopment team is working with theUniversity to make a search facilityavailable.

The position of the Learning Centre buttonon the Denton home page was criticised byone Mystery Shopper. It can be difficult tospot on some screen configurations. Thispoint should be raised again with theUniversity.

BurtonAlthough our website came out morefavourably than the others there is stillplenty of room for improvement. The mainproblem we identified was the apparentoverload of information for the Off Campususer. A lot of the information containedwithin the website is not relevant to the OffCampus student therefore we created an OffCampus section aimed specifically at thattype of user. Whilst this section assists theuser, they still have to navigate themselves

to various other parts of the website; this iswhere the problems arose. Therefore wehave started to create a searchable databasefor Off Campus users.

In conjunction with a private company, weare in the process of developing a databasethat will include everything that an OffCampus user would need to support theirstudies. There will be direct access to onlinedatabases, electronic journals, usefulwebsites, our own library catalogue etc. Allof this will be available from one searchablewebsite making it much easier for them toaccess the various services we offer. As youcan imagine this site will be very timeconsuming to construct but we hope tolaunch a fully operational version to all OffCampus students by September 2002.

Evaluation of the Mystery Shopperexercise

Before repeating this exercise certain areasneed clarifying, for example:

• Questions need refining to ensure thatthe responses illuminate the specificaims of the project.

• It is suggested that more shoppers shouldbe selected to give a clearer picture ofthe accessibility of our websites to thevarious types of Off Campus users.

• Fewer questions should be included onthe questionnaire so that specificproblem areas can be analysed in greaterdetail.

• Model answers should be available at anearly stage to ensure accuracy in analysisof the responses.

Even within the limitations this year, we dofeel that the basic methodology seems validand that the exercise has provided valuableinsights into how easily our users are able toaccess and obtain information from ourwebsites. It could usefully be repeated nextyear after each university has had theopportunity to remedy shortcomingsidentified this time.



Usage of Electronic Resources

One of the key Off Campus services wedeliver is the provision of electronicinformation. As we all know, the precisemeasurement of usage of e-resources is acomplex and developing area, and subject todebate at national and international level.

The aims of the Off Campus project wererelatively modest

• To share experience of collecting andrecording usage of data

• To ascertain whether comparisons acrossthe institutions was a) possible andb) meaningful

The sharing of experience was undertakeninformally, and in particular drew on thework being carried out at Ashton University.At Ashton, usage statistics from all availablesources are extracted, manipulated into anaccessible format (e.g. Microsoft Access orExcel), and presented in a digestible formfor library staff. They are then stored on anintranet and are regularly updated.

The comparison of usage statistics acrossinstitutions was undertaken more formally,and is recorded below:

Methodology

Four sample electronic resources wereselected for comparison. The criteria forselecting the resources chosen were:

• They had to be common to all fouruniversities

• Usage statistics had to be available

Usage statistics were extracted at eachUniversity library and tabulated forcomparison. Number of Logins/Sessions andnumber of Searches were the particularfigures compared. These are commonlysupplied measures of usage and are bothdistinct terms comparable across differentservices. Results are given in Table 1f(page 17).

ProblemsIt must be stated from the outset that thefigures quoted do not represent solely OffCampus usage of any of the resourcessampled. The figures at best represent acombination of accesses from a specificURL domain (on campus) and accesses fromsite identifiable Athens authentications (OffCampus, or themselves on and Off Campus).As the project as a whole is looking at OffCampus provision, it is recognised that thisrepresents a major flaw in these particularfindings. However, these were the onlyavailable figures that were known of at thetime. The results (such as they are) arepresented here as a basis for discussionabout how the exercise could be repeatedmore effectively when available technologyenables on and Off Campus usage to bemonitored more easily.

Results

InferencesGALE – In fact, of the individual databasessubscribed to under GALE only two werecommon across two of the sites. Both Burtonand Carlton subscribe to the InfotracComputer Database within GALE.

When comparing this particular databasethere were fourteen times as manylogins/Sessions at Burton than at Carlton,with over fifteen times as many searches. Itcan be seen from the table that users atBurton have had access to this database forapproximately seven years in comparison tothe situation at Carlton where it is arelatively new resource first taken about ayear ago.

Even with a relatively dynamic student usergroup, cultural embedding over time couldbe a possible explanation of increased usage,and in common with paper subscriptions alonger back-run will necessarily invite moresearches.



Table 1f Comparison of usage figures of selected e-resources during November 2001

Overarching service: GALE

SiteSpecificdatabase/service

Date firstsubscribed

Logins/sessions

SearchesSearches:

logins

General AcademicASAP

1988 298 1138 3.82Carlton

Infotrac ComputerDatabase

2001 50 127 2.54

European BusinessASAP

30-11-00 170 454 2.67

F & S Index Plus Text 1-3-00 97 193 1.99Ashton

UK NationalNewspapers

1-4-01 334 4371 13.09

DentonInfotrac GeneralPeriodicals

1-8-99 315 988 3.14

BurtonInfotrac ComputerDatabase

c.1995 698 1941 2.78

Overarching service: EBSCO

SiteSpecificdatabase/service

Date firstsubscribed

Logins/sessions Searches

Searches:logins

Carlton Business Source EliteNovember

20002187 5328 2.43

AshtonBusiness SourcePremier

1-2-01 655 1395 2.13

Business SourcePremier

September1999

3375Denton

Academic SearchElite

September1999

10022

Burton Business Source Elite c.1996 1867 3739 2.00

Overarching service: Web of Science

Site Date first subscribed Logins/sessions SearchesSearches:

logins

Carlton 2000 848 2340 2.75

Ashton 1-3-01 780 1807 2.32

Denton c. September 2000 250 381 1.52

Burton c.1998 1956 5859 3.00

Overarching service: ZETOC

Site Date first subscribed Logins/sessions SearchesSearches:

logins

Carlton April 2000 279 560 2.01

Ashton September 2000 44 123 2.80

Denton 8-5-01 149 357 2.40

Burton 2000 172 (on) 69 (off) 329 (on campus) 1.91



EBSCO – Two sites take Business SourceElite and two sites take Business SourcePremier via EBSCO. In addition Denton alsotakes Academic Search Elite. This latterpoint proved a slight complication asEBSCO only provides figures for thenumber of Sessions/Logins for the service asa whole, and then breaks down the numberof searches between the separate databases.

Thus, comparing numbers of searches onBusiness Source Premier between Dentonand Ashton, Denton the longer subscriber atapproximately sixteen months shows 2.4times the number of searches. Apart fromthe possible reasons suggested above forGALE, another factor here might be thesecond database subscribed to via EBSCO atDenton. Even if the two databases taken atDenton are largely accessed independently,if the user interfaces and search mechanismsare the same, user familiarity with one mightpredispose them to use the other.

When the figures for Business Source Eliteare considered, in contrast to the situationwith GALE and Business Source Premier,the more recent subscriber (Carlton inNovember 2000) shows 17% more sessionsand 42% more searches than Burton whohave taken the database for significantlylonger (c.1996). Other factors might be atplay here. Carlton for example might bepromoting the service in a more effective orhigh profile way, or indeed irrespective ofoverall student numbers might have a largerrelevant user group in terms of a specificsubject area or school/department.

Web of Science – Burton is the longestsubscriber, having taken the service sinceapproximately 1998, and displays by far thehighest number of Sessions/Logins andnumber of searches. The number of searchesper login would also seem to be higher. Thereasons stated for GALE might also be atplay here i.e. cultural embedding andfamiliarity, together with a longer searchable‘holdings’. Denton, by far the largest in

terms of student numbers, has the lowestnumber of Sessions/Logins as well as thesmallest Sessions/Logins:Searches ratio.This is an interesting result, particularly for adatabase of such broad subject coverage.The ‘relative importance’ of this particularservice at Denton, in terms of promotion anduser education, might be worthinvestigating. Perhaps an alternative,comparable service is preferred by librarystaff and given a higher profile when offeredto users.

ZETOC – Is a free service and the ‘Date FirstSubscribed’ indicates the date each site firststarted promoting the service to its users.Carlton shows a significantly higher numberof Sessions/Logins and number of searches.The figures for Denton are againcomparatively low considering the size of itsstudent population. Although usage is lowestat Ashton, it is interesting to note that theratio of Sessions/Logins: Number ofSearches is the highest. A possibleexplanation for this is that a particular groupof users is being given extra training andtheir increased confidence is encouragingthem to spend longer searching the database.

Conclusion/recommendations:

The whole area of e-resources is a verycomplex and dynamic one. The number ofe-titles taken by University Librariescontinues to increase, not least to supportflexible and possibly remote teachingmethods. As sites begin to experience areduction in the number of physical visits totheir libraries the provision of reliablefigures for ‘virtual access’ to their services isbecoming increasingly important.

As particular e-titles are made available viadifferent routes i.e. through different‘aggregating services’ and ‘deals’ theaspiration to compare usage across differentuniversities is a valid one. By exchangingexperiences each site can hopefully becomebetter informed to tailor its portfolio of



e-titles and achieve optimum accessibilityand cost effectiveness.

This feasibility study has been very much alearning exercise. When repeated, theselection of titles being compared shouldtake into account the followingconsiderations:

• Is the resource discrete in itself or acompilation/aggregation?

• If the resource provides more than oneservice are the same services taken byeach site?

• Can the service be accessed by morethan one route?

• Is the service accessed by the same routeacross each site?

• Can the statistics provided differentiatebetween on and Off Campus usage?

• Is the service promoted evenly across theuser population or targeted at particulargroups?

• Usage of a particular resource cannotnecessarily be considered in isolation.Availability of alternative andcomparable services (in terms of subjectcoverage) needs to be looked at as well.

Finally, with ATHENS providing usagestatistics themselves and with theintroduction of once only authenticationgateways to electronic resources such asSwetswise and OneLog it should soon bemuch more straight forward to differentiatebetween on and Off Campus usage.

Summary of Key Findings

Postal loan services

Take up of the postal loan service wasrelatively low in all four institutions. Thismay be explained in part by the newness ofthe service (in three out of four institutions)

and in part by the costs to users (especiallyat Ashton). Anecdotal evidence indicatedthat those students who did make use of thepostal loan service valued it and tended touse it on a regular basis. The project grouprecognised that effective promotion of theservice was critical, but also that a service ofthis nature is only ever likely to be taken upby a minority of users. The group felt it wasimportant that libraries were seen to beresponding to student need in a flexible andproactive way.

Off Campus access to web basedresources

The Mystery Shopper exercise producedsome very interesting results, and confirmedwhat we already suspected in terms of easeof access to resources : username/passwordauthentication presents huge obstacles to offcampus access. Where authentication wasrequired in the exercise, the success rate fellto 75% or below, and as low as 25% on thequestion requiring use of Cinahl.

Navigation around the websites themselveswas slightly less problematic, but in ten ofthe thirty-two exercises, the time taken tocomplete the exercise was in excess of the60 minutes allowed for. Such excessivelengths of time indicate the need for all fourinstitutions to improve ease of access to theirelectronic resources.

As might be expected, the recommendationsfrom the exercise centred on the need toprovide seamless access to electronicresources, single sign-on, a single searchfacility, and more straightforwardnavigation.

Usage of electronic resources

In terms of tangible outcomes this was theleast satisfactory part of the Off Campusproject. It proved surprisingly difficult toidentify four services which were commonto all institutions, and which were accessedvia the same route. It was impossible to



consistently differentiate between on campusand off campus usage, and it was alsodifficult to identify the potential size of theuser population – thereby renderingcomparison between the institutions almostmeaningless. This notwithstanding, theprocess of attempting to compare usage wasan interesting one, and there could be meritin the future in refining the methodology andrepeating the exercise.

Conclusions

The Off Campus benchmarking project wasbroad ranging, and rather than attempt toproduce a single model of best practice itaimed to investigate a number of disparateareas. Much of the work was exploratory,but nonetheless produced some extremelyinteresting results.

Sharing experience between the fourinstitutions was undoubtedly one of the mainbenefits of the project. Offering a postal

loans service was a new venture for most ofthe institutions, and being able to liaise withcolleagues in similar posts in the projectinstitutions was invaluable. Similarly, withregards to the measurement of electronicusage statistics, the group was able to learnfrom the work undertaken by one of theinstitutions. Although the exercise tocompare usage across the four institutionswas inconclusive, the process itself was alearning experience.

The Mystery Shopper exercise was afascinating project in itself, and representedthe first time any of the four partnerinstitutions had systematically sought to testthe accessibility of their web pages. Thisapproach will be refined and extended nextyear, to measure the improvements to thecurrent web pages.

Appendix 1.1: Benchmarking Off Campus Services Template


Appendix 1.1 Benchmarking Off Campus Services Template

Remote Services/User categories

Also applies to W

eb s

ite

Web

OP

AC

CD

RO

M n

etw

ork

Dig

itis

ed c

ou

rse

reso

urc

es

E-b

oo

ks

Inte

rnet

dat

abas

es

Em

ail a

cco

un

t

Ded

icat

ed p

ho

ne

line

Ele

ctro

nic

En

qu

iry

Des

k

Ele

ctro

nic

Su

bje

ct

sear

ches

UK

Lib

s P

lus

Oth

er c

o-

op

erat

ive

Ag

reem

ents

Po

stal

bo

ok

loan

Po

stal

ph

oto

cop

y d

eliv

ery

Po

stal

ILL

del

iver

y

Campus-basedFT students

PT students

Distance learnersPlacement studentsOut-centre students

Distance learners (UK&Ireland) Distance learners (Overseas) Special needsstudents

Other students

Franchise Overseas Accredited institutionPartner institution

Staff (ft, pt)Staff (contract)

Staff (associated)

Staff (other)

Accredited lecturersFranchise staffPartner institution

Other users

Appendix 1.2: Mystery Shopper Questionnaire


Appendix 1.2 Mystery Shopper Questionnaire

ASHTON UNIVERSITY * BURTON UNIVERSITY * CARLTON UNIVERSITY * DENTONUNIVERSITY

Off Campus benchmarking project : Mystery shopper exercise

The four Universities above are undertaking a project on Off Campus library services tostudents. Part of the project involves an assessment of the ease with which Off Campusstudents can access information through the web pages of their institution’s library/learningcentre – hence this ‘mystery shopper’ exercise. Thank you for agreeing to take part.

Before starting the exercise please read through all the questions.

Now record the date and start time here : Date______________Start time __________

1 a) From the University home page, how easy is it to find the Library/Learning Centre pages?

Please circle on the scale below:

0 1 2 3 4 5

Very Very easydifficult

b) Please write down the URL of the page you ended up at:

c) Approximately how long did it take to get there (in minutes or seconds)?

2. Check the library catalogue to see if the Library/Learning Centre has any copies of:

Charles Handy Understanding organizations

a) Please write down the class number

b) Approximately how long did it take you to find the information?

3. a) What are the weekend opening hours of the Learning Centre during term time?


4. a) Who would you contact for subject information on Law? (give the name of the librarian oran email address)




5. a) You want to search either Art Index or Art Abstracts (this may be a CD-Rom database).How easy is it to get to the search screen?


0 1 2 3 4 5


b) Please note any problems you had in finding the information

c) Approximately how long did it take you to find the information?

6. a) If you were having problems accessing electronic resources (e.g. with logging in), howwould you get help?


7. a) Do we have any issues of the following journals electronically in full text?

British Journal of Social Work Yes / NoBritish Journal of Sociology Yes / No

b) Choose one of these titles and indicate how easy it is to access the full text


0 1 2 3 4 5


c) Please note any problems you had in finding the information

d) Approximately how long did it take you to find the information?

8. a) Using Cinahl 1999 do a keyword search on the Abortion Act 1967. What is the title of thejournal in which you will find an article on legal issues?



9. a) How best could you access the National Institute for Social Work website, starting fromthe Library/Learning Centre home page? Please describe briefly the route you took:





10. (Carlton only)

a) Find the full text of an exam paper for Psychology stage 2, January 1999, entitled“Psychology of Mental Illness” and write down its course code.



10. a) (Ashton only) Find the full text of an exam paper for the module ‘Sociology of the Media’.



10. a) (Denton only) Find the resource called Skills for Learning and login. Under which maintopic is information on writing a CV found?

b) Please note any problems you had in finding the information.


10. a) (Burton only)

Bridget Prentice is the Member of Parliament for which constituency?

b) Please note any problems you had in finding the information.


Now record the time you finished this exercise ___________________

Many thanks for your help.

Appendix 1.3: Mystery Shopper Biographies


Appendix 1.3 Mystery Shopper Biographies

Ashton University

Ashton 1 (less experienced)

Has worked as a part time shelver and lending counter assistant for several years. She would onlyuse the Internet during the normal course of her work to look up book prices on Dawsons, and tosearch the Intranet for staff information. ‘Ashton 1’ has the Internet at home which she is used tousing to find flight information, ordering books via Amazon and for theatre and hotel bookings.

Ashton 2 (moderately experienced)

Has been a part time library assistant in the Inter-Library Loans Department for about 12 months.She has the Internet at home but only uses it for e-mail and doing the occasional search forholidays etc. At work ‘Ashton 2’ is used to searching for e-journals via our own webpac, andaccessing our bibliographic databases via a link to our IDN on the staff pc’s. With the benefit ofhindsight this almost acted against her in that it came as quite a surprise how complicated theextra authentication processes were for a student accessing resources from outside.

Burton University

Burton 1 (less experienced)

Recently joined us as a General Assistant and had never used the internet prior to that. Uses theweb still very infrequently, approx. 10 minutes per day. She does come from a bankingbackground, so is accustomed to IT applications, but only of the in-house variety as employed bythe bank.

Burton 2 (more experienced)

Due to conclude her Postgraduate Library Diploma this Summer. A psychology graduate, she hasworked as a Learning and Information Assistant for 4 years, and regularly covers

the information desk by herself. She is very IT literate and pretty experienced in using the webfor information retrieval

Carlton University

Carlton 1 (less experienced)

Part-time Learning Centre assistant for just over a year. Familiar with library managementsystem and with word processing and email. Only a very limited familiarity with the Internet, anddoes not have Internet access at home.

Carlton 2 (more experienced)

Member of reception staff and shelver. Politics graduate of De Montfort, and also an MA.Currently studying adult education teaching certificate at Nottingham. Considerable experienceof the Internet, both for her courses and at home.

Appendix 1.3: Mystery Shopper Biographies


Denton University

Denton 1 (less experienced)

Peak time Learning Centre assistant. Uses web catalogue at work. No other use of electronicresources. Uses internet at home to access information to help with children’s homework andfamily business.

Denton 2 (more experienced)

Graduate Trainee. First degree in history. Used web catalogue for locating books and journals.No other use of electronic resources. Uses Internet frequently for personal use including searchengines.

Uses Internet occasionally at work to check bibliographic details.

Appendix 1.4: Results of Mystery Shopper Exercise


Appendix 1.4 Results of Mystery Shopper exercise

Table A1.1 Success in finding required information

TargetShopper

Q1Found

Q2Found

Q3Found

Q4Found

Q5Success

Q6Found

Q7Found

Q8Found

Q9Success

Q10Found

Totalfound

CarltonC1 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 10A1 ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ 7

D1 ✗ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✗ 4B1 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ 6C2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ 9

A2 ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 4D2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ 9

B2 ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ 9

Total 7 7 8 6 6 5 6 3 5 5 58

AshtonC1 ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ 8A1 ✓ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✓ ✗ 5D1 ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✓ 8

B1 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ 8C2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ 8

A2 ✓ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✗ 5D2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ 9

B2 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ 8

Total 8 6 8 8 5 5 5 2 6 6 59

DentonC1 ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ 8

A1 ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✓ ✓ 7D1 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ 8

B1 ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ 5C2 ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✓ 7

A2 ✗ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ 5D2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ 9

B2 ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ 9

Total 6 8 8 5 4 5 8 2 5 7 58

BurtonC1 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 10

A1 ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ 8D1 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ 8

B1 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ 8C2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ 8

A2 ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✓ ✓ 7D2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ 9

B2 ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ 9

Total 8 8 8 8 7 3 7 3 7 8 67

Average 7.25 7.25 8 6.75 5.5 4.5 6.5 2.5 5.75 6.5 60.5(out of 8) (out of 80)

Total 29 29 32 27 22 18 26 10 23 26 242(out of 32) (out of 320)



Table A1.2 Ease of access

Q1 Q5 Q7 Q8 Q9 Q10Target Shopper

Rating* Rating* Problems Rating* Problems Problems Stages Problems ProblemsTotal

CarltonC1 4 3 ✓ 5 ✗ ✗ 4 ✗ ✗

A1 5 ✓ ✓ ✓ 2 ✗ ✓

D1 3 0 ✓ ✓ ✓ ✓ ✓

B1 5 2 ✓ 3 ✗ ✓ ✓ ✓

C2 5 4 ✗ 4 ✗ ✓ 2 ✗ ✓

A2 3 0 ✓ 0 ✓ ✓ ✓ ✓

D2 2 4 ✗ 4 ✗ ✓ 2 ✓ ✓

B2 4 3 ✗ ✗ ✗ 4 ✓ ✓

Average 3.9 2.0 2.0 2.6Count (n) 3 5 2 3 1 14

AshtonC1 5 4 ✗ 2 ✓ ✓ 4 ✗ ✗

A1 5 ✓ ✓ ✓ 2 ✗ ✓

D1 5 0 ✓ 0 ✓ ✓ 1 ✗ ✗

B1 5 3 ✗ 4 ✗ ✓ 3 ✗ ✗

C2 5 1 ✓ 5 ✗ ✓ ✓ ✗

A2 5 0 ✓ ✓ ✓ 4 ✓ ✓

D2 5 3 ✗ 0 ✓ ✓ 3 ✗ ✗

B2 4 2 ✓ 4 ✗ ✓ 5 ✗ ✗

Average 4.9 1.6 1.9 2.8Count (n) 3 3 0 6 6 18

DentonC1 4 3 ✓ 2 ✗ ✓ 8 ✗ ✗

A1 5 2 ✓ 4 ✓ ✗ 2 ✗ ✗

D1 5 3 ✗ ✓ ✓ ✓ ✗

B1 5 ✓ 2 ✓ ✓ ✓ ✗

C2 5 2 ✗ 3 ✗ ✓ ✓ ✗

A2 3 2 ✓ 0 ✓ ✓ ✗ ✓

D2 5 3 ✗ 2 ✓ ✓ 3 ✗ ✗

B2 3 3 ✓ 4 ✗ ✓ 5 ✓ ✗

Average 4.4 2.3 2.1 2.9Count (n) 3 3 1 4 7 18

BurtonC1 4 4 ✗ 4 ✗ ✗ 9 ✗ ✗

A1 3 4 ✓ 0 ✓ ✓ 2 ✓ ✓

D1 5 5 ✗ 4 ✓ ✓ 2 ✗ ✗

B1 5 3 ✗ 2 ✓ ✓ 2 ✗ ✗

C2 4 1 ✓ 5 ✗ ✓ ✓ ✗

A2 2 ✓ 0 ✓ ✓ 2 ✓ ✗

D2 3 4 ✗ 4 ✗ ✓ 3 ✗ ✗

B2 4 4 ✗ 4 ✗ ✗ 5 ✗ ✓

Average 3.8 3.1 2.9 3.3Count (n) 5 4 2 5 6 22

Totals

Average (out of 5) 4.2 2.3 2.22.9

(out of 5)

Count (n) (out of 32) 14 15 5 18 2072

(out of 160)

Average Count (n)(out of 8)

3.5 3.75 1.25 4.5 518

(out of 40)

* Rating scale from 0 = very difficult to 5 = very easy



Table A1.3 Time taken (in minutes)

Target ExtraShopper Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 time Total

Carlton

C1 0.3 0.75 0.7 1 1.5 2 2 4 2 2.5 41.25 58

A1 0.5 1 5 5 5 0.5 20 33 70

D1 2 1 3 3 5 4 7 5 4 6 3 43

B1 0.5 0.5 1 0.3 10 10 5 5 10 2.7 45

C2 1 2 1 1 0.5 1 2 2 6 38.5 55

A2 2 4 18 10 20 10 15 10 31 120

D2 1 0.3 1 0.5 0.5 1 1 1 5 5 3.7 20

B2 0.2 0.1 0.3 0.25 1 1 0.5 1.5 26.15 31

Total time 7.5 9.65 30 20.8 37.75 29 33 25.5 18.5 51 179.3 442

Ashton

C1 1 2 1 2 2 1 12 6 0.75 6 11.25 45

A1 0.1 10 1 5 10 1 10 22.9 60

D1 0.5 3 2 3 6 2 6 7 5 2 7.5 44

B1 0.3 2 0.5 0.3 10 1 2 10 0.5 7.5 -4.1 30

C2 2 5 5 1 10 10 2 15 -25 25

A2 1 5 3 5 20 5 5 12 15 5 -1 75

D2 0.2 0.5 0.25 0.3 0.5 0.5 2 1 0.3 1 13.45 20

B2 0.2 0.2 0.25 0.2 15 10 10 1 18.15 55

Total time 5.3 27.7 13 16.8 63.5 29.5 39 46 22.55 47.5 43.15 354

Denton

C1 0.75 1 2.5 3 1.7 0.75 5 3 2 1 46.3 67

A1 1 5 5 10 10 10 20 5 1 10 23 100

D1 1.25 0.8 0.5 4 2 5.3 10.5 7 1.5 31.15 64

B1 0.5 1 1 5 10 5 10 5 5 2.5 45

C2 1 10 15 8 3 10 8 -6 49

A2 6 2 2 15 10 15 10 10 70

D2 0.5 0.7 0.5 0.3 2 0.3 5 7 1 0.5 17.2 35

B2 0.3 1.5 1 0.75 0.5 1 0.5 2 2 0.75 25.7 36

Total time 11.3 22 12.5 53.05 44.2 25.35 76 32 13 26.75 149.85 466

Burton

C1 0.5 2 1 1 2.5 0.3 6 4 2 22.7 42

A1 5 4 5 35 15 3 20 10 10 5 8 120

D1 2 1 1 2 3 8 10 5 6 4 13 55

B1 0.3 3 2 5 5 5 10 2 2 30.7 65

C2 2 4 1 2 8 3 12 5 15 8 60

A2 10 2 1 8 30 12 12 10 10 20 -12 103

D2 0.3 0.5 0.25 0.5 0.3 2 1 2 0.5 2 5.65 15

B2 0.2 0.2 0.2 0.25 0.1 0.25 1 0.4 0.4 1.5 18.5 23

Total time 20.3 16.7 11.45 53.75 63.9 33.55 72 36.4 45.9 34.5 94.55 483



Table A1.4 Results from Carlton

Shopper 1 Shopper 1 Shopper 1 Shopper 1 Shopper 2 Shopper 2 Shopper 2 Shopper 2Target SiteCarlton Denton Burton Ashton Carlton Denton Burton Ashton

Q1

a Rating 4 4 4 5 5 5 4 5b Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

c Time taken 20 secs 45 secs 30 secs 1min 1 min 1 min 2 mins 2 mins

Q2a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 45 secs 1 min 2 mins 2 mins 2mins 10 mins 4 mins 5 mins

Q3

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 40 secs 2.5 mins 1 min 1 min 1 min 1 min 5 mins

Q4

a Found ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓

b Time taken 1 min 3 mins 1 min 2 min 1 min 15 mins 2 mins 1 min

Q5a Rating 3 3 4 4 4 2 3 1

b Problems ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓

c Time taken 1.5 mins1min 40

secs2.5 mins 2 mins 30 secs 8 mins 8 mins 10 mins

Q6

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 2mins 45 secs 20 secs 1 min 1 min 3 mins 3 mins 10 mins

Q7

a Found ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓

b Rating 5 2 4 2 4 3 4 5

c Problems ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✗

d Time taken 2 mins 5 mins 6 mins 12 mins 2 mins 10 mins 12 mins 2 mins

Q8a Found ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗

b Problems ✗ ✓ ✗ ✓ ✓ ✓ ✗ ✓

c Time taken 4 mins 3mins 4 mins 6 mins 5 mins

Q9

a Route No ofstages

4 8 9 4 2

b Problems ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓

c Time taken 2 mins 2mins 2 mins 45 secs 2 mins 15 mins

Q10a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Problems ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗

c Time taken 1.5 mins 1 min 6 mins 6 mins 8 mins 15 mins



Table A1.5 Results from Ashton

Shopper 1 Shopper 1 Shopper 1 Shopper 1 Shopper 2 Shopper 2 Shopper 2 Shopper 2Target SiteAshton Carlton Denton Burton Ashton Carlton Denton Burton

Q1a Rating 5 5 5 3 5 5 3 2

b Found ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓

c Time taken 5 sec 30 secs 1 min 5 mins 1 min 2 mins 6 mins 10 mins

Q2

a Found ✗ ✓ ✓ ✓ ✗ ✓ ✓ ✓

b Time taken 10 mins 1 min 5 mins 4 mins 5 mins 4 mins 2 mins 2 mins

Q3a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 1 min 5 mins 5 mins 5 mins 3 mins 18 mins 2 mins 1 min

Q4a Found ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓

b Time taken 5 mins 5 mins 10 mins 35 mins 5 mins 10 mins 15 mins 8 mins

Q5a Rating 2 4 0 0 2

b Problems ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

c Time taken 10 mins 15 mins 20 mins 20 mins 10 mins 30 mins

Q6a Found ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗

b Time taken 10 mins 3 mins 5 mins 10 mins

Q7a Found ✗ ✗ ✓ ✗ ✗ ✗ ✓ ✓

b Rating 4 0 0 0 0c Problems ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

d Time taken 20 mins 20 mins 5 mins 15 mins 15 mins 12 mins

Q8

a Found ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗

b Problems ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓

c Time taken 10 mins 5 mins 5 mins 10 mins 12 mins 10 mins 10 mins 10 mins

Q9a Route No of

stages2 2 2 2 4 2

b Problems ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓

c Time taken 1 min 30 secs 1 min 10 mins 15 mins 10 mins 10 mins

Q10

a Found ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✓

b Problems ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗

c Time taken 10 mins 20 mins 10 mins 5 mins 5 mins 15 mins 20 mins



Table A1.6 Results from Denton

Shopper 1 Shopper 1 Shopper 1 Shopper 1 Shopper 2 Shopper 2 Shopper 2 Shopper 2Target SiteDenton Burton Ashton Carlton Denton Burton Ashton Carlton

Q1

a Rating 5 5 5 3 5 3 5 2b Found ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓

c Time taken 1.5 mins 2 mins 30 secs 2 mins 30 secs 20 secs 10 secs 1 min

Q2a Found ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓

b Time taken 50 secs 1 min 3 mins 1 min 45 secs 30 secs 30 secs 20 secs

Q3

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 30 secs 1 min 2 mins 3 mins 30 secs 15 secs 15 secs 1 min

Q4

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 30 secs 2 mins 3 mins 3 mins 20 secs 30 secs 20 secs 30 secs

Q5

a Rating 3 5 0 0 3 4 3 4b Problems ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗

c Time taken 2 mins 3 mins 6 mins 5 mins 2 mins 20 secs 30 secs 30 secs

Q6

a Found ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 5 mins 8 mins 2 mins 4 mins 20 secs 2 mins 30 secs 1 min

Q7

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Rating 4 0 2 4 0 4

c Problems ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗

d Time taken 10.5 mins 10 mins 6 mins 7 mins 5 mins 1 min 2 mins 1 min

Q8a Found ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗

b Problems ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

c Time taken 5 mins 7 mins 5 mins 7 mins 2 mins 1 min 1 min

Q9

a Route No ofstages

2 1 3 3 3 2

b Problems ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✓

c Time taken 7 mins 6 mins 5 mins 4 mins 1 min 30 secs 20 secs 5 mins

Q10a Found ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓

b Problems ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✓

c Time taken 1.5mins 4 mins 2 mins 6 mins 30 secs 2 mins 1 min 5 mins



Table A1.7 Results from Burton

Shopper 1 Shopper 1 Shopper 1 Shopper 1 Shopper 2 Shopper 2 Shopper 2 Shopper 2Target SiteBurton Ashton Carlton Denton Burton Ashton Carlton Denton

Q1

a Rating 5 5 5 5 4 4 4 3b Found ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓

c Time taken 20 secs 20 secs 30 secs 30 secs 10 secs 10 secs 10 secs 20 secs

Q2a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 3 mins 2 mins 30 secs 1 min 10 secs 10 secs 5 secs 1.5 mins

Q3

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Time taken 2 mins 30 secs 1 min 1min 10 secs 15 secs 20 secs 1 min

Q4

a Found ✓ ✓ ✓ ✗ ✓ ✓ ✗ ✓

b Time Taken 5 mins 20 secs 20 secs 5 min 15 secs 10 secs 45 secs

Q5

a Rating 3 3 2 4 2 3 3b Problems ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✓

c Time taken 5 mins 10 mins 10 mins 10 mins 5 secs 15 mins 15 secs 30 secs

Q6

a Found ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✓

b Time taken 5 mins 1 min 10 mins 5 mins 15 secs 10 mins 1 min 1 min

Q7

a Found ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

b Rating 2 4 3 2 4 4 4

c Problems ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✗

d Time taken 10 mins 2 mins 5 mins 10 mins 1 min 10 mins 1 min 30 secs

Q8a Found ✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓

b Problems ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓

c Time taken 10 mins 5 mins 25 secs 30 secs 2 mins

Q9

a Route No ofstages

2 3 5 5 4 5

b Problems ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓

c Time taken 2 mins 30 secs 5 mins 25 secs 2 mins

Q10a Found ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓

b Problems ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗

c Time taken 2 mins 5-10 mins 10 mins 5 mins 1.5 mins 1 min 1.5 mins 45 secs

Appendix 1.5: Historical Development of the Websites


Appendix 1.5 Historical development of the websites

Ashton University

The website for Ashton University’s Library was first created in the mid 90’s by a member of thelibrary staff. At this time she was a subject librarian but had the necessary skills to create whatwas a basic, text-based, website. Over the years, the site was developed with the assistance ofsubject librarians and a working party, until approximately 2 years ago.

Since then the web pages have been re-designed twice, again using existing library staff.Responsibility for updating the content has been shared by media technicians and teamrepresentatives, under the steer of a project group.

In September 2001 two new posts were developed. One was a web developer, whose specific aimwas to steer web page development and a second to co-ordinate electronic resource subscriptions.The latter works alongside the Library Systems team to try to improve access to electronicresources. The web developer is based within Library Systems and his primary tasks are tocontrol web page development, undertake web page updates (with a half-time assistant), planstrategically for future web development, investigate web related technologies with a view topersonalising content for users in the near future, investigate authentication methods with a viewto automating authentication to web based resources.

Currently, the web developer is working on a completely new site, which will be database driven(this being the first step towards personalised content) and will be launched in time for the newacademic year in Sept 2002. Alongside this the electronic resource “search and authenticate” webpage will be launched, where users will be able to launch and authenticate to any web basedresource which we subscribe to (if Off Campus).

Burton University

The library website was first created in late 1994/early 1995 by a member of staff. At this time hewas a subject librarian but had the necessary skills to create what was, at first, a rudimentarywebsite.

Over the years, the site has been developed with the assistance of subject librarians, untilapproximately two years ago. The employment of an Electronic Resources co-ordinator hasconsolidated the updating and improvement of the website and now all changes go through thisposition.

It must be noted that from the initial creation of the site until the present day, all people involvedhave been Library employees, i.e. no external web professionals have been involved.

Prior to the results of this exercise, we had already embarked upon a website improvementprocess. We have entered into an agreement with a company named Esprit Soutron Partnership(ESP). We are already into the process of creating a searchable website, initially for Off Campususers. However, it is intended that the website will expand to become the new site for the libraryand learning resources.

Appendix 1.5: Historical Development of the Websites


Carlton University

From 1996 to 2000 the Carlton University website consisted of static web pages listing internet,cd-rom and external web databases. Roughly the same information that appeared in the libraryprinted guide was converted to web pages.

From September 2000 the electronic resource pages have been produced dynamically allowingincreased entry points (e.g. by subject, by title, from where the resource can be used) and asearch function.

The pages have been designed and produced by the two IT Advisors in the library. The design ofthe pages has been led by a general template produced for all university departments and bytechnical limitations.

The proliferation of databases used to produce the pages (from one to seven) and a move to acentralised web form has led to a redesign of the databases and navigational design of the pages.Improvements to reliability and navigational design will be in place by September 2002. Otherimprovements to searchability and subject access will be in place by September 2003.

Denton University

In September 2001 the Learning Centre web pages were completely redesigned, rebranded andrelaunched. The new website was intended to be straightforward for an independent user toexploit. An A-Z index was introduced so a user could find the resources without detailedknowledge. A key objective was intuitive navigation to enable users to get where they wanted tobe with as few clicks and as little scrolling as possible. User education was developed by the useof subject pages at an expert or novice level.

The initial concept for the redesign was developed by two senior members of Learning Centrestaff, both with IT and library expertise. These ideas were then presented to a forum of subjectlibrarians and Information Officers and the input contributed to further developments. The webteam provided the technical support to make the redesign a reality.

Project 1 Off Campus Services Benchmarking Project 2001/02 ... · Project 1: Off Campus Services As...

Documents

Transcript of Project 1 Off Campus Services Benchmarking Project 2001/02 ... · Project 1: Off Campus Services As...