JISC User Behaviour Monitoring and Evaluation Framework

Professor Jennifer Rowley, with the JUSTEIS (University of

Aberystwyth), and JUBILEE (University of Northumbria) Teams

Structure

• Introduction

• Methodology

• Profiling User Behaviour

• Managing for Effective EIS

• Framework Futures

Introduction - Aims

• To assure the Joint Information Services Committee (JISC) that their investment in digital content and network infrastructure facilitates use and has a positive impact on learning

• To identify barriers and facilitators to use of Electronic Information Services (EIS)

Introduction – Strands and Projects

• 3 Strands– Strand A – JUSTEIS (JISC Usage Surveys:

Trends in Electronic Information Service)– Strand C – JUSTEIS – General survey of EIS

provision– Strand D – JUBILEE (JISC User Behaviour in

Information seeking: longitudinal evaluation of EIS)

Introduction - Cycles

• Three annual cycles– 1999/2000 – Developing data collection methodology

– 2000/2001 – Further population of data sets, Validation of methodology, Pilot in Further Education (FE), Further integration, Help desk survey

– 2001/2002 – Crystallisation of deliverables, Full FE Project, including Action Research Strand

• Consistent approaches, but evolution

Introduction - Deliverables

• Multi-dimensional across sector (FE and HE) methodology

• Profile of user behaviour with some cycle by cycle comparison

• EIS Diagnostic Toolkit• Methodology for monitoring EIS resource

availability• Profile of EIS resources available• Dissemination and sharing of practice - contacts

Methodology - User Behaviour

• Strand A - Interviews and postal and e-mail questionnaires based on critical incident technique, and critical success factors. Also vignettes in Cycle 3 - Sampling at institution, departmental, and individual level.

• Strand D - Interviews/focus groups, questionnaires - Sites chosen on basis of 3 disciplines per cycle, and 6 institutions.

Methodology - EIS Availability

• Strand C - EIS Availability– stratified random sample of HEI’s.– analysis of EIS made available through the LIS

web site– used taxonomy of types of sources.

Methodology - Data collection

SitesA

NumberA

SitesD

NumberD

SitesC

Cycle1

26 653 6x3 763 44

Cycle2

35 711 6x3 506 48

Cycle3

29 611 6x3 431 29

Total 90 1974 54 1700 121

Profiling User Behaviour 1

• 80% of student use of EIS is for coursework, and 20% for personal reasons

• Students mix leisure and study use, and home and in-university facilities.

• Internet and search engines are the most popular EIS - Yahoo to Google.

• Use of gateways e.g. SOSIG and OMNI) and JISC-negotiated services, such as Web of Science, is limited but possibly increasing

• Electronic journals are used and use is increasing

Profiling User Behaviour 2

• There are disciplinary differences in EIS use, with Humanities and Arts students using databases least.

• Levels of information literacy are variable• Main influences on student use are tutor expectations and

explicit assessment requirements, followed by peer group influences.

• Information skills are learnt from various sources• Academic staff need support in developing their EIS skills• LIS provide access to a wide range of resources• Concerns about provision of resources (licensing, archives

etc) have not changed through the three cycles.

Student Voices

• ‘It was an Internet based journal site (Interviewer: Can you tell me what that was?) …Medline’ (Medical student)

• ‘I’m looking for general articles or something like that I tend to go through the university web site… and locate the database or e-journals through there.. They tend to have full journal articles.’

• ‘OK, last night I’m looking for information about Edinburgh Create Centre.. So I didn’t know the address, but I used Freeserve and Google.. I just put in the name and it came up with the address’.

Student Voices

• ‘ABI Inform was recommended by one of the other people on my course, and Yahoo, I just always go to Yahoo, and search for anything.. I’m not that good with the universities databases.’

• ‘probably go to the pharmacology textbook, then maybe try the Internet..’

• ‘It just gave a list of different things and I tried them all.. I could have come into the library but I was too lazy so I just did it from home.’

• ‘Can be confusing simply by the huge amount of information provided’

Comments from the Professionals

• ‘You’ve got some students who know how to use it really well, but other students who have got fairly basic knowledge of it’ (Law academic)

• ‘. .. Its more about giving the skills to acquire and seek information.. We’re expecting them more and more to find their own information.. You’ve got to be able to filter that information and be critical about it…’

• ‘Instant gratification. One stop shop .. Which is why Google is so popular, it gives you the illusion that you’re quickly into the relevant sites’ (Library staff)

Comments from the Professionals

• ‘In the area of art and design we are very much based on a kind of a student led approach in the project and studio environment .. We don’t think that it is relevant to our… courses’ (Art academic)

• ‘We warn them that there is an awful lot of rubbish as well as some good stuff. I am not convinced that they can tell the difference’ (Geography academic)

• ‘People working in information have gone into overload.. users are in the same boat’ (Library staff)

Barriers to Use of EIS

• Information overload• Increasing workloads• Remote access• Authentication procedures• Legal restrictions• Technical support• Cost management• Skills training and the ‘fear factor’• Apparent irrelevance

Student Voices - Barriers

• ‘Sometimes too much information making it difficult for me to pinpoint exactly what I need’ (Business student)

• ‘I’m not sure how to use some packages so I’m frightened of crashing the computer’ (Art and Design student)

• ‘Some sites are difficult to define and find’ (Business student)

• ‘It takes time to find information’ (Business student)

Enablers to use of EIS

• Access• Speed• Coverage and currency• Space• Cost• Market demand• Management• Staff EIS champions• Promotion and awareness

Student Voices - Positives

• ‘It has helped me improve my studies because I have been able to get lots of information to help with assignments’ (Art and Design student)

• ‘Information is accessible without leaving the comfort of your own home’ (Business student)

• ‘Given me access to information and graphics that would not otherwise be so easily found, viewed and down loaded’ (Art and Design student)

• ‘Access to more and better standard of information’ (Geography student)

Managing for effective EIS Use

• Responding to user behaviour at institutional level.

• EIS Diagnostic Toolkit – user defined themes:– Access to information– Resource base– User skills– Course design and student learning– Quality

Framework Futures 1

• Ambitious project, successfully completed – cross sectoral nature – joined-up

• Multidimensional, multi-team framework has strengths, but integration needs attention

• Simple quantitative surveys offer limited insights, and a more sophisticated view of longitudinal surveying is important

• Student, academic staff and LIS managers voices all need to be heard

• Access to respondents is not easy

Framework Futures 2

• A consistent methodology across FE and HE offers useful insights

• Further refinement of The EIS Diagnostic Toolkit will emerge from its use and extension to FE

• The Fieldwork creates rich data sets that could be further analysed.

• Fourth Cycle 2002/2003

Recommendations

• Adoption of innovation is ‘slow’• Need to create a pedagogy that embeds EIS in

learning• Developments with VLE’s and MLE’s require

partnerships across academics, educational developers, and LIS staff.

• Need to join up initiatives from various national bodies: JISC, Learning and Teaching Support Network, and Institute for Teaching and Learning activities.

Questions Welcome

rowley@edgehill.ac.uk

www.jisc.assist

Framework in FE – Survey (2001/2002)

Framework in FE-Survey 2001/2002

• Methodology

• Summary of Results

Methodology

• Parallels HE Methodology• JUSTEIS - South of UK- 270 respondents

(students, LIS staff, academics)• JUBILEE – North East England – 528

respondents (students, LIS staff, academics)• Various data gathering approaches

including interviews, questionnaires, e-mail interviews, and focus groups.

Summary of Results 1

• Sector in change, and very varied – difficult to spot change.

• Modest levels of LRC resourcing:– No obvious EIS leader

– Absence of expertise in academic and LIS staff

– Little time to develop

– Poor access to IT for tutors

• Closer relationships between tutors and students

Summary of Results 2• Different student profile

– Integrated EIS use across study, leisure and work

– EIS skills acquired from peers

– Books, the Internet and organisational web sites preferred as sources of information

– Low evaluation awareness

– Basic search strategies

– Difficult to detect disciplinary differences in FE

– Text messaging may be more important than e-mail.

Summary of Results 3• FEFC initiatives have improved the

situation– Much improved ICT resource base– Access to JISC resources is welcomed by LIS

staff, but low use is reported– LIS managers face difficult budgetary decisions– Suitable EIS are not always available– Colleges are working towards VLE’s and

MLE’s.

Framework in FE - Action Research

(2001/2002)

Structure

• Rationale

• Methodology

• Findings

Rationale

• Context– 2000/2001 Pilot study in FE on user behaviour– Imperative for rapid development of EIS in FE– Full scale user behaviour survey in FE

2001/2002

• Aims– To work with FE staff to develop solutions for

improving EIS (and specifically JISC EIS) uptake

Methodology - Sites

• JUBILEE-1 – Small College- English• JUBILEE-2 – Medium College –

Geography/Environmental Science• JUBILEE-4 – Large College – Sports Science• JUSTEIS-1 – Small college – Clinical medicine• JUSTEIS-2 – Large College (2 sites)– Motor

Vehicle engineering• JUSTEIS-3 – Medium College (franchise) –

Sports Sciences

Methodology – Action Research

• Stages– Problem identification– Intervention design– Intervention implementation– Intervention evaluation

• Interventions– Access to LION– Subject guide– Training session on Infotrac– Overhaul of ICT and LR approaches and resources– Website development

Findings - Methodology

• Difficult in managing interventions within project funding schedules

• Interventions as products and services don’t ‘stick’ unless wider organisational and learning issues are addressed

• Levels of engagement of stakeholders vary and affect the impact of the action research.

Findings - Outcomes

• Awareness of EIS can be raised but immediate impact may be low

• Targeting of learning resources for subject, level and course is important

• Use of specialised EIS is constrained by:– Preference for books and Internet– Absence of FE targeted EIS– Poor access to IT– Licensing and franchising