UKOLN is supported by:

Evidence, Impact, Metrics: Proposal For Further Work

Brian Kelly

UKOLN

University of Bath

Bath, UK

This work is licensed under a Attribution-NonCommercial-ShareAlike 2.0 licence (but note caveat)

Acceptable Use PolicyRecording this talk, taking photos, discussing the talk using Twitter, blogs, etc. is permitted but please try to minimise distractions to others.

Acceptable Use PolicyRecording this talk, taking photos, discussing the talk using Twitter, blogs, etc. is permitted but please try to minimise distractions to others.

Blog: Twitter:http://ukwebfocus.wordpress.com/ @briankelly

TimetableUKOLN’s EIM Work 2010/11

Context Examples Additional documents

Proposals for 2011/12 Work for DI programmes Work relevant to

DI programmes Work for sector

2

Discursive

Tangible deliverables, identified owners & dates

• That’s interesting• I didn’t know that• I disagree with that

• I want a piece of the action

• That will help my programme

• This is already being done

• It’s too hard

Background

Evidence, Impact and Metrics Work• Engage with community in order to explore

ways of gathering evidence of the value of online services and explore appropriate metrics to facilitate comparisons

• Scoped to institutional use of web services and professional/institutional use of social web services

• Three 1-day workshops (~100 participants); 6 additional talks and ~24 evidence-based blog posts (21,800+ views)

3

Discursive

Changes In Cultural

Some initial scepticism and mistrust• We know we’re doing well• Bean-counters• You’ll use evidence against us• Metrics are flawed• Systems can be gamed• ….

Fortunately we had external support in changing the culture …

4

Discursive

The Context •The need to demonstrate value in light of Government cuts 5

NB 1D view: no comments on ROI, comparisons with print, TV, … costs

Discursive

Daily Telegraph

Daily Telegraph data journalist used FOI to gather figures on Web site redevelopment costs.

UK Web Focus & SiD blog posts pointed out flaws in interpretation.

Sector realised that they couldn’t ignore such threats.

6

Discursive

Open DataGovernment to ‘force’ Unis to publish data about courses, student experiences, …• Builds on Labour’s

Open Gov initiative• An opportunity to be

welcomed? • Can take initiative in

publishing open data about uses of online services

7

Discursive

Numbers Matter

8

“Sales of 25 million iPads in just 14 months. 200 million iOS devices in total. 15 billion songs downloaded since 2003. 130 million books. 14 billion apps downloaded from a store that now runs to 425,000 apps.”

BBC News 7 Jun 11

JISC-funded OII Impact report: “The media and the public are influenced by numbers and metrics”

Discursive

Bath University Cares• Bath University is the “”• The Sunday Times (and THE) University

League Tables are based on metrics• Ignoring the reality can be dangerous

9

Discursive

Evidence-Gathering

Evidence-based surveys and discussions published on UK Web Focus blog

Aims:• Explore methodologies (strengths &

weaknesses)• Provide benchmarks for future comparisons• Identify patterns of uses which can help to

inform practice & policy• Invite & encourage discussions on survey

methodologies and interpretations and sense-making processes

10

Evidence

Examples• The Nudge Principle (cf Michael Neilson’s

talk at SOLO11) applied to RSS use across Scottish Unis

• Use of HTML and RDFa by Welsh Unis• Evidence of JISC PoWR’s impact• Cookie usage by Russell Group Unis• Mobile-readiness of Russell Group Unis• Effectiveness of comms channels (e.g.

JISCMail, Slideshare) and how evidence can inform policy

11

Evidence

Proposal 1Evidence-gathering to support Linked U call

• Snapshot of existing approaches to HTML across selected HEIs to act as benchmark (Nov 2011)

• More comprehensive snapshot in Jun 2012• Report to inform JISC • Presentation at IWMW 2012 aimed at engaging

sector

Audiences: JISC PM, Web managers

Activity: Survey, interpretation and open discussion, presentation.

Times: Part 1 in Nov 2011 Part 2 in Jun/Jul 2012

12

Proposal

Proposal 2Evidence-gathering on use of Cookies

• Snapshot of existing uses of cookies and privacy policies across selected HEIs and other sectors to act as benchmark (Dec 2011)

• More comprehensive snapshot in Jun 2012• Analysis to inform discussions with JISC Legal • Presentation at IWMW 2012 aimed at engaging

sector• Report on recommendations

Audiences: Institutions, Web managers, JISC PMs

Activity: Survey, interpretation and open discussion, presentation.

Times: Part 1 in Dec 2011 Part 2 in Jun/Jul 201213

Proposal

Proposal 3Consultancy to JISC PMs

• Advice provided to JISC PMs on possible ways of gathering quantitative evidence related to their programmes

Audiences: JISC PMs

Activity: Discussions and, if required, suggestions

Times: As appropriate

14

Proposal

Proposal 4Evidence-gathering to support xxxx call

• Snapshot across additional JISC Call in area relating to UKOLN expertise (Libraries, mobiles?)

• Report to inform JISC and sector

Audiences: JISC PMs, xxx stakeholders

Activity: Survey, interpretation and open discussion, presentation.

Times: To be negotiated

15

Proposal

Discussion

16