Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013

Digital Humanities:

the application of digital technology to humanities disciplines reflection upon the impact of digital media upon humanity

> 50 academics & researchers

~ £2.5 million research income per annum

5+ million digital objects in 107+projects

200+ million hits over the last 5 years

www.kcl.ac.uk/ddh/

http://www.kdcs.kcl.ac.uk/innovation/impact.html

Measuring the Impact of Digitized Resources: The Balanced Value Model

“the measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community”

www.kdcs.kcl.ac.uk/innovation/impact.html

The case for Impact

We are more effective and efficient in delivering change and tangible benefits (Internal Impact);

Our organisation is gaining strategic advantage through the innovation inherent in this digital activity (Innovation Impact);

We are delivering a strong economic benefit to our community that demonstrate the worth and value of our endeavours in clear monetary terms (Economic Impact); and

the community has been changed by the resource in beneficial ways that can be clearly identified (Social Impact)

Impact taskforce

Simon Tanner

Aggregated • 2200+ content providers • 173 aggregators • 26.9 million objects

Network • 600+ individual members, working in taskforces and on strategy

Breakdown Galleries: 2 Libraries: 111 Archives: 26 Museums: 60 National Aggs: 22 Publishers: 2 Creative Ind: 5 Research: 78 Ministries: 9 Other: 174

Stakeholder Survey and Mapping

A person, group, community, or organization who affects or can be affected by the ecosystem of the digital resource to be assessed.

Questions to be answered in establishing and identifying stakeholders include:

Have all primary stakeholders been listed? Have all secondary stakeholders been listed? Have all potential supporters and opponents been identified? Have the interests of vulnerable or minority groups been identified? What new primary or secondary stakeholders are likely to emerge?

Stakeholders – the kinds of broad groups

Consumers

One Stop Consumers

Partners and Collaborators

Paymasters

Producers and Creators?

Commentators

Marginalised

Leavers

Non-users

Champions

Competitors

Stakeholder Survey and Mapping

Stakeholder mapping is a powerful tool to enable planning

It enables a list of stakeholders to become an active tool in three simple steps

The steps:

Name your stakeholders and put them in categories Map where the stakeholder is now in relation to your organisation or the initiative for which you want their support Now map where you need the stakeholder to be to enable your organisation or initiative to succeed.

Use this information to plan actions/timing etc.

Stakeholder Mapping

Knowledge

Influence / Initiative

Champions & Activists

Context Setters

Influencers

Bystanders

Stakeholder Mapping: Example

Knowledge


Govt. Funder

Fund project

Publisher

Gain IP agreement

Govt.

CEO

Champion

Exercise: Draw your own

Knowledge


Champions & Activists

Context Setters

Influencers

Bystanders

PERSPECTIVE + VALUE DRIVER

OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS

CURRENT USE (SOCIAL)

Users

Those with an interest in the intellectual content find it useful to their research

Users of the Code-breakers resource.

1. Discovery 2. Engagement 3. Usefulness

Google Analytics review Site surveys sampling users on the Codebreakers microsite. Tracking of recommendations to others or reviews. User panels. Citations of content on the site.

1. Discovery of resource: a. Web visits/visitors b. Views to digitised content c. Relative use to historical use figures (where

applicable) 1. Engagement with resource:

a. Average time spent on digitised content b. Repeat visitors to Codebreakers c. Downloads of content d. User journeys across Codebreakers

1. Usefulness of resource: a. Site surveying to collect reported usage and utility of Codebreakers resource b. Citation indices – this can only be a very long term measure due to research and publishing timescales c. User panel – recruited from actual users of the Codebreakers resource. Engage in enquiry of the function, content and discoverability of Codebreakers.

d. Desk research to find innovative use of the content e. Online media monitoring to capture people’s mentions and recommendations of Codebreakers.



CURRENT USE (SOCIAL)

Community

Peer organisations and members of our professional community have changed their policy or practice concerning digitisation projects.

Practitioners, peer organisations and members of our professional community who have been influenced by the project

1. Awareness of the project

2. Take up of methods/ approaches/ standards

3. Impact of take up on partner and peer organisations

Survey of partner organisations who worked on the project Survey of event attendees / key peer organisations Desk research

1. Awareness: a. Number attending digitisation open days/events

held at Wellcome Library b. Contacts from peer organisations/practitioners c. Online media monitoring for blogs, conference

presentations, events, workshops, open days, social media etc.

d. Citations/references to the project 1. Take up: a. Survey of partner and peer organisations, and

practitioners identified in stage 1. b. Desk research to identify stakeholders influenced

by the Codebreakers project (e.g. Ronan Deazley work on archives and copyright)

1. Impact: a. Survey of partner and peer organisations, and practitioners identified in stage 1. b. Desk research

PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS POTENTIAL USE (INNOVATION)

Users

The Codebreakers project has enabled new potential activities and research methods for those interested in the intellectual content.

Potential users of the Codebreakers resource

1. Delivery of the planned functionality/ usability of the Codebreakers resource.

2. User understanding of the new research enabled by Codebreakers.

3. Unforeseen potentials for new research.

Evaluation of functional capabilities of the finished site against our initial goals. Heuristic evaluation of Codebreakers resource. Site survey. Focus group with potential users

1. Delivery: a. Desk work to check the functionality of the site against our

initial specification. b. Heuristic evaluation of usability 1. User understanding: a. Site survey of users on Codebreakers microsite b. Focus group research c. Usability research with potential users d. Heuristic evaluation 1. Unforeseen potentials:

Staff interviews to discover where unexpected benefits occurred during the build of the site.

Focus group recruited from potential Codebreakers users to discuss new opportunities for researchers.

Community

The extent to which Codebreakers has created new possibilities for organisations and professional members of the cultural heritage community.

Practitioners, peer organisations and members of our professional community who may be influenced by the project

1. The accessibility of technical developments.

2. Awareness of access to developments.

3. Uptake of practices initiated in the project as industry standard.

4. Unforeseen t ti l

Desk work to determine accessibility of technical and process developments. Qualitative measures such as questionnaires, desk research, structured interviews. A comparison of our initial goals with the final site.

1. Accessibility of developments: a. Desk work to check accessibility of technical developments

to the peer community b. Availability of documentation of process/organisational

developments 1. Awareness of access: a. Survey of partner and peer organisations, and practitioners

identified in Current Community Awareness. 1. Uptake as industry standards: a. As described in Current Community uptake 1. Unforeseen potentials: a. Survey of partner and peer organisations, and practitioners

identified in Current Community Awareness. b. Staff interviews to discover where unexpected benefits

occurred during the build of the site.

PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS

INTERNAL

Development

How have the staff of the Wellcome Trust had their skills, abilities, capacity and knowledge enhanced by developing Codebreakers

Staff of the Wellcome Library and Trust.

1. Changes in individual knowledge or skills.

2. Changes in working practices and behaviours

3. Changes in organisational capacity or ability.

Survey of Trust staff connected with the project. Interviews with line-managers. Interviews with senior managers.

1. Individual knowledge: a. Survey of Trust staff connected to Codebreakers activity. b. Interviewing line-managers of staff involved in the

Codebreakers project. 1. Working practices and behaviours: a. Survey of Trust staff connected to Codebreakers activity. b. Interviewing line-managers of staff involved in the

Codebreakers project. 1. Changes to organisation: a. Interviews with senior managers.

Inheritance / Bequest

How does Codebreakers represent the inheritance of the Wellcome Trust Library’s activities since collecting began and how does it prepare the Library for the future and bequeath benefits to future generations?

Staff and members of the Wellcome Trust.

1. Change in usage enabled by Codebreakers resource.

2. Value for future digitisation activity

3. Benchmarking against peer organisations.

4. Comparison to historical strategic direction of Library.

Google Analytics Interviews with senior managers Review of peer organisation activity Desk research

Change in usage: a. See Current Users usage Value for future digitisation activity: a. Interviews with senior managers b. Data from Internal Development Benchmarking: a. Review of peer organisations – desk research and

interviews to compare the Wellcome Library’s digital status in comparison with its peers worldwide.

Comparison to historical strategic direction: a. Desk research b. Interviews with senior managers



ECONOMIC

Users

What is the net economic effect of making the content freely available online?

Users of the Codebreakers resource.

Economic gain to individual users of the resource. Economic value generated for organisations that are end-users of the resource.

Google Analytics review Site survey User panels.

This will be based on methodology developed by the British Library in their 2013 economic evaluation. The full British Library report is available here. It will include: 1. Comparison of Codebreakers usage with archive

usage records over the last 5 years with an assessment of the cost of use.

2. User time spent on the Codebreakers resource 3. Users’ geographic location. 4. Equivalent cost implication for users consulting

across collections previously held in physically separate locations.

5. Contingent valuation questions included in site survey and user panels.

http://www.bl.uk/aboutus/stratpolprog/increasingvalue/index.html�

Thanks!

Thank you to all the folks at the Wellcome Library for allowing me to share this with you. Contacts Alexander Green Email: [email protected] Christy Henshaw Email: [email protected]

The case for Impact

We are more effective and efficient in delivering change and tangible benefits (Internal Impact);

Our organisation is gaining strategic advantage through the innovation inherent in this digital activity (Innovation Impact);

We are delivering a strong economic benefit to our community that demonstrate the worth and value of our endeavours in clear monetary terms (Economic Impact); and

the community has been changed by the resource in beneficial ways that can be clearly identified (Social Impact)

With thanks to Alice Maggs for the Impact illustrations [email protected]

Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013

Technology

Transcript of Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013