Impact, explosion, fallout: what have we learned

from the 2014 Research Excellence Framework?

Roger Watson

Purpose of the REF• The REF is a process of expert review

• It replaces the RAE as the UK-wide framework for assessing research in all disciplines

• Its purpose is:

- To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)

- Provide accountability for public funding of research and demonstrate its benefits

- To provide benchmarks and reputational yardsticks

Overview:

Overview of the processREF assessed the

quality of research in all UK universities, in all

disciplines. It was carried out by 36 expert panels, grouped into 4

main panels.

2011-12PreparationPanels were appointed.Guidance and criteria were published.

2012-13SubmissionsUniversities made submissions in whichever subjects they chose to.

2014Assessment36 expert panels reviewed the submissions, guided by the 4 main panels.

Main Panel A: Medical and life sciences

Main Panel B: Physical sciences and engineering

Main panel C: Social sciences

Main Panel D: Arts and humanities

Expert Panels• Submissions were assessed by 36 Sub-panels working under

the guidance of four Main Panels

Each Main Panel comprised:• The chair

• Chairs of each sub-panel

• International members

• User members

Each Sub-Panel comprised:• The chair and deputy chair

• Panel members

• Additional assessors (for outputs and

impact)

• On average ~30 people

• The Equalities and Diversity Panel (EDAP) reviewed complex staff circumstances

Working methods

• Work was allocated to members and assessors according to expertise

• Outputs were cross-referred to other sub-panels where necessary

• Strict procedures were in place to avoid conflicts of interest• Significant attention was given to ensure comparability of

assessment standards, at sub- and main panel level: − Calibration exercises− Ongoing review of emerging scores

• Where used, citation data played a minor role in informing peer review of the outputs

Development of sub-profiles• Each outputs was graded on the scale:

4* : World-leading

3* : Internationally excellent

2* : Recognised internationally

1* : Recognised nationally

U : Below the standard for 1* or deemed ineligible• Each impact case study, the impact template, and each section

of the environment template was graded on a scale from 4* - U, using ‘half-marks’ for borderline judgments.

• Each sub-profile for each submission was collectively agreed by the sub-panel and recommended to the main panel for approval.

• The overall quality profile for each submission aggregates the three sub-profiles, according to the standard weightings.

EXAMPLE - 2014 Research Excellence Framework ResultsQuality profiles for all submissions (sample)

Institution name

Main panel

Unit of assessment name Profile

FTE Category

A staff submitte

d 4* 3* 2* 1*unclassified

University X A Biological Sciences Outputs 50.45 12.8 32.8 43.0 11.4 0.0University X A Biological Sciences Impact 50.45 20.0 45.0 35.0 0.0 0.0University X A Biological Sciences Environment 50.45 0.0 40.0 40.0 20.0 0.0University X A Biological Sciences Overall 50.45 12 37 41 0 0University X B General Engineering Outputs 65.20 25.9 43.1 27.0 4.0 0.0University X B General Engineering Impact 65.20 17.9 60.1 21.0 1.0 0.0University X B General Engineering Environment 65.20 10.0 70.0 20.0 0.0 0.0University X B General Engineering Overall 65.20 22 51 24 3 0University X A Architecture Outputs 40.00 17.0 60.0 20.0 2.0 1.0

The overall quality profile is weighted 65% for Outputs, 20% for Impacts, 15% Environment

Published results include the three sub-profiles and the overall quality profile for each submission

For the first time, REF has demonstrated the impact of UK research

• Across the exercise, over 250 research users judged the impacts, jointly with academic panel members.

• Across the exercise, 44% of impacts were judged outstanding (4*) and a further 40% were judged very considerable (3*).

• Impressive impacts were found from research in all subjects.

• REF shows many ways in which research has fuelled economic prosperity, influenced public policy and services, enhanced communities and civic society, enriched cultural life, improved health and wellbeing, and tackled environmental challenges.

HEFCE HEI BUDGET: £4 billion

The assessment framework

Overall quality

Outputs

Maximum of 4 outputs per researcher

Impact

Impact template and case studies

Environment

Environment data and template

65% 20% 15%

Overview:

Submissions • Each HEI may submit in any or all of the 36 units of

assessment (UOAs)

• Each submission in a UOA provides evidence about the activity and achievements of a ‘submitted unit’ including:

- Staff details (REF1a/b/c)

- Research outputs (REF2)

- Impact template and case studies (REF3a/b)

- Environment data (REF4a/b/c)

- Environment template (REF5)

• A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit

Overview:

Publication of results• The primary outcome of the REF is an ‘overall quality

profile’ to be awarded to each submission:

- Using the same scale as RAE2008, but in steps of 1%

• Further reports and feedback will be provided:

- Overview reports by panels

- Concise feedback on submissions, to the heads of HEIs

- The output, impact and environment sub-profiles for each submission will be published

- A report by the Equality and Diversity Advisory Panel

• Submissions will be published (except for confidential or sensitive information)

Overview:

The results have been published!

Outputs Impact Environment

4* 3* 2* 1* U

20 45 35 0 0

4* 3* 2* 1* U

0 40 40 20 0

65%

Overall Quality Profile

12

4*

0104137

U1*2*3*

4* 3* 2* 1* U

12.8 32.8 43 11.4 0

20% 15%

The overall quality profile is comprised of the aggregate of the weighted sub-profiles produced for outputs, impact and environment.

Quality Level

% of Research Activity

Example of a quality profileOverview:

Main panel working methods• Each main panel has developed a consistent set of

criteria for its group of sub-panels• Each main panel will guide its sub-panels throughout

the assessment phase, ensuring:- Adherence to the published criteria

- Consistent application of the overall standards of assessment

• Main panels will undertake calibration exercises and keep the emerging outcomes under review

• Main panel international and user members will be engaged at key stages across the sub-panels

REF panels:

Sub-panel working methods• Sub-panels will review their expertise to ensure

appropriate coverage

• Work will be allocated to members/assessors with appropriate expertise

• Each sub-panel will run calibration exercises for outputs and impacts, guided by the main panels

• All outputs will be examined in sufficient detail to contribute to the formation of the outputs sub-profiles

• Each case study will normally be assessed by at least one academic and one user

• Graduated sub-profiles will be formed for each aspect of submissions

REF panels:

Definition of impact• Impact is defined broadly for the REF: an effect on,

change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

• Panels recognise that impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location

• Panels provide examples of impact relevant to their disciplines, intended to stimulate ideas - not as exhaustive or prescriptive lists

Impact:

Some examples of impactImpact:

Public debate has been shaped or informed by research

A social enterprise initiative has been created

Policy debate or decisions have been influenced or shaped by research

A new product has been commercialised

Enhanced professional standards, ethics, guidelines or training

Jobs have been created or protected

Improved business performance

Changes to the design or delivery of the school curriculum

The policies or activities of NGOs or charities have been informed by research

Improved management or conservation of natural resources

Improved forensic methods or expert systems

Production costs have reduced

Levels of waste have reduced

Improved quality, accessibility or efficiency of a public service

Enhanced preservation, conservation or presentation of cultural heritage

Organisations have adapted to changing cultural values

New forms of artistic expression or changes to creative practice

More effective management or workplace practices

Changes to legislation or regulations

Enhanced corporate social responsibility policies

Research has informed public understanding, values, attitudes or behaviours

Improved access to justice, employment or education

Enhanced technical standards or protocols

Improved risk management

Improved health or welfare outcomes

Research has enabled stakeholders to challenge conventional wisdom

Changes in professional practice

Submission requirements• Sets out the submitted unit’s general

approach to supporting impact from its research:

• Approach to supporting impact during the period 2008 to 2013

• Forward strategy and plans

Impact template (REF3a)

• Specific examples of impacts already achieved, that were underpinned by the submitted unit’s research:

• 1 case study per 10 FTE staff submitted (plus 1 extra)

• Impacts during 2008 to 2013; underpinned by research since 1993

Case studies (REF3b)

Impact:

20% of the impact sub-

profile

80% of the impact sub-

profile

Case studies • Each case study should:

- Clearly describe the underpinning research, who undertook it and when

- Provide references to the research and evidence of quality

- Explain how the research led/contributed to the impact

- Clearly identify the beneficiaries and define the impact

- Provide evidence/indicators of the impact

- Provide independent sources of corroboration

• All the material required to make a judgement should be included in the case study

• Submitted case studies need not be representative of activity across the unit: pick the strongest examples

Impact:

Underpinning research• Each case study must be underpinned by research that:

- was produced by staff while working in the submitting HEI

- is evidenced by outputs published between 1 Jan 1993 to 31 Dec 2013

- meets the quality threshold of at least equivalent to 2*

- made a material and distinct contribution to the impact (there are many possible ‘routes’ to impact, but in each case a distinct and material contribution must be shown)

• Once the panel is satisfied that these criteria have been met, it will assess and grade the case study in terms of the ‘reach and significance’ of the impact

Impact:

Evidence of impact• Case studies should provide a clear and coherent

narrative linking the research to the impact

• Including evidence most appropriate to the case being made

• Evidence may take many different forms, including quantitative (where possible) and qualitative. Panels provide examples, which are not exhaustive or prescriptive

• Key claims should be capable of verification. Independent sources of corroboration should listed, to be used for audit purposes

Impact:

Assessment criteria• The criteria for assessing impact are reach and

significance

• In assessing a case study, the panel will form an overall view about the impact’s reach and significance taken as a whole, rather than assess each criterion separately

• ‘Reach’ is not a geographic scale. Sub-panels will consider a number of dimensions to the ‘reach’ as appropriate to the nature of the impact.

• In assessing the impact template, the panel will consider the extent to which the unit’s approach is conducive to achieving impacts of ‘reach and significance’

Impact:

Assessment criteriaThe criteria for assessing impacts are reach and significance*

Four star Outstanding impacts in terms of their reach and significance

Three star Very considerable impacts in terms of their reach and significance

Two star Considerable impacts in terms of their reach and significance

One star Recognised but modest impacts in terms of their reach and significance

UnclassifiedThe impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

* Each main panel provides descriptive account of the criteria

Impact:

What counts as impact?

Impact includes but is not limited to, an effect on, change or benefit to:

• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

• of an audience, beneficiary, community, constituency, organisation or individuals

• in any geographic location whether locally, regionally, nationally or internationally

What counts as impact?

Impact includes the reduction or prevention of harm, risk, cost or other negative effects.

What does not count as impact?

For the purposes of the impact element of the REF:

• impacts on research or the advancement of academic knowledge within the higher education sector are excluded

• impacts on students, teaching or other activities within the submitting HEI are excluded.

• other impacts within the higher education sector, including on teaching or students, are included where they extend significantly beyond the submitting HEI

Reach and significance

Impacts will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred, whether locally, nationally or internationally.

Common errors in impact statements

• Not following the guidance

• Including inappropriate impact– Research impact– Teaching impact

• Including staff not employed in the correct period

• Using personal letters in support of impact

• URLs that led to no information

• Simply not addressing the issue of impact

Easy shots

• Be familiar with the guidance in detail

• Follow the guidance to the letter

• For named staff, provide dates of employment

• Don’t ‘overuse’ media consultants and journalists

• Don’t use too many inane diagrams

Things to do

• Respond to all and any consultations on 2020 REF

• Build impact into research proposals

• Consider impact at research meetings

• Start gathering evidence now

• Create the evidence now:– Webpages with counters– Research outcomes translated for the general

public– Hold public understanding of science lectures– Use social media effectively

Further informationwww.ref.ac.uk includes:

• The results and submissions

• Summary data and analysis

• Panel overview reports

• Confidential feedback on submissions was provided to institutions in January

• Evaluations of the REF are currently being carried out by the funding bodies, with reports due in March

email: r.watson@hull.ac.uk

@rwatson1955