Dr Shantala MohanWestern Sydney University

Engagement and Impact Assessment

“Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia” (ARC 2015).

“Research engagement is the interaction between researchers and research end-users (including industry, government, non-governmental organisations, communities and community organisations), for the mutually beneficial exchange of knowledge, technologies and methods, and resources in a context of partnership and reciprocity” (ARC 2017).

Definitions

• May 2017: EI Assessment Piloto Engagement pilot (quant indicators plus narrative

statement) – across 4 two-digit FoRs – 03;11;21;and 22o Impact pilot (case studies plus quantitative data) –

across 6 two-digit FoRs – 05; 07; 09; 13; 19; and 20

• 2018: First full systematic EI national assessment alongside ERA o Nominations to the ERA RECs and EI Assessment

Panels – Late February 2017 (researchers and research end users)

• The pilot exercise will be used to inform the first full EI assessment in 2018

• Engagement and impact will be assessed separately – a Review Panel will examine pilot material and outcomes, and assess overall methodology

• A selection of 10 two-digit FoRs will be assessed (4 FoRs for engagement and 6 FoRs for impact) - UoA

• Timeline:o Submissions May 2017 (5-18 May)o Assessment May-June 2017o Reporting Mid-late 2017

2017 EI Pilot: Overview

• Will primarily assess quantitative information (engagement indicators) supplemented with qualitative information (narrative statement)

• FoRs selected:o 03 Chemical Scienceso 11 Medical and Health Scienceso 21 History and Archaeologyo 22 Philosophy and Religious Studies

• Reference period: 1 January 2008 - 31 December 2013 (6 yrs.)• Two assessment panels (A&B): will comprise a mix of academics and

research end-user experts across the chosen FoRs• ARC will use ERA 2015, ERA 2012 and other existing data for the

engagement indicators component – which will be provided to institutions for verification:o No recoding of research outputs, income or patent datao Where no new data are required, institutions don’t need to

resubmit ERA 2015 and ERA 2012 submission data

Engagement Pilot

Engagement IndicatorsIndicator FoRs

03, 11, 21

FoR 22

Counts of patents granted and patent family Y NCount of patent-cited papers Y NCo-authorship of research publications Y YCo-funded research outputs Y YProportion of total outputs available via open access Y YRatio of ARC Linkage grants to ARC Discovery grants Y YCash provided by research end-user partner on grants Y YIn-kind support provided by research end-user partner on grants

Y Y

Research commercialisation income Y YHERDC Category 1 specified schemes Y YHERDC Category 2 (other public sector research) income

Y Y

HERDC Category 3 (industry and other research) income

Y Y

HERDC Category 4 (CRC research) income Y Y

The engagement narrative (500 word maximum) will provide a description of:

• how the UoA engaged with parties, outside of academia, during the reference period for the mutual benefit of the institution researchers and research end-users;

• the duration and extent of the engagement activities; and

• how the engagement indicators reflect this activity.

Engagement Narrative

• Serving on external advisory boards• Consultations with community groups,

professional/practice organisations, government bodies

• Consultation with/advice to Government• Expert witness in court cases• Contributions/submissions to public

inquiries on industry-research related issues

• Public lectures, seminars, open days, school visits

• Presentations to practitioner communities

• Connections to cultural institutions, seminars/workshops, internships and engagement with the public

• HDR students in internships/placements• HDR student destinations• Support for cultural events/institutions

—e.g., Writers' Festival, Film Festival, Vivid, etc.

• Number of contracts for research, consulting, expert witness and testing

• Number of licences and assignments and options

• Donations for research, linkage grants, and other funding arising from engagement with external bodies

• Consultancy and contract income as a share of total research income

• Number of different clients with contracts worth greater than a threshold value

• Philanthropy linked to research support and in-kind support

• Book sales• Repeat business with industry• Start-up/spin-out companies

Additional Quantitative Indicators

• Collect data around performance and the different types of public activities in which researchers generally report their work to the community or use their research capacity to further the work of community organisations

• Event participation statistics (public lectures, cultural events, exhibitions, etc.)

• Outreach activities (public lectures, policy engagements, media engagements, community

• Media coverage of exhibitions and new works

• Modification of traditional 'commercialisation' metrics such as 'spin-offs' to capture arts entrepreneurship such as setting up galleries, ensembles, groups and other professional practice entities

• Metrics which capture social media activity

• Other quantitative information not listed above.

• Evidence of significant institutional partnerships—e.g., Sydney Health Partners; various global research consortia, OECD, World Bank, World Health Organization, UN, UNESCO

• Co-designing and collaborating on performances and exhibitions

• Mobility of researchers (i.e. researchers employed or placed outside academia, for example embedded with research end-user partners, and/or research end-user employees that are employed or placed within academia)

• Mentoring external research partners• Established networks and relationships

with research users• Involving users at all stages of the

research, including working with user stakeholder and participatory groups

• Memoranda of Understanding (MOU)/Agreements

Additional Quantitative Indicators

Mature• Strong evidence of research engagement within the UoA across a broad

range of indicators• Strong evidence that research engagement is well integrated into the

development and ongoing conduct of research within the UoA. Emerging• Evidence of research engagement within the UoA across a number of

indicators.• Evidence that research engagement is incorporated into relevant parts of

the research process within the UoA and/or that research engagement is improving

 Limited• Little or no evidence through the indicators of research engagement within

the UoA.• Little or no evidence that research engagement is incorporated into the

research process or that research engagement activities are being developed.

Assessment of Engagement

• Will primarily assess qualitative information (impact studies) supplemented with quantitative information if available

• Test a selection of 6 FoRs:o 05 Environmental Scienceso 07 Agricultural and Veterinary Scienceso 09 Engineeringo 13 Educationo 19 Studies in Creative Arts and Writingo 20 Language, Communication and Culture

• Each institution may submit:o one impact study for each UoA (can be interdisciplinary)o an institutional interdisciplinary impact study (involve at least

one of the 6 chosen FoRs); ando an Indigenous research impact study (in any FoR)

• Institutions must nominate a primary FoR code for the impact study and up to two other relevant FoR codes – a percentage weighting must be apportioned to each code

Impact Pilot

• Will describe specific impacts that have occurred during the reference period 1 January 2011-31 December 2016, that were underpinned by the research undertaken by the submitting institution 1 January 2002 to 31 December 2016

• Each impact study must provide details of:o The institutions' approach to promoting the translation

of research into impacto The impact of the researcho The underlying research

• For the purposes of the pilot, the assessment of impact will focus on the institution’s approach to impact – i.e. the mechanisms used by institutions to promote or enable research impact

Impact Studies

Mature• Strong evidence of well-established institutional mechanisms supporting the

translation of the underlying research within the UoA into significant social, economic or environmental benefits.

• Strong evidence that the mechanisms for translating research are well integrated into the development and ongoing conduct of research within the UoA.

• Evidence that the underlying research has had a significant and identifiable impact beyond academia.

Emerging• Evidence that mechanisms were in place to encourage or promote the translation

of underlying research into social, economic or environmental benefits.• Evidence that the mechanisms for translation have been incorporated into

relevant parts of the research process within the UoA and/or that these are improving

• Evidence that the underlying research has had an identifiable impact beyond academia.

  Limited• Little or no evidence that the university has taken an active role within the UoA to

assist the translation of underlying research into social, economic or environmental benefits.

• Little or no evidence that the mechanisms for translation were incorporated into the research process with in the UoA or are being developed.

• Little or no evidence that the underlying research has had an identifiable impact beyond academia.

Assessment of Impact

• Co-ordinated and collective approach involving:o Central facilitieso Schools/Instituteso Libraryo Researchers/Research teams

• Education and Training:o Workshops, seminars, fora, guest lectureso Online training moduleso Roadshowso Competitions

• Identifying opportunities for collaboration with industry/business

• Industry placements and scholarships for students• Encouraging, recognising and rewarding positive behaviour in

planning, monitoring and evaluating impact• Showcasing the impact of our research through social media

WSU Approach to EI

Impact Framework

The intended or desired medium term effects /change expected to be realized from successful delivery of research outputs.It usually requires the collective effort of partners.

An effect on, change or benefit to the economy, society or environment, beyond those contributions to academic knowledge. The desired change at this level is through the collective effort of partners and outside the control of the research manager.

The research solutions, services, and/or capacities that result from the completion of activities within a research portfolio or project.

Resources applied to deliver activities, such as budget, people, equipment, etc.

Actions taken or work performed through which inputs, such as funds, technical assistance and other types of resources are mobilised with the intention of achieving specific outputs.

METHO

DOLOGY

DEFINITION

EXAM

PLES

Within your control

Intended resultsPlanned work

Outside your control

Uptake• Training accessed by users• Awareness of new research

protocols and techniques

Adoption• industry, government &/or

community usage• process changes implemented• behavioural change• sales of new products • licenses / IP sold

Economic impact• increased economic activity• higher quality workforce• productivity improvementEnvironmental impact• water savings• habitat rehabilitation• prevention of invasive species• reduced CO2 emissions Social impact• improved health & wellbeing• increased social cohesion • Improved quality of life

• Publications• prototypes• patents granted• training packages• students completed• new services• new/updated standards• reports

• staff FTE• non-staff FTE• appropriation funding• external funding• grants• in-kind contributions • equipment/facilities

• research/technology development• education• industry engagement (incl.

SMEs)• international engagement

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT

Engagement Feedback

The Three-Pronged Approach

Prospective Evaluation

Retrospective

EvaluationProgress

Evaluation

Evidencing ImpactStakeholder Information

Testimonials Online Traffic

Positions of Responsibility

• Annual reports from companies, government, NGOs, institutes

• Press coverage• Work cited in further

funding applications by community/voluntary groups

• Surveys• Dialogue through

public events• Newsletters• Community

meetings and minutes

• Company websites and press releases

• Citation by international charity and governments

• National statistics (ABS data)

• Letters of support

• One-to-one testimonials

• Focus groups

• Event feedback

• Personal letter from individual at third party organisations

• Surveys: Paper and online

• On-going testimonials from community party

• Comments on TV programmes/news articles and websites

• Social media website hits, tweets

• Steering group positions

• Secondments –offer letters

• Dialogue through public facing (recorded) events

• HANSARD

• Changing the research culture and driving behavioural change amongst academics

• Emphasising the focus on engagement and knowledge exchange/transfer

• Removing barriers to engagement and creating high quality research-industry partnerships

• Promoting an entrepreneurial culture in the University - with focus on contract and collaborative research (universities as agents of innovation) – not a threat to academic values

• Effective use of available resources (central and unit specific) to help our researchers make a real difference to the world

Opportunities and Challenges