Mapping the impact of MSF research: an evaluation framework and preliminary results
description
Transcript of Mapping the impact of MSF research: an evaluation framework and preliminary results
Mapping the impact of MSF research: an evaluation framework and preliminary results
Adam Kamenetzky1, Louise Bishop1, Sarah Venis1, Rafael Van den Bergh2 1Médecins Sans Frontières (MSF), London, UK; 2MSF, Brussels, Belgium
Background
“MSF’s research should improve patients’
lives and the effective delivery of medical
interventions. Does it?”
MSF’s research is rooted in its experience in
delivering humanitarian medical aid in resource-
limited and conflict settings.
In the past 20 years, MSF has published over 1,000
articles in peer-reviewed journals.1
Yet the contribution of MSF research to real-world
outcomes (e.g. programme and policy change) and
impacts (e.g. improvements in patient care) has not
been systematically analysed across the organisation.
Recently, MSF has defined a series of indicators
to assess the impact of its operational research.2
And UK charitable funders have joined together
to develop a practical framework with which
to evaluate medical research impact.3
We conducted a pilot study with the aim of
determining how best to capture and evaluate
the long-term impact of MSF research. Here
we present preliminary findings.
Methods
We used a mixed methods approach.
Quantitative information was sought via
a survey, using a series of pre-defined impact
indicators (A). Interviews with researchers
explored perceptions of their experiences,
and drew out underlying narratives (B).
A. Survey (indicators described by Zachariah et al.)2
B. Interviews (topic guide informed by focus group)
Results
Almost all respondents to the survey indicated
their work had been disseminated. Most also
reported contribution to policy change.
Few indicated their research had directly
improved MSF programmes or patient
outcomes.
Iterative analysis of interview transcripts
is currently underway.
Preliminary findings suggest:
• The importance and relevance of research to
patient care is not well understood in many MSF
project settings.
• There is uncertainty about what constitutes “good”
research, and the value of collecting data in a
consistent way to improve MSF’s operations.
• Documenting the real-world outcomes of research
as they happen is crucial for realistic and thorough
impact reporting and evaluation.
Conclusions
The workload associated with delivering the
questionnaire survey method was not felt to
be sustainable.
The range of research methods and topics
within MSF necessitates a more nuanced
approach.
Above: stuck in the mud? Innovation in how MSF shares research information within
and outside of the organisation could prevent valuable data becoming internally
siloed, or lost when field staff move on (photo w/ permission from Jen Jennings).
References 1 Bishop, L et al. Trends in MSF research publications. Poster: MSF UK
Scientific Day. 10 May 2013
2 Zachariah R et al. Is operational research delivering the goods? The
journey to success in low-income countries. Lancet Infect Dis. 2012
May;12(5):415-21
3 Association of Medical Research Charities: Tracking the impact of
charity research funding. 13 February 2013, available at: bit.ly/10Oz8Xv
Above: efforts to map the impact of MSF’s research must take into account the
complex network of associations that link different areas of its humanitarian work.
0
10
20
30
40
50
60
70
80
90
100
Met one or more criteria
for dissemination
Contributed to national /
international policy
Improved performance of
MSF programmes
Reduced patient
morbidity / mortality
% of projects responding to survey (n=27) reporting achievements against indicators
Left: snapshot of categories of questions
forming the basis of a standardised set of
medical research impact evaluation criteria,
currently being trialled in MSF via a pilot
of Researchfish – a system adopted by a
majority of UK medical research funders.
Attempts to map the long-
term impact of MSF’s
research must use
a comprehensive series
of indicators.
Both quantitative (metric)
and qualitative (narrative)
information is needed to
build a complete picture
of MSF’s research
achievements.
Efforts are underway
to evaluate whether a
standardised series
of outcome measures
can be used to document
MSF’s research impact
prospectively across
a cohort of MSF
researchers.
Advocacy
Impact? Publication
Operations
Policy
*approved by MSF/
national ethical review
boards in 2009
Dissemination
• Publication
• Interviews/workshops
• Presentations/media
Policy/practice
• Rapid advice/circular
• Guidelines
• Training materials
n=31*
projects principal
investigators
emailed survey;
self-reporting
purposively selected;
*at end April 2013
n=10*
interviewees
principal
investigators
telephoned;
semi-structured
interview
interviews transcribed;
themes identified through
an iterative and
deductive process