Tilburg University Research for policy Hegger, Ingrid · van mijn proefschrift Research for Policy...

Tilburg University

Research for policy

Hegger, Ingrid

Document version:Publisher's PDF, also known as Version of record

Publication date:2017

Link to publication

Citation for published version (APA):Hegger, I. (2017). Research for policy: A study on improving the contribution of scientific knowledge to evidence-informed health policy. Gildeprint.

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

- Users may download and print one copy of any publication from the public portal for the purpose of private study or research - You may not further distribute the material or use it for any profit-making activity or commercial gain - You may freely distribute the URL identifying the publication in the public portal

Take down policyIf you believe that this document breaches copyright, please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.

Download date: 19. Oct. 2020

https://research.tilburguniversity.edu/en/publications/2584ada7-1c66-4242-a74f-97df73c191f8

Research for Policy A

study on improving the contribution of scientific know

ledge to evidence-informed health policy

Ingrid Hegger

Research for PolicyA study on improving the contribution of

scientific knowledge to evidence-informed health policy

Ingrid Hegger

Evidence-informed health policy-making is generally considered as an important approach

for safeguarding public health: governments should take into account the best available

research evidence in health policy-making. Most countries have established a National

Public Health Institute to support their government in essential public health operations by

activities such as health protection, population health assessment and research to produce

evidence for policy-making. However, researchers experience that achieving contributions

to health policy-making appears to be more difficult than one would expect in view of the

institute’s mission. In the body of scientific literature on knowledge utilization, alignment

between researchers and policy-makers is recognized as an important key for enhancing

contributions of scientific knowledge to policy-making.

This thesis describes a study investigating how alignment is achieved and can be improved

in research projects conducted by a National Public Health Institute in commission of

governmental organizations.

Uitnodiging

voor het bijwonen van de openbare verdediging van mijn proefschrift

Research for PolicyA study on improving

the contribution of scientific knowledge to evidence-informed

health policy

op woensdag 6 september 2017 om 16:00 precies in de aula van de

Universiteit van Tilburg,Warandelaan 2 te Tilburg

Aansluitend bent u

van harte welkom opde receptie ter plaatse

Ingrid HeggerSoestdijkseweg Zuid 19

3732 HC De [email protected]

06-38464916

Paranimfen:Thomas Bakker

[email protected]

Esther [email protected]

De aula bevindt zich in het Cobbenhagengebouw,

te bereiken via het Koopmansgebouw. Bij de universiteit is voldoende

parkeergelegenheid. Station Tilburg Universiteit is op

10 minuten loopafstand.https://www.tilburguniversity.edu/nl/

contact/campus-map/

Research for Policy

A study on improving the contribution ofscientific knowledge to evidence-informed health policy

Ingrid Hegger

Colofon

The research for this thesis was supported by the Strategic Research Programme 2011-

2014 of the Dutch National Institute for Public Health and the Environment (RIVM) (project

number S/270206).

The printing was financially supported by Tilburg University and the National Institute for

Public Health and the Environment (RIVM).

ISBN

978-94-6233-673-5

Cover

‘Connections’ by Louise van Terheijden, Tilburg

http://louisevanterheijden.com/

Lay-out

Nicole Nijhuis - Gildeprint, Enschede

Printed by

Gildeprint - Enschede

Copyright ©Ingrid Hegger

All rights reserved. No part of this thesis may be reproduced, stored in a retrieval system or

transmitted in any form by any means without prior permission of the author.

Research for Policy

A study on improving the contribution ofscientific knowledge to evidence-informed health policy

Proefschriftter verkrijging van de graad van doctor aan Tilburg University

op gezag van de rector magnificus, prof.dr. E.H.L. Aarts,

in het openbaar te verdedigen ten overstaan van een

door het college voor promoties aangewezen commissie in de aula van de Universiteit

op woensdag 6 september 2017 om 16.00 uur

door

Ingrid Heggergeboren te ‘s-Gravenhage

Promotiecommissie

Promotores

Prof.dr.ing. J.A.M. van Oers

Prof.dr.ir. A.J. Schuit

Copromotor

Dr. S.W.J. Janssen

Overige leden

Prof.dr. L.A.M. van de Goor

Prof.dr.ir. M.W.J. Jansen

Prof.dr. M.A. van der Steen

Prof.dr. G.P. Westert

Dr. M.P.M. Bekker

Mijn proefschrift draag ik op aanmijn ouders, Hanna en Han, en mijn kinderen, Matthijs, Thomas en Esther.

Contents

Chapter 1 9

Introduction

Chapter 2 29

Thecomplexrelationshipbetweenresearchandhealthpolicy:acomprehensive

overviewoftheoreticalapproaches

Chapter 3 43

Analyzingthecontributionsofagovernment-commissionedresearchproject:

a case study

Chapter 4 75

Enhancingthecontributionsofresearchtohealthcarepolicy-making:

a case study on the Dutch Health Care Performance Report

Chapter 5 91

Contributionsofknowledgeproductstohealthpolicy:acasestudyonthe

Public Health Status and Forecasts Report 2010

Chapter 6 105

ResearchforPolicy(R4P):Developmentofareflectiontoolforresearchers

toimproveknowledgeutilization

Appendix Chapter 6: Research for Policy tool

Chapter 7 151

Discussion and Conclusion

Summary 179

Samenvatting 189

Dankwoord 201

About the author 205

ListofPublications(2011-2017) 207

Chapter 1

Introduction

R1R2R3R4R5R6R7R8R9

R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39

10 | Chapter 1

R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39

Introduction | 11

1Public health for a healthy population

‘The Netherlands healthy and well.’ These words represent the motto of the Ministry of

Health, Welfare and Sport in the Netherlands. On its website, the Ministry declares to have

the ambition of keeping everyone as healthy as possible, as long as possible. Furthermore,

the Ministry intends to support people with a disability and promote societal participation

[1]. This short ministerial mission statement reflects the Dutch implementation of the

worldwide responsibility for all national governments to maintain and enhance the health

of its population as defined in the Universal Declaration of Human Rights [2]. Guaranteeing

public health and maintaining effective health services is a demanding task with many

difficult issues and a major impact on the national budget, asking for wise national and

local policies. In order to ensure the population’s health, governments need to invest in

a health system comprising a public health infrastructure and a well performing health

care system. The World Health Organization (WHO) defined ten Essential Public Health

Operations (EPHOs) for the public health infrastructure to assure detecting, measuring, and

tackling health challenges through population-based measures [3]. The health care system

has to offer health services such as preventive, curative and palliative interventions directed

to individuals or to populations [4]. Many countries have established knowledge institutes

for providing support in these governmental tasks, the so-called National Public Health

Institutes (NPHIs) [5, 6].

Worldwide, NPHIs show a great variety in age, history, size, organization, legal basis,

mandate, autonomy, accountability, and financial resources [6]. Due to these differences, a

strict definition of a NPHI does not exist, but they all have a role in the EPHOs and work on

a number of Core Functions (as defined by the International Association of National Public

Health Institutes). These include activities such as population health assessment, health

protection and research to produce evidence [5,7]. NPHIs provide solicited and unsolicited

advice in all phases of public health policy-making. Their research findings support problem

definition and policy prioritization; they integrate knowledge and scientific evidence for

policy development and implementation; by performing health protection tasks, such

as disease surveillance, outbreak investigation and control, environmental monitoring

and managing environmental accidents, they are also involved in the implementation,

maintenance and supervision of public health policy. Finally, they conduct population health

and health systems assessment for policy evaluation [8].

R1R2R3R4R5R6R7R8R9


12 | Chapter 1

In the Netherlands, the National Institute for Public Health and the Environment (RIVM),

functions as a typical NPHI for already more than a century. RIVM has an important role in

the EPHOs during the entire policy cycle for public health and environmental health. RIVM

conducts research and knowledge synthesis in order to support national and international

governmental organizations in the field of public health, health care, medical products,

food and environmental health. Its operational tasks include population health monitoring,

prevention by vaccination, infectious disease management, and population screening [9].

Most NPHIs are (partly) financed by national government funding, whether or not

supplemented with other financial sources. As part of the Ministry of Health or being an

autonomous governmental organization, NPHIs often have to account to the Minister of

Health in some way. For policy advice issues, these characteristics can result in a position

of the NPHI on the nexus of science and policy, where both scientific quality and policy

relevance are at stake.

Research evidence for health policy-making

In recent years, the interest in contributions of knowledge to health policy in order to

enhance public health has grown, among scientists and policy-makers. Worldwide, these

contributions appear to be often more cumbersome than anticipated [10-15]. For example,

NPHIs produce many high quality public health reports, but effective ways of feeding this

knowledge into health policy are difficult to establish. The issue of how to improve scientific

knowledge contributions to the policy-making process has been the research subject of

an extended scientific field, which yielded an impressing amount of scientific publications

[9, 11, 15-23]. Initially, from 1965 onwards, research focused on innovation diffusion and

technology transfer [20]. From around 1985, the emerging Evidence Based Medicine

domain also resulted in increasing research on knowledge utilization in public health [15,

24, 25]. However, empirical studies on knowledge contributions to health policy describing

real-life situations as experienced by scientists and policy-makers are still limited. To find

ways for breaking down barriers in knowledge utilization, detailed insights are needed into

the processes of knowledge production and interaction between researchers and policy-

makers in everyday practice. In this thesis, we describe our study on how contributions of

knowledge from a NPHI to health policy-making are achieved. We have investigated how

these contributions can be enhanced by focusing on the processes involved in the complex

relationship between research and health policy-making and by taking into account the

continuously changing context of the research process.


Introduction | 13

1Science, research, knowledge and evidence

In the first part of this introduction, the terms science, research, knowledge and evidence

have already appeared without further clarification. These terms are easily used as synonyms,

but their meaning is different. To clarify the term science we refer to the definition by the

Science Council, UK: ‘Science is the pursuit and application of knowledge and understanding

of the natural and social world following a systematic methodology based on evidence’

[26]. In NPHIs, a significant proportion of the scientists have been educated in natural

sciences, in which a positivist view on science is dominant. From a positivist perspective,

science is considered a rational system creating objective, value-free facts by measuring

and observation. These facts reflect the objective single reality. However, for studying the

dynamics of knowledge utilization in policy-making, this perspective does not offer sufficient

understanding. In this thesis, we used the constructivist perspective that considers reality

not a single truth, but a social construct. In this view, science is considered a dynamic social

institution where norms, ideologies, practices, networks and power play a role [27,28].

Where science is the complete system, research is the systematic methodology to investigate

and explain phenomena. As put forward by Hanney, health research can roughly be divided

into basic, clinical and applied research [15]. Clinical research is basically conducted in a

health care setting outside knowledge institutes. A characteristic of basic research is

that the researchers mainly determine priorities; these research findings will generally

not directly be utilized in policy-making. In applied research, other stakeholders, such as

commissioners, governmental organizations and funding organizations, will generally

also play a part in determining the research agenda. Although one could expect that this

involvement will enhance scientific knowledge utilization, reality shows that this is not a

guarantee for research impact and utilization of the findings [15]. In this thesis, we focus

on applied research as conducted by NPHIs commissioned by governmental organizations.

We now come to knowledge, another broad concept closely related to science and research.

Knowledge encompasses all forms of knowing such as scientific findings, theories, data,

experiences and practical skills [17, 29]. From a constructivist perspective, knowledge is

a social construct depending on individuals and organizations involved and created when

people interpret and process different information, such as research findings, theories

and experiences. In this thesis, the term knowledge refers to scientific knowledge based

on research findings. Other types of knowledge are for example practical knowledge, local

knowledge and lay knowledge, but these are not included in this study [17]. In line with

R1R2R3R4R5R6R7R8R9


14 | Chapter 1

science as dynamic social institution, the interpretation and processing of information into

knowledge is also influenced by social processes [28]. Scientific knowledge is neither solely

determined by scientific facts nor value free, which makes it impossible to draw a sharp

line between scientific knowledge and non-scientific knowledge. This also implies that the

boundary between the systems science and policy is not predetermined and more difficult

to set than is (often implicitly) expected [8, 17, 30]. In Chapter 2, we further explore the

complex relationship between science and policy.

Although in scientific literature the term evidence often refers to evidence from scientific

research [31], evidence for policy-making can be different types of information next to

findings from research and scientific knowledge. As described by Bowen and Zwi, types of

evidence can be ideas, interests and information from politics and economics, anecdotal

evidence, knowledge and expertise of experts as well as lay persons, judgements, history,

analogies, local knowledge and culture (table 1.1) [24, 25]. Following the next explanation on

health policy, we return to the issue of evidence types in the section on evidence-informed

health policy.

Table 1.1 Types of evidence and how they are used in policy making [24]

Type of Evidence InformationandInfluenceonDecision-MakingResearch Empirical evidence from randomized control trials and other trials

AnalyticstudiessuchascohortorcasecontrolstudiesTime series analysesObservations,experiences,andcasereportsQualitativeresearchBeforeandafterstudies

Knowledgeandinformation Resultsofconsultationprocesseswithnetworks/groupsInternetPublisheddocuments/reports(includingpolicyevaluationsandstatisticalanalyses)

Ideas and interests Opinion and view- “expert knowledge” of individuals, groups, networks (shaped by pas personal and professional experiences, beliefs, values, skills)

Politics InformationrelevanttotheagendaofgovernmentOpportunityCrises

Economics FinanceandresourceimplicationsCosteffectivenessorotherformsofeconomicevaluationOpportunity cost

BowenS,ZwiAB(2005)Pathwaysto“Evidence-Informed”PolicyandPractice:AFrameworkforAction.PLOSMedicine2(7):e166.https://doi.org/10.1371/journal.pmed.0020166http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020166


Introduction | 15

1Policy, policy-makers, health policy and the policy process

In this part of the introduction, we explain how we interpret the concepts public policy,

policy-makers, health policy and the policy process.

In general, policy is a very broad concept related to the rational control and governance of

society; we focus on public policy made by government and governmental organizations as a

guide to action and for making choices on goals in a specified area and allocating resources

and capacities to achieve these goals in a specified timeframe [32, 33]. The goals of public

policy are inherently related to public interest and frequently aim to solve so-called ‘common

problems’ that comprise the conflict between public and private interest [34]. The term

policy-makers is used when we refer to the professionals of government / governmental

organizations, who develop policies, often under the authority of a political official [35].

In the context of this thesis, we limit health policy to public policy made by the Ministry of

Health or governmental organizations on an national level with the aim to impact positively

on population health and including both health care policy and public health policy [32].

Agenda setting

Policy formulation

Decision making

Policy implementation

Policy evaluation

Figure 1.1 Policy cycle

Public policy is established in very complex processes for which several different theories

exist nowadays [35, 36]. As Sabatier described, ‘in the process of public policy-making,

problems are and brought to government for solution; governmental institutions formulate

R1R2R3R4R5R6R7R8R9


16 | Chapter 1

alternatives and select policy solutions; and those solutions get implemented, evaluated and

revised’ [36]. The policy-making process is often represented as the policy cycle that includes

the steps agenda setting, policy formulation, decision making, policy implementation and

policy evaluation in a logical, rational sequence.

It should be noted that although the above presented straightforward and rational policy-

cycle provides a helpful overview, this model rather reflects the ideal policy-making process

[34]. In the complex reality of policy-making, the stages are linked but ideas, interests, actors

and value interact in a non-linear fashion [34]. As already recognized by Lindblom (1959),

policy-making has a complex and incremental nature, which he characterized as ‘muddling

through’ [37]. By taking incremental steps, policy-makers can manage the extremely complex

set of interacting elements that is involved in the policy-making process and avoid mistakes.

These characteristic elements include the involvement of many different actors with

different interests, the long lead-time to come from agenda setting to policy evaluation, the

simultaneous action on the same policy issue by different programs at different government

levels, the influence of very technical disputes and the determining role of power elements,

such as interests, money and authoritative coercion [36].

The complex reality of policy-making is important for understanding the use of scientific

knowledge in policy-making, where information, knowledge and evidence have a specific

role. In the political context, the interpretation of information is more powerful for

attaining political goals than the correctness of information, as explained by Stone [34].

For policy-makers, it is crucial to put efforts in strategically controlling the interpretation

of information. Some practices that could be regarded questionable in a scientific context

can be important tools of political strategy, such as selecting the most wanted information,

ignoring information, delaying publication of information and rephrasing information.

Framing of information in a narrative is an important tool for creating the necessity of a

proposed policy; a method that has a different logic than the linear scientific description of

a problem, its cause and solution.

Another relevant characteristic of policy-making is the role of numbers. To attain political

urgency for a problem, the best thing is to assign a number to it, making measuring and

monitoring ambiguous activities in a policy-making context [34]. Findings from measuring or

monitoring will influence the perception of a problem: does the problem (really) exist and to


Introduction | 17

1whatextent?Fromapoliticalpointofview,theoutcomecouldbeundesirable.Measuring

and monitoring can reveal (new) problems, which is not always politically opportune since

identification of a problem always raises the question of who has to account for it or even

who is to blame for it. This policy-making logic may conflict with the logic pursued by

scientists, who operate on the principle of accepting research findings as they are.

In next chapters, we will often refer to the complexity of the policy-making process and the

consequences for enhancing knowledge contributions.

From evidence-based medicine to evidence-informed health policy

Medicine and health policy-making are naturally closely related worlds influencing each

other. Evidence-informed health policy is a concept coming from Evidence Based Medicine

(EBM) that gained influence by the end of the 20th century [20]. EBM is ‘the conscientious,

explicit, and judicious use of current best evidence in making decisions about the care

of individual patients’ and integrates ‘the best external evidence with individual clinical

expertise and patients’ choice’ [31]. In this context, the best external evidence is considered

to be ‘clinical evidence from systematic research’ such as randomized controlled trials

(RCTs) [31]. EBM implies that the best clinical decisions need three types of knowledge:

research evidence, clinical experience and the patient’s preferences. In line with EBM, the

idea of evidence-based health policy (EBHP) came to the forefront over the past decade

as an important way to improve health systems performance worldwide [38]. In the EBHP

concept, robust research evidence should have an important role as basis for making and

justifying political choices. However, the problems in public health and health systems, such

as health inequalities or lifestyle consequences, are complex, though and persistent; they

are so-called ‘wicked problems’ Wicked problems are difficult to understand and describe,

since they have many causes, uncertainties and levels. They cannot be solved in a finite time-

period by applying straightforward approaches such as an RCT. Many possible outcomes of

a wicked problem exist and the desirability of an outcome depends on values and interests

making the concept of EBHP a wicked problem in turn [21, 39-41].

As we described above, health policy-makers have nevertheless to address these wicked

problems in an often unpredictable and rapidly changing political reality. They have to take

into account many different conflicting interests and next to scientific knowledge, other

knowledge types and considerations influence the policy decision as visualized by Armstrong

et al (see figure 1.2) [13, 24, 33].

R1R2R3R4R5R6R7R8R9


18 | Chapter 1

Know the place of evidence.

Rebecca Armstrong et al. J Public Health 2006;28:168-172

© The Author 2006, Published by Oxford University Press on behalf of Faculty of Public Health. All rights reserved.

Figure 1.2 Know the place of evidence; Armstrong et al (2006) [13]

Putting the role of research evidence in policy-making into perspective clarifies why the

term evidence-informed policy-making (EIPM) is nowadays preferred instead of the term

evidence-based policy-making [40]. Evidence-based could unjustly suggest that policy

decisions should be largely determined by research evidence preferably from RCTs [13,

24, 42]. According to Oxman et al, evidence-informed health policy-making is ‘an approach

to policy decisions that aims to ensure that decision making is well-informed by the best

available research evidence’ [42].

Role of national public health institutes in evidence-informed health policy

To act in accordance with the EIPM concept, policy-makers have to be able to find and

appraise research evidence and acknowledge its possible limitations [42]. NPHIs have

the task to produce knowledge based on research evidence in order to make research

evidence readily available for use in policy-making. Graham et al describe primary scientific

information as first-generation knowledge that is ‘in its natural state and largely unrefined,

like diamonds in the rough’ [29]. Next to first-generation knowledge, they also distinguish

second- and third-generation knowledge. Second-generation knowledge is created by using

systematic, reproducible methods to identify, appraise and synthesize relevant scientific

information into knowledge products such as systematic reviews [29]. For policy-makers, it


Introduction | 19

1is practically an impossible, time-consuming task if they have to turn to first-generation and

second-generation knowledge (of variable quality) outside their own expertise field. The

NPHI weighs and translates relevant information for the purpose of policy-making and the

knowledge is often presented in a way that specifically meets the needs of the users, such

as tools, websites, infographics, reports or other knowledge products. These knowledge

products represent third-generation knowledge and aim at improving knowledge utilization.

Third-generation knowledge could also intend to influence the behavior of stakeholders by

providing decision tools, guidelines or recommendations [29].

We now reach the point where the inherent tension between usefulness of knowledge

in policy-making and the border between science and policy-making emerges. The more

a knowledge product contains political sensitive findings and recommendations on how

to translate the knowledge into action, the more it comes close to policy-making. As we

explained before, knowledge and research findings have a complex role in policy-making

determined by the actual political context. Such a close-to-policy advice could very well

meet the policy-maker’s need if in line with the actual political view, but could just as well

give rise to concerns about political interference by the NPHI, particularly in case it does

not fit in. The NPHI’s role depends on its legal position, authority and distance to the policy-

making process and its staff has to be aware of their role. A NPHI being part of the Ministry

of Health has to act in a different way than an independent agency not accountable to

the Ministry. Researchers should know what the respective roles are of their institute and

the government department, what their own role should be, how to handle uncertainty

in expertise and how to interact with the policy-makers involved, for example by installing

advisory committees for commissioned research. In order to prevent troubling situations,

NPHIs researchers should also be aware of possible barriers in research utilization that

may relate to expectations placed at the start of the research, the (in)effective transfer of

research findings and the acceptance and interpretation of research findings by individual

policy-makers [11]. They have to negotiate the boundary of their advice by finding the

right balance in wording, format and presentation [43]. This also implies that achieving

optimal knowledge utilization requires a two-way process that includes the policy-makers’

involvement as well [25]. Joint efforts will create knowledge products representing a

serviceable truth, a concept defined by Jasanoff, which means ‘a state of knowledge that

satisfies tests of scientific acceptability and supports reasoned decision making, but also

assures those exposed to risk that their interests have not been sacrificed on the altar of an

impossible scientific certainty’ [44].

R1R2R3R4R5R6R7R8R9


20 | Chapter 1

For researchers at NPHIs, it is very satisfactory when contributions of knowledge to policy-

making are smoothly attained and become clearly visible, specifically when the outside

world recognizes the role of knowledge as basis for the policy choices. However, even

contributions of carefully balanced, third-generation knowledge to policy-making often

appear not to be that straightforward raising the question of what researchers can do to

enhance contributions.

When we consider the differences in the role of information, knowledge, numbers and

measurements in the policy-making context compared to the scientific context and the

need to address wicked problems, the complexity of providing relevant research evidence

for policy-making emerges. In fact, researchers and policy-makers have to align the political

context and the scientific findings [25, 28]. According to Kok and Schuit (2012), the term

alignment, a state of being aligned, ‘emphasizes that accommodation can take place on

the side of research and/or on the side of action, instead of a one-way research to action

dynamic’ [28]. In our study, the side of action was the policy-making context of the Ministry

of Health. We consider alignment as a state in which all actors involved consider the

knowledge products acceptably balanced to the policy-making needs. This balance can

be reached after deliberately bringing together the research context and political context

and by accommodation on either side. Alignment increases the likelihood that beneficial

contributions to health policy can be obtained [28]. Kok and Schuit (2012) propose to call

the specific anticipatory efforts to enhance the contributions to policy-making alignment

efforts [28]. In our study, we also distinguish alignment areas, meaning subject areas where

alignment efforts may focus on.

Despite the large reservoir of scientific literature on the policy-making process, the role of

knowledge in policy-making and knowledge utilization, it remains difficult to identify what

researchers (and policy-makers) can do to enhance contributions. In this study, we intended

to gain knowledge that affords us a glance behind the scenes of this problem. In the specific

context of an NPHI producing knowledge for policy advice in commission of and sponsored

by the government, we investigated contributions of knowledge produced by RIVM to

national health policy-making in the Netherlands.


Introduction | 21

1Aim of the study

The aim of our study was to acquire insights in how knowledge institutes such as NPHIs

can enhance the contributions to evidence-informed health policy of knowledge produced

in commission of governmental organizations. To investigate this issue, we formulated the

following research questions:

1. What alignment areas are important for enhancing the contributions to health policy

of government commissioned research commissioned by National Public Health

Institutes?

2. Based on the empirical findings gained in this study, what is a practical approach to

enhance knowledge contributions from NPHIs to evidence-informed health policy-

makingatnationallevel?

These research questions rendered specific objectives for this study:

• To map in detail the process of three RIVM projects, the alignment efforts in the

projects and the projects’ contributions to health policy-making at national level;

• To identify the most decisive alignment areas for knowledge contributions to

health policy-making;

• To prepare and evaluate a draft tool based on the theoretical framework and

empirical findings for supporting researchers in alignment.

Study design and outline of the thesis

Foransweringexploratoryquestionssuchas“Whyisavailableknowledge(not)used?”or

“Whatfactorsplayaroleinknowledgeutilization?”qualitativeresearchisoftenseenasthe

most suitable research design, because it pays appropriate attention to the contemporary

and contextual conditions in relation to the topic under research. For this study, we choose

an approach that offered insights into the processes of research projects at the level of daily

practice. Therefore, the study-design was organized along the case study methodology [45].

We selected three cases of research projects according to the following criteria:

• The research projects jointly cover the scope of the RIVM health domain, i.e. public

health, health care and health products, such as medicines, medical devices and

consumer products;

• From the commissioning organization’s perspective, the research projects deal

with sensitive policy issues;

R1R2R3R4R5R6R7R8R9


22 | Chapter 1

• The research projects render third generation knowledge products typical for a

NPHI institute;

• The research projects have different status and size

The first case in the multiple case study was a new project commissioned by the Health

Care Inspectorate in the domain of health products. The project Risk Model started in

2007 and was relatively small in terms of budget and number of researchers involved. The

commissioner’s question was to develop a risk-based approach for stratified selection of

high-risk clinical trials. The Inspectorate intended to implement a risk–based selection

of inspection objects and the risk model aimed to facilitate the selection of the high-risk

trials from the large number of clinical trials conducted every year in the Netherlands. As

an inspection tool, the risk model developed had to remain confidential and could not be

published by RIVM. Although the project started as a one-year project, the Inspectorate

continued the commission in following years.

The second case was the development of the Dutch Health Care Performance Report 2010

(DHCPR) during the period 2008-2010. The Dutch Ministry of Health, Welfare and Sport

commissioned this long-term health care performance monitor initially biannually (editions

in 2006, 2008 and 2010). The last edition was published in 2014, four years after the 2010

edition. The DHCPR had the aim to contribute to strategic health care policy-making, but

was not officially embedded in the policy cycle of the Ministry. Both team and budget were

substantial and many stakeholders outside RIVM were involved. The DHCPR was considered

one of the RIVM flagship-products in the health care domain and was sent to parliament for

informing the members of parliament.

The third case was the production of the Public Health Status Forecasting Report 2010

(PHSF). The PHSF is a major flagship-product and is officially embedded in the policy cycle

by law. The Directorate Public Health of the Ministry of Health, Welfare and Sport acts as

commissioner, but the intended users are all policy-makers of the Ministry. RIVM produces

an edition of the PHSF report every 4 years since 1993. The project team is extended and

cooperates with several other research institutes. The PHSF provides the policy themes for

the legally obliged National Health Memorandum, which is published every four years by

the Minister of Health.


Introduction | 23

1For analyzing the data collected in the case studies, the Contribution Mapping approach

developed by Kok and Schuit offered a useful model and method to analyze the processes

[28]. Contribution Mapping conceptualizes the utilization and impact of knowledge through

so-called contributions, which are activities that enable the conversion of knowledge into a

component of policies and practices or in innovation. This concept reflects that knowledge

conversion into policy-making can take many, even very subtle, shapes and forms, already

during the research process. For the realization of meaningful contributions, Kok and Schuit

consider specific actions, called alignment efforts, essential to attune research and use. The

qualitative nature of the method and detailed analysis provided in depth understanding of

the alignment processes (or lack of alignment) in research projects and the contributions of

the created knowledge to health policy-making.

In Chapter 2, we provide a concise overview of the theoretical background for the complex

relationship between research and health policy. It describes an influential typology of

knowledge use and successive theoretical models on knowledge utilization [11, 46].

Chapter 3 describes the case study ‘Risk Model’. It contains an outline of Contribution

Mapping as theoretical framework and method for analysis [28]. We analyzed the process

of an RIVM project in which risk ranking models for clinical trials were developed in co-

creation with and in commission of the Health Care Inspectorate. This case study provided

information on important categories of alignment efforts.

In a second case study (Chapter 4), we analyzed the developmental process of the Dutch

Health Care Performance Report 2010 (DHCPR 2010). The DHCPR monitors health care

performance in the Netherlands by using indicators for quality, accessibility and affordability.

This study focused specifically on the process at project level of RIVM researchers and

policy-makers of the Ministry of Health. The study revealed several areas where alignment

is relevant for enhancing the contributions of future reports.

Chapter 5 describes the case study Public Health Status and Forecast Report 2010 (PHSF

2010). The PHSF integrates research data and identifies future trends in public health in the

Netherlands. Since the report has a formalized position in the national public-health-policy

cycle and policy-makers at the Ministry of Health, Welfare and Sport are considered its key

users, PHSF is considered an example of successful boundary work in literature. This study

R1R2R3R4R5R6R7R8R9


24 | Chapter 1

revealed insights on alignment issues in projects that already include several alignment

efforts.

In Chapter 6, the findings of the three case studies are integrated for identifying the most

decisive areas for alignment. The development of the practical tool Research for Policy tool

(R4P tool), based on the decisive alignment areas and intended for researchers, is described.

The R4P tool is presented as an Annex to this chapter.

In Chapter 7, the study as a whole is discussed including a reflection on the methods applied.

Conclusions are drawn in relation to the initial research questions and are completed with

recommendations on practical follow-up and further research opportunities.

Chapter 2 to 6 have been written as articles for international scientific journals. They can be

read independently from each other. However, this also implies that overlap between these

articles exists in theoretical background on the one hand and that divergences in format and

lay-out occur on the other hand.


Introduction | 25

1References1. Rijksoverheid. Ministerie van Volksgezondheid, Welzijn en Sport, Organisatie. Available from:

https://www.rijksoverheid.nl/ministeries/ministerie-van-volksgezondheid-welzijn-en-sport/inhoud/organisatie. Accessed 25 May 2016.

2. United Nations. The Universal Declaration of Human Rights. 1948. Available from: http://www.un.org/en/universal-declaration-human-rights/. Accessed 28 May 2016.

3. WHO. The 10 Essential Public Health Operations. Available from: http://www.euro.who.int/en/health-topics/Health-systems/public-health-services/policy/the-10-essential-public-health-operations. Accessed 10 August 2016.

4. WHO. The World Health Report 2000. Health systems: improving performance. Geneva: World Health Organization; 2000.

5. Binder S, Adigun L, Dusenbury C, Greenspan A, Tanhuanpaa, P. National Public Health Institutes: Contributing to the Public Good. J Public Health Policy. 2008;29(1):3-21.

6. Lomas J. Connecting research and policy. Isuma: Can J Policy Research. 2000;1(1):140-144.7. IANPHI. Framework for the Creation and Development of National Public Health Institutes.

IANPHI; 2007.8. Wehrens R. Beyond two communities -from research utilization and knowledge translation to

co-production?PublicHealth.2014;128(6):545-551.9. De Leeuw E, McNess A, Crisp B, Stagnitti K. Theoretical reflections on the nexus between

research, policy and practice. Critical Public Health. 2008; 18(1):5-20.10. Jansen M, van Oers H, Kok G, de Vries N. Public health: disconnections between policy, practice

and research. Health Res Policy Syst. 2010;8(1):37.11. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriers

between epidemiological research and local health policy development. Health Res Policy Syst. 2010;8:26.

12. Brownson RC, Royer C, Ewing R, McBride TD. Researchers and Policymakers: Travelers in Parallel Universes. Am J Prev Med. 2006;30(2):164-72.

13. Armstrong R, Doyle J, Lamb C, Waters E. Multi-sectoral health promotion and public health: the role of evidence. J Public Health. 2006;28(2):168-72.

14. Black N. Evidence based policy: proceed with care. BMJ. 2001;323(7307):275-8.15. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-

making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.16. Mitton C, Adair CE, McKenzie E, Patten SB, Perry BW. Knowledge transfer and exchange: review

and synthesis of the literature. Milbank Q. 2007;85(4):729-68.17. Sedlačko M, Staroňová K. An Overview of Discourses on Knowledge in Policy: Thinking

Knowledge, Policy and Conflict Together. Cent Eur J Public Health. 2016;9(2):44.18. Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis

of the literature. Health Res Policy Syst. 2014;12(1):34.19. Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The Use of Research

Evidence in Public Health Decision Making Processes: Systematic Review. PLoS ONE. 2011;6(7):e21704.

20. Estabrooks CA, Derksen L, Winther C, Lavis JN, Scott SD, Wallin L, Profetto-McGrath J. The intellectual structure and substance of the knowledge utilization field: A longitudinal author co-citation analysis, 1945 to 2004. Implement Sci. 2008(3):49.

21. Keijsers JFEM, Paulussen TGWM, Peters LWH, Fleuren MAH, Lammers F. Better use of knowledge: Information behaviour of national policy makers. (In Dutch: Kennis beter benutten: Informatiegedrag van nationale beleidsmakers). Woerden: NIGZ and TNO; 2005.

R1R2R3R4R5R6R7R8R9


26 | Chapter 1

22. Landry R, Amara N, Pablos-Mendes A, Shademani R, Gold I. The knowledge-value chain: A conceptual framework for knowledge translation in health. Bull World Health Organ. 2006;84(8):597-602.

23. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239-244.

24. Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS medicine. 2005;2(7):e166.

25. Strydom WF, Funke N, Nienaber S, Nortje K, Steyn M. Evidence-based policymaking: A review. S Afr J Sci 2010;106(5/6):249.

26. Science Council United Kingdom. Definition of science. Available from: http://sciencecouncil.org/about-us/our-definition-of-science/. Accessed 15 May 2016.

27. Miller C. Hybrid Management: Boundary Organizations, Science Policy, and Environmental Governance in the Climate Regime. Sci Technol Human Values. 2001;26(4):478-500.

28. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10:21.

29. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation:timeforamap?JContinEducHealthProf.2006;26:13-24.

30. Van Egmond S, Bekker M, Bal R, van der Grinten T. Connecting evidence and policy: bringing researchers and policy makers together for effective evidence-based health policy in the Netherlands: a case study. Evid Policy. 2011;7(1):25-39.

31. Sackett DL, Rosenberg W, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71-2.

32. De Leeuw E, Clavier C, Breton E. Health policy-why research it and how: health political science. Health Res Policy Syst. 2014;12:55.

33. Smith KE, Katikireddi SV. A glossary of theories for understanding policymaking. J Epidemiol Community Health. 2013; 67(2): 198-202.

34. Stone DA. Policy paradox: The art of political decision making. New York Norton & Company Ltd; 2002.

35. Buse K, Mays N, Walt G. Making Health Policy. 2nd ed. Maidenhead: Mc Graw Hill Open University Press; 2012.

36. Sabatier PA. Theories of the Policy Process. Boulder, CO: Westview Press; 2007.37. Lindblom C. The science of “muddling through”. Publ Admin Rev. 1959;19:79.38. Dobrow MJ, Goel V, Upshur RE. Evidence-based health policy: context and utilisation. Socl Sci

Med. 2004;58(1):207-217.39. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion Theory, and Knowledge Dissemination,

Utilization, and Integration in Public Health. Annu Rev Public Health. 2009; 30:151-174.40. Lomas J, Brown A. Research and advice giving: a functional view of evidence-informed policy

advice in a Canadian ministry of health. Milbank Q. 2009;87:903-926.41. Pielke RA, Jr. The honest broker: Making sense of science in policy and politics. Cambridge:

Cambridge University Press; 2007.42. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health

Policymaking(STP)1:Whatisevidence-informedpolicymaking?HealthResPolicySyst.2009;7Suppl 1:S1.

43. Halffman W, Hoppe R. Science/policy boundaries: a changing division of labour in Dutch expert policyadvice.In:MaasseS,WeingartP,editors.Democratizationofexpertise?Exploringnovelforms of scientific advice in political decision-making. Dordrecht: Kluwer; 2005. p. 135-152.

44. Jasanoff S. The fifth branch: Science advisers as policymakers: Harvard University Press; 1994.


Introduction | 27

145. Yin RK. Case Study Research Design and Methods. 4th rev ed. Thousand Oaks: SAGE Publications

Inc; 2008.46. WeissC.Policyresearch:data,ideasorarguments?In:WagnerCHWP,WittrockB,WollmanH,

editors. Social Sciences and modern states. Cambridge: Cambridge University Press; 1991. p. 307–332.

Chapter 2

The complex relationship between research and health

policy: a concise overview of theoretical approaches

Submitted for publication

Ingrid Hegger

Lisanne K. Marks

Roland A. Bal

Hans A.M. van Oers

R1R2R3R4R5R6R7R8R9


30 | Chapter 2

Abstract

It often appears difficult to realize the integration of scientific knowledge and research

evidence into health policy; a problem which has been subject of extended scientific research

in social sciences for several decades. This article provides a comprehensive overview of the

different theoretical approaches in the complex relationship between science and policy-

making. It describes successive theoretical models on knowledge utilization to inform

researchers who intend to contribute to health policy about their own position in this

complex constellation.

The influential typology of knowledge use provided by Weiss makes a division between

instrumental, conceptual and symbolic use of knowledge. Positivist, interactive and

constructivist models have been developed to explain actual knowledge use and to identify

barriers and enhancers to improve the knowledge utilization. All models offer some part

of understanding about what is going on in the relationship between science and policy.

Although knowledge utilization will remain complex, awareness of the insights from

theoretical models on knowledge utilization in policy-making is a first step and may support

researchers from other scientific disciplines in their efforts to enhance the integration of

evidence into to health policy.


The complex relationship between research and health policy | 31

2

Introduction

Researchers producing knowledge to support policy-makers have to deal with the complex

relationship between science and health policy. In this paper, we intend to provide these

researchers with a concise overview of the different theoretical approaches to make them

aware of their own position in this relationship and to provide them with insights that can

be of help in cross-disciplinary co-creation processes and in making research relevant and

usable for policy-makers. After all, the awareness of researchers about the sensitive social

issues arising at the nexus of science and policy has been found a decisive factor for the

impact of research in policy-making [1-5].

Different perspectives on knowledge utilization

When aiming at integration of scientific evidence into health policy, researchers should

realize that different perspectives on the use of scientific knowledge in policy-making exist,

depending on the concepts for knowledge production, policy processes and knowledge use

taken as starting points [6].

Knowledge production

In traditional knowledge production, scientists produce knowledge within their own

discipline. Their focus is on science and knowledge in the first place and not necessarily on

the usability of the knowledge produced.

Gibbons et al. described this as ‘Mode 1 knowledge production’ [7]. In the last decades,

Mode 1 knowledge production moved to ‘Mode 2 knowledge production’ in which academics

from different disciplines co-create knowledge together with other partners in a multi-

disciplinary way [7].

Typology of knowledge use

Even if one considers ‘knowledge’ as a clearly defined item, such as an article, it still can

be used in different ways. The most known typology of knowledge use in policy is the one

provided by Weiss that makes a division between research as deliverer of data, ideas and

arguments [8]. This is often described as instrumental, conceptual and symbolic use of

knowledge respectively [9, 10].

R1R2R3R4R5R6R7R8R9


32 | Chapter 2

Instrumental use means that knowledge is considered as technical, objective data and

findings that are utilized in a direct and specific way to solve a particular problem [11].

Instrumental use of knowledge is mostly present in situations in which there is consensus,

both in science and in policy [8].

Research is used in a conceptual way when it delivers ideas that ‘enlighten’ policy-makers

[11]. From the original research, a more or less ‘layman story’ remains which can change

people’s perceptions about problems and can frame those problems for the policy agenda.

This indirect way of use can be observed in cases of high uncertainty about a problem or in

the early stages of a policy discussion [8].

Research influences policy in a symbolic way when it is used as arguments to take an advocacy

position. In this case, politicians and policy-makers may use research data selectively and

strategically to support or strengthen their own position [8]. The knowledge is not used to

inform decision-making, but to justify a pre-chosen position (political use) or to legitimize

(in)action (tactical use) [11]. This type of research use can be observed in conflict situations

[8].

In addition to the typology of Weiss, agenda-setting is sometimes named as a fourth way of

research impact for cases where the research findings itself gives rise to social or political

debate [12]. This however shows large overlap with the conceptual function of knowledge

use.

Modelsforknowledgeutilization

Most people will recognize the different types of knowledge use as described by Weiss from

their own experience. From this typology, several questions naturally follow, such as ‘What

factorsdeterminetheactualknowledgeuseandhowcanweinfluencethe(typeof)use?’.To

handle the complex area of knowledge utilization, successive theoretical models have been

developed which may help to find an answer to these questions. Some of these models—

more specifically the constructivist ones—also challenge the distinction made by Weiss. In

the next sections, we give a brief outline of three approaches: positivist, interactive and

constructivist models.



2

Positivist approach models

Positivism is an approach of reality that assumes the existence of a reality driven by natural

laws and mechanisms and of research studying objects without influencing them [13].

Traditional or positivist models on knowledge utilization assume two separate worlds of

science and policy with a strict boundary in between and a one-way relationship [14]. The

positivist perspective regards knowledge as a product for use in policy-making at a certain

point in time. Consequently, these models take the boundaries of both domains as a starting

point to improve knowledge utilization and to look for solutions that try to bridge the so-

called ‘gap’ between these worlds. (Figure 2.1)

Science Policy-making

Figure 2.1 The gap between the world of science and policy-making

A first example of a positivist model is the classic knowledge-driven (or push) model. In this

model, there is a sequence from research to policy utilization with the assumption that

knowledge existence automatically presses to its use [15, 16]. Besides attributes of the

research content, such as complexity and validity, and the type of research, such as method

and domain, high quality research automatically leads to better use by policy makers

according to this model [17].

Another example of a positivist model is the problem solving (or demand pull) model.

Instead of the researchers, the initiative to start research is now shifted to the policy-makers

[17]. Policy-makers identify a particular problem and ask researchers to help formulate

alternatives and solutions to make a policy choice [15].

These rational positivist models mainly focus on the instrumental use of research and

correspond with a technocratic approach to policy-making where decisions unfold through

different sequential stages as part of a policy cycle. (Figure 2.2)

R1R2R3R4R5R6R7R8R9


34 | Chapter 2

Agenda setting

Policy formulation

Decision making

Policy implementation

Policy evaluation

Figure 2.2 Policy cycle

After identifying the problem (agenda setting), policy-makers gather and consider data on

all possible solutions or alternatives relevant for the public interest (policy formulation).

Thereafter, they select a solution depending on predefined goals (decision making). After

the solution has been put into practice (policy implementation), the results are monitored

(policy evaluation) and adjusted when necessary [18]. Assuming that research evidence

easily finds its way to policy-makers and offers ‘neutral’ and ‘objective’ facts, this rational

policy cycle also reflects the different possible roles for research that offers objective and

value-free facts.

However, this model of a one-way policy cycle has been criticized for not corresponding with

the complex and incremental nature of policy-making including many different stakeholders

with their own interests and preferences [19]. In an incremental model of policy-making,

research has a weaker contribution and in comparison with the rational policy cycle,

symbolic use of research is more likely.

Furthermore, the rational science-policy models have been criticized for not attaching much

importance to interaction between producers and users of knowledge and not taking into



2

account the organizational and social contexts that may influence the uptake and production

of research. The science-policy model that researchers have (often implicitly) in their mind is

often the rational model. It may be helpful for them to learn about alternative models and

the implications of these models for their own work if taken as a starting point.

Interactive approach models

In response to the drawbacks of the rational models, interactive models of science-

policy relations pay extra attention to context, communication and interaction between

researchers and policy-makers. ‘Socio-organizational explanations’ of knowledge utilization

include organizational interest explanations, ‘two communities’ explanations and interaction

explanations [20]. According to the organizational interest explanation, researchers have to

pay specific attention to the knowledge users’ needs, which depend on their policy domain

or organization. In these models, knowledge use increases when policy-makers consider the

research findings relevant, credible, timely and fitting in with their needs [21].

A model that gained a lot of influence is the ‘two communities’ model [10, 22, 23]. This

model explains the lack of research use by policy-makers as a consequence of a cultural gap

in norms and values between science and policy; both researchers and policy-makers have

their own language and different ways of communication.

Although maybe difficult to achieve, effective interaction to better understand each

other’s worlds is a starting point for increasing the use of research by policy-makers

[22]. The ‘linkage and exchange’ model extended this view by focusing on the broader

institutional and political context and acknowledges both research and policy-making to

be ‘processes’ [23, 24]. Consequently, the use of research by policy-makers is higher when

the research supports an existing policy or corresponds with the ideological environment,

values or interest [10]. In this perspective, adaptation from both researchers and policy-

makers, is necessary to increase research use [9]. For instance, researchers can invest in

the presentation of the research (more readable and understandable) and present specific

(policy) recommendations. At the same time, users can try to influence the topics on the

research agenda in order to get research outcomes that fit with their needs [17].

The interaction model further bridges the gap between the ‘two communities’ by considering

the process of knowledge use as a disorderly and iterative set of interactions between the

R1R2R3R4R5R6R7R8R9


36 | Chapter 2

users and researchers [16, 21]. Relationships between researchers and policy-makers at

different stages of the research process (from formulation to interpretation) have priority

in this model and more investment in different types of linkage mechanisms result in higher

use of research [21]. To analyze knowledge utilization in local policy-making, De Goede et

al. for example developed a model that consists of a policy network and a research network

and identified four categories of barriers in knowledge utilization: expectation, transfer,

acceptance and interpretation [17]. (Figure 2.3) In their model, both networks may overlap

when, for example, policy-makers get involved into the research process by formulating

research questions or researchers in the policy process by communicating or presenting

their results.

Barriers:Expectation, Transfer, Acceptance, Interpretation

Policy network Research network

Research Use

Context

Figure 2.3 Interaction model de Goede [17]

Constructivist approach models

Following the line of knowledge utilization typology via successive models to the current view

that interaction between researchers and policy-makers is a precondition for knowledge

utilization in policy-making, the question is how to design this interaction in order to reach

optimal knowledge utilization. The interaction and network models already show a growing

awareness of the importance of interaction and context. However, these models still assume

an a priori separation of science and policy domains and search for tools and structures to

bridge differences, but the demarcation between science and policy-making is often not

predetermined and as sharp as supposed [1].



2

Another way to look at knowledge utilization is the constructivist view that emphasizes the

co-construction of knowledge in a social process and considers boundaries between research

and policy as the outcome of negotiation processes between researchers and policy-makers

[1, 25, 26]. The idea of separate worlds is replaced by the concept of ‘hybrids’ where policy-

makers and researchers co-create knowledge in mutual interaction. In this view, science is

not value-free and only producing facts but is, instead, another social institution with its own

social norms and practices and no conclusive boundaries. In the constructivist perspective,

the notion of knowledge use is problematized, as the positions of producer and user of

knowledge are seen as intertwined.

A constructivist way to look at the relationship between science and policy is represented by

the concept of ‘boundary work’ conducted by scientists and policy makers. Boundary work

is about work done by actors to demarcate science from non-science. Although people use

the concept ‘science’ easily in everyday life, it is in fact impossible to demarcate science in

an unambiguous way. To handle this problem, Gieryn formulated the concept of boundary

work that has to be understood as ‘creating a social boundary that distinguishes science

from intellectual activities that are non-science’ [27]. By conducting boundary work, both

researchers and policy-makers negotiate the boundary between science and policy-making,

where science may come close to policy-making, but not too close [28].

Successful co-creation of knowledge heavily depends on the way the co-creation process

is organized. In this respect, organizations located on the boundary between science and

politics having characteristics of both domains and involving both scientists and policy-

makers in knowledge production, play an important role and are sometimes described as

boundary organizations [29]. Although the boundary between science and policy-making

will continuously be the subject of discussion within such a boundary organization, a stable

boundary can be presented to the outside world.

To provide a further understanding of the interaction between research and policy practices

in boundary organizations, the concepts of frontstage and backstage work by Goffman

(1990) are very helpful [25, 28, 30]. Backstage work stands for the invisible (to outsiders)

coordination process behind the scenes in a knowledge producing process and includes

the knowledge development and learning processes that (invisibly for outsiders) take

place in interaction between the user (policy-maker) and producer (researcher). It is the

R1R2R3R4R5R6R7R8R9


38 | Chapter 2

negotiation process and the search for consensus backstage, including informal contacts

between researchers and policy-makers, that creates trust and makes knowledge (products)

‘acceptable’ at the frontstage [25].

Frontstage work can be described as the way policy-makers and researchers show their

work and work processes to the public [25]. At the frontstage, they present a clear division

of labour and their separated responsibilities. Here, science and policy are seen as two

separate domains with their own tasks; science is depicted as objective and neutral to be

used for politics to make decisions. This image fits the rational and positivist view of policy-

making and science, which is often still a prevalent way to describe the relation between

science and policy. It is at the frontstage where the research (process) gains external

legitimacy and accountability (as being ‘evidence based’), but it is at the backstage where

knowledge utilization is determined. (Figure 2.4)

Knowledge product

Governmental organization

Knowledge production

Research Institute

Front Stage Back Stage

Figure 2.4 Backstage and Front Stage in Knowledge Production

To clarify the constructivist approach, we provide an example.

Commissioned by the Ministry of Health, a public health institute conducts a research on

the implications of nanomaterial use for human health. For the integration of their findings,

they initiate an advisory group in which experts from local health services and national



2

policy-makers participate. Both researchers and experts feel the need to include policy

recommendations in the final report. However, it becomes clear that some recommendations

are politically controversial and unacceptable for the policy-makers who argue that proposing

political solutions is their job. The questioned policy recommendations would undermine

the acceptance of the report by the Ministry. After exciting negotiations, the group agrees

on reformulated recommendations that maintain the research findings but leave room for

an adequate political response by the Ministry. The director of the public health institute

officially presents the final report to the Minister in presence of the press.

Putting theory into practice

The theoretical models described so far in this article provide useful insights into the

complexity of the science- policy relation, but do not offer many practical leads yet to apply

at the level of research projects. To fill this gap, researchers have to find approaches that

fit their needs while taking into account the theoretical insights into knowledge utilization.

Different theoretical approaches resulted in different practices to improve knowledge

utilization. Knowledge translation including activities as research priority setting, knowledge

distribution and evaluation of uptake is a commonly-used positivist approach [31]. From

the interaction perspective, knowledge brokers linking researchers and knowledge users

act as valuable intermediates to bridge the gap between the two worlds. Furthermore, a

constructivist approach taking into account the complexity of the knowledge production

process offers clear understanding of how to improve alignment between researchers

and knowledge users. For example, Kok and Schuit developed the Contribution Mapping

approach for monitoring research projects [20]. This approach based on constructivism

brings up the moments and areas where specific alignment between knowledge producers

and knowledge users may improve research contributions to policy and practice [3, 4].

Conclusion

In this paper, we provided an overview of theoretical models for knowledge utilization in

health policy-making. All models offer some part of understanding about what is going on in

the relationship between science and policy and it is up to researchers to take advantage of

these models and concepts in aligning with policy-makers.

By acknowledging the complexity of knowledge production as a social process and the need

for intensive interaction with potential users during a research process, a difficult task is

R1R2R3R4R5R6R7R8R9


40 | Chapter 2

added to the responsibilities of researchers. In everyday work, they probably will need to

put flesh on the theoretical concepts described above.

The first step is to become aware of the complexity of science-policy interactions and

the possible ways to approach this complexity. Perhaps, alignment may seem an obvious

precondition to achieve contributions, but in fact it often proves to be very difficult to

achieve [3-5, 17, 32].

The next step that researchers can take to enhance the contributions of their work is to

evaluate their research processes in a structured way and to identify areas for improving

alignment with policy-makers. This takes time and capacity, but renders valuable input

for future projects. In this process, researchers should not hesitate to question their own

normative position and to reflect on the limits of scientific knowledge. As Jasanoff argued,

it is important to take a humble position and to recognize that other information might also

be highly relevant for taking a well-informed and just decision in policy-making [33].

Finally, we advise researchers to take time for regular reflection on their research processes

to stay aware of their role in knowledge utilization and to develop routines for enhancing

the contributions of their work. Like many good habits, this asks for self-discipline, but may

yield much profit.



2

References1. Wehrens R. Beyond two communities -from research utilization and knowledge translation to

co-production?PublicHealth.,2014;128(6):545-551.2. Bal R, Bijker WE, Hendriks R. Paradox van Wetenschappelijk Gezag: over de maatschappelijke

invloed van adviezen van de Gezondheidsraad. Den Haag: Gezondheidsraad, 2002.3. Hegger I, Janssen SWJ, Keijsers JFEM, Schuit AJ, van Oers, HAM. Analyzing the contributions of

a government-commissioned research project: a case study. Health Res Policy Syst. 2014; 12(1): 8.

4. Hegger I, Marks LK, Janssen SWJ, Schuit AJ, van Oers HAM. Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report. J Health Serv Res Policy. 2016; 21(1): 29-35.

5. Hegger I, Kok MO, Janssen SWJ, Schuit AJ, van Oers HAM. Contributions of knowledge products to health policy: a case study on the Public Health Status and Forecasts Report 2010. Eur JPublic Health. 2016; 26(6): 922-927.

6. Sedlačko M, Staroňová K. An Overview of Discourses on Knowledge in Policy: ThinkingKnowledge, Policy and Conflict Together. Cent Eur J Public Health. 2016; 9(2): 10-31.

7. Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M. The new production of knowledge: the dynamics of science and research in contemporary societies. London: Sage; 1994.

8. WeissC.Policyresearch:data,ideasorarguments?In:WagnerCHWP,WittrockB,WollmanH,editors. Social Sciences and modern states. Cambridge: Cambridge University Press; 1991. P. 307–332

9. Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication, 2004. 26(1): 75-106.

10. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002; 7(4): 239-244.

11. Lavis, JN, Robertson D, Woodside JM, McLeod CB, Abelson J; Knowledge Transfer Study Group. How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers? Milbank Q. 2003; 81(2): 221- 248.

12. Bekkers V, Fenger M, Homburg V, Putters K. Doorwerking van strategische beleidsadvisering. Erasmus Universiteit Rotterdam en Universiteit van Tilburg. 2004.

13. Carpiano RM, Daley DM. A guide and glossary on postpositivist theory building for population health. J Epidemiol Community Health. 2006; 60(7):564-570.

14. Gibbons M. Science’s new social contract with society. Nature. 1999. 402(SUPP).15. Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979; 39(5): 426-431.16. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-

making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003; 1(1): 2.17. deGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriers

between epidemiological research and local health policy development. Health Res Policy Syst. 2010; 8(26).

18. Howlett M, Ramesh M. Studying Public Policy: Policy cycles and Policy Subsystems. 1995. Toronto: Oxford University Press.

19. Lindblom CE. The science of “muddling through”. Public Adm Rev. 1959; 19 (2): 79-88.20. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research

to enhance its impact. Health Res Policy Syst. 2012; 10: 21.

R1R2R3R4R5R6R7R8R9


42 | Chapter 2

21. Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Adm Rev. 2003; 63(2): 192-205.

22. Caplan N. The two-communities theory and knowledge utilization. Am Behav Sci. 1979; 22(3): 459-470.

23. Lomas J. Research and evidence–based decision making. Aust N Z J Public Health. 1997; 21(5): 439-441.

24. Lomas J. Connecting research and policy. Isuma: Can J Policy Research. 2000; 1(1): 140-144.25. Bekker M, van Egmond S, Wehrens R, Putters K, Bal R. Linking research and policy in Dutch

healthcare: infrastructure, innovations and impacts. Evid Policy. 2010; 6(2): 237-253.26. Jasanoff S. The fifth branch: Science advisers as policymakers. Cambridge, MA: Harvard

University Press; 1990.27. Gieryn TF. Boundary-work and the demarcation of science from non-science: Strains and

interests in professional ideologies of scientists. Am Sociol Rev. 1983; 48(6) 781-795.28. Bijker WE, Bal R, Hendriks R. The paradox of scientific authority: the role of scientific advice in

democracies. Cambridge, MA: MIT Press; 200929. Guston DH. Stabilizing the boundary between US politics and science: The role of the Office of

Technology Transfer as a boundary organization. Soc Stud Science. 1999; 29(1): 87-111.30. Goffman E. The presentation of self in everyday life. London: Penguin; 1990.31. De Leeuw E, McNess A, Crisp B, Stagnitti K. Theoretical reflections on the nexus between

research, policy and practice. Critical Public Health. 2008; 18(1): 5-20.32. De Goede J, Steenkamer B, Treurniet H, Putters K, van Oers H. Public health knowledge utilisation

by policy actors: an evaluation study in Midden-Holland, the Netherlands. Evid Policy. 2011; 7(1): 7-24.

33. Jasanoff, S., Technologies of Humility: Citizen Participation in Governing Science. Minerva. 2003; 41(3): 223-244.

Chapter 3

Analyzing the contributions of a government-

commissioned research project: a case study

Published: Health Research Policy and Systems 2014, 12 (1):8

Ingrid Hegger

Susan W.J. Janssen

Jolanda F.M. Keijsers

Albertine J. Schuit

Hans A.M. van Oers

R1R2R3R4R5R6R7R8R9


44 | Chapter 3

Abstract

Background

It often remains unclear to investigators how their research contributes to the work of the

commissioner. We initiated the ‘Risk Model’ case study to gain insight into how a Dutch

National Institute for Public Health and the Environment (RIVM) project and its knowledge

products contribute to the commissioner’s work, the commissioner being the Health Care

Inspectorate. We aimed to identify the alignment efforts that influenced the research project

contributions. Based on the literature, we expected interaction between investigators and

key users to be the most determining factor for the contributions of a research project.

Methods

In this qualitative case study, we analyzed the alignment efforts and contributions in the Risk

Model project by means of document analysis and interviews according to the evaluation

method Contribution Mapping. Furthermore, a map of the research process was drafted and

a feedback session was organized. After the feedback session with stakeholders discussing

the findings, we completed the case study report.

Results

Both organizations had divergent views on the ownership of the research product and the

relationship between RIVM and the Inspectorate, which resulted in different expectations.

The RIVM considered the use of the risk models to be problematic, but the inspectors had

a positive opinion about its contributions. Investigators, inspectors, and managers were

not aware of these remarkably different perceptions. In this research project, we identified

six relevant categories of both horizontal alignment efforts (between investigators and key

users) as well as vertical alignment efforts (within own organization) that influenced the

contributions to the Inspectorate’s work.

Conclusions

Relevant alignment efforts influencing the contributions of the project became manifest at

three levels: the first level directly relates to the project, the second to the organizational

environment, and the third to the formal and historical relationship between the

organizations. Both external and internal alignments influence the contributions of a

research project. Based on the findings, we recommend that research institutes invest

in a reflective attitude towards the social aspects of research projects at all levels of the

organization and develop alignment strategies to enhance the contributions of research.


Case study Risk Model | 45

3

Background

For knowledge institutes such as the Dutch National Institute for Public Health and the

Environment (RIVM), it is important to know what factors influence the impact of their

research. This article aims to give insight into the process of a government-commissioned

RIVM research project and the relevant alignment efforts needed to enhance its contributions

to the commissioner’s work.

The RIVM is an independent knowledge institute with expertise in the fields of public

health, infectious diseases, health care, medicines, lifestyle, nutrition, and environmental

safety. Being a governmental institution, the RIVM conducts research on behalf of other

governmental organizations to support them in their policy-making and supervisory tasks.

One of the RIVM’s principal contracting agencies is the Health Care Inspectorate (hereafter:

Inspectorate), which supervises the quality of health services, prevention measures, and

medical products in the Netherlands. In a yearly program cycle, the Inspectorate submits

knowledge questions to be answered by research conducted by the RIVM (Additional

file 1 Yearly cycle for RIVM research in commission of the Health Care Inspectorate).

For commissions to the RIVM, the Minister of Health puts a dedicated budget at the

Inspectorate’s disposal, which means that the Inspectorate cannot use this budget for

other purposes. Although RIVM investigators and inspectors interact during all phases of

the research project, the Inspectorate, as the commissioning body, does not have authority

over the research methods used, nor the outcome of studies as is laid down in the Act on

the RIVM [1].

These days, the Inspectorate has to account for the effectiveness of its supervisory methods

[2]. In its long-range plans, the Inspectorate expresses its objective to develop and use

scientific knowledge for evidence-based supervision [3, 4]. Consequently, research projects

commissioned to the RIVM are often intended to contribute to the scientific basis of the

Inspectorate’s work.

An evidence-based approach assumes that use of knowledge will lead to better professional

practices resulting in both better and more legitimate outcomes [5, 6]. This assumption

strengthens the investigator’s expectations with regard to how research products will be

used: investigators will produce scientific knowledge for the professionals, who will be

eager to use it to improve their professional practices. However, this expectation often does

R1R2R3R4R5R6R7R8R9


46 | Chapter 3

not come true: utilization of research findings is difficult to achieve in practice. By now,

the widely recognized difficulties in actual contributions of research have led to extensive

research on knowledge utilization in health care and public health. Knowledge institutes

use the insights from this research field in specific strategies to enhance the likelihood that

their research will indeed contribute to improvements in practice; we call these strategies

alignment efforts [7]. The RIVM also recognized the importance of alignment efforts and

established them into procedures and guidelines on an institutional level. Some examples

of such official alignment efforts are periodic meetings between managers of the RIVM and

the commissioning organizations, training of project leaders where they learn to keep in

touch with the commissioner’s contact person, and guidelines for the periodic progress

reports and final RIVM products.

The RIVM’s aim is to support the Inspectorate in its supervisory tasks and tries to enhance

its contribution to the Inspectorate’s work by specific alignment efforts. However, it often

remains unclear for RIVM investigators to what extent their research really contributes and

how its contributions could be improved. As a part of an RIVM strategic research project on

improving knowledge utilization, we initiated a case study to explore how a RIVM project

proceeded [8].

The objective of this study was to gain insight into how an RIVM project, including its

knowledge products, contributed to the Inspectorate’s work and to identify the alignment

efforts in the research project that influenced the research process and its contributions. In

this article, we first present our study framework, the method used, and our findings in a

‘map’ of the RIVM research project. We describe the research project’s contributions and

the alignment efforts that influenced the project. We finally reflect on the method used,

especially on the feedback session with inspectors, investigators, and managers, as well

as on the study limitations. In our conclusions, we provide also suggestions for improving

research projects and its contributions.

Study framework

The starting point for our study was the awareness that interaction between investigators

and the intended users of research is generally considered the decisive factor to enhance

the contribution of research. Based on the Interaction Model for research utilization in

the field of health policy-making, de Goede et al. [9] developed an analytical framework



3

that consists of a distinct policy network and a distinct research network. In the overlap of

these networks, interaction takes place between the policy network and research network

promoting research utilization (Figure 3.1).

In a way, this model also describes the context of RIVM research for the Inspectorate:

both are distinct organizations with their own networks and they overlap at different

levels. The two networks ‘RIVM’ and ‘Inspectorate’ specifically overlap in the research

projects commissioned by the Inspectorate. Both organizations recognize the importance

of interaction during the research project for the contributions of a research project as

reflected in institutionalized alignment efforts.

Barriers:Expectation, Transfer, Acceptance, Interpretation

Policy network Research network

Research Use

Context

Figure 3. 1 The interaction model by de Goede et al. (2010) [9]. De Goede et al. visualized the distinct research and the policy networks as partially overlapping circles. In the Interaction Model, policy-making is an interactive and incremental process and extensive interaction between investigators and policy-makers promotes research utilization. In the overlap of these networks, interaction takes place between the policy network and research network promoting research utilization.

Although the Interaction Model by the Goede et al. [9] is an adequate model to describe

the overall relationship between the RIVM and Inspectorate organizations, this model did

not offer a method to zoom in on the overlap of the two networks to analyze and explain

the course of a research project. We needed a model for the complex research process

with many different actors on different organizational levels evolving in a certain period.

Contribution Mapping (CM), recently formulated by Kok and Schuit [7], offers a useful

framework for our study, taking into account the social context of a research project and

contains a three-phase process model of the research process. CM also includes a practical

method for analysis, which we applied to our case study.

R1R2R3R4R5R6R7R8R9


48 | Chapter 3

During the research process, the research network and policy network form a hybrid world

(Figure 3.2). Research is a process of co-creation where actors are, on the one hand, the

investigators who perform the research and, on the other, the linked actors who in some

way are connected to the project. Investigators and linked actors go through a joint process

of co-creation in a research project. In this process of knowledge production, the three-

phase process model distinguishes three demarcated phases in which typical activities take

place:

1. Formulation phase: activities to initiate the research process, including funding,

prioritization, and commissioning;

2. Production phase: activities to realize the knowledge products;

3. Knowledge extension phase: activities to make the knowledge available to potential

users and to initiate and stimulate utilization.

It is difficult to assess the final impact of research on action for health and to measure the

actual use of research products. A direct link from a research product to improved practices

is often hard to trace. To handle this attribution problem, Kok and Schuit introduced the

concept of contribution to action: “Contribution to refers to the activities which turn novel

combination of knowledge into a ‘going concern’, as part of practices, a component of

successful innovation, as an element in decision making and their implementation” [7]. All

phases of the research process can be relevant for contributions to health practices and Kok

and Schuit distinguished four categories of research-related contributions:

1. Changes in abilities and actions of involved and linked actors due to the research

activities;

2. The contributed knowledge products as such;

3. Contributions through the utilization of the produced knowledge by investigators

and linked actors;

4. Contributions through utilization at distance by non-linked actors.



3

Figure 3.2 Three Phase Model by Kok and Schuit [7]

In research projects, actors are usually aware of problems in research transmission and

utilization. They therefore design alignment efforts; during all phases of the research

process, both investigators and linked actors undertake specific actions to enhance

the desired contributions of the research process and the likelihood that the research

indeed contributes and that the research products are used. Actors have their own actor-

scenario which may have (decisive) influence on the process in all phases and thus on the

contributions of the research process, on the knowledge product and/or utilization of the

knowledge product. In practice, actors are not only involved in one research project, but

are part of their own network and organizational environment and their actor-scenario

continuously evolves. Kok and Schuit describe knowledge utilization as “incorporating

knowledge into an actor-scenario as a means of contributing to its strength and scope” [7].

For incorporation into an actor-scenario, research knowledge has to be translated into a form

and content that fits the actor-scenario. Alignment efforts will be most effective to enhance

contributions of research knowledge when investigators and involved actors recognize and

take account of the different actor-scenarios. Based on the three-phase process model, Kok

and Schuit developed a method to systematically evaluate the complex research process

and the contributions of the research to action. According to this CM evaluation method,

the investigators of a research project and the linked actors, such as potential key-users,

are interviewed after an initial document analysis. Based on the findings, a map of the

R1R2R3R4R5R6R7R8R9


50 | Chapter 3

research process is drafted taking alignment efforts and an estimation of the contributions

into account. In a feedback session, the draft map is presented to relevant stakeholders to

validate the findings and to identify inconsistencies. Next, the contribution map is finalized

and used for learning, improvement, and accountability purposes [7].

In our case study, we used the three-phase process model of CM as a study framework

and we used the CM evaluation method to investigate the RIVM project ‘Risk Model’,

commissioned by the Inspectorate, and its contributions. We specifically focused on the

interaction between actors, especially the investigators and key users, using the theoretical

concept of CM that alignment is the precondition for the contributions of a research project.

Alignment efforts to create alignment are important to enhance its contributions; Kok and

Schuit described various alignment efforts such as “engaging potential users in setting

research priorities, formulating research and interpreting results, employing double-role

actors in research and disseminating research results to potential key-users” [7]. In our case

study, we inductively identified the alignment efforts that appeared to be relevant for the

RIVM-Inspectorate relationship.

Methods

To explore the contributions of a RIVM project to the work of the Inspectorate, we used a

qualitative case study approach, including document analysis and semi-structured interviews

[10]. This case study concerns the Risk Model project, a small project with a limited number

of involved actors commissioned by the Inspectorate during three consecutive years.

Contribution map

For the document analysis, we manually analyzed documents from the project archives

within the RIVM and relevant public documents of the Inspectorate in order to reconstruct

the course of events with respect to the project. We interviewed actors from RIVM and

the Inspectorate involved in theproject (n=10),namely the investigators (RIVMproject

leader and another RIVM investigator), the key users (four inspectors), and two RIVM

managers and two managers from the Inspectorate as relevant informants. We used a topic

list consisting of items regarding the research process, the actors and their organizational

environment, the interaction between actors, and the contributions in the three phases of

knowledge production, all based on the CM evaluation method. An additional file shows

this in more detail (Additional file 2 Topic List interviews actors Case Study Risk Model). The



3

RIVM principal investigator, who was also involved in the Risk Model project, conducted the

semi-structured interviews face to face. Each interview lasted approximately one hour and

was recorded and transcribed verbatim.

After importation of the data into Atlas-ti version 6.2.25, the principal investigator analyzed

the data by coding [11]. We based the code list initially on the conceptual framework and

completed it with inductive codes. The coding of the interviews started while data collection

was ongoing and the topic list for the interviews was iteratively adapted in order to achieve

data saturation. Three other investigators independently coded two interviews and we

validated the coding in an open-coding session (agreement 72%; response 70%).

Based on the documentary analysis and interview data, we constructed a map of the

research project according to the method CM by assessing the course and contributions of

the project [7].

Feedback session

Next, we organized a feedback session with two inspectors, a manager from the Inspectorate,

the RIVM project leader, and a RIVM manager, to present and discuss our findings. All those

present gave their feedback on the draft assessment and discussed actions for improving

contributions and interaction. This discussion was recorded and transcribed verbatim. Based

on the findings and the outcome of the feedback session, we drafted the final version of the

contribution map and we inductively identified six categories of relevant alignment efforts.

Results: the research process and its contributions

In this section, we present the course of the research process based on the data collected

by documentary analysis and the interviews. We identified contributions in three

categories: the utilization of the produced knowledge by inspectors, investigators and their

organizations (contributions through linked utilization), the project’s knowledge products

(contributed knowledge products), and changes in abilities and actions by inspectors,

investigators and their organizations due to the research project (change in involved and

linked actors). Contributions through utilization at-a-distance, the fourth contribution

category distinguished by Kok and Schuit, did not emerge due to the confidential nature

of the knowledge products [7]. In brackets, we name the identified contributions and the

occasions where we observed the presence of alignment efforts.

R1R2R3R4R5R6R7R8R9


52 | Chapter 3

Background of the project

The Risk Model project lasted for three consecutive years with an extension in the fourth year

(2007–2010; Figure 3.3). According to the yearly cycle for RIVM research, the Inspectorate

submitted a knowledge question for every project year to which the RIVM made a

corresponding offer. For every project year, we distinguish the three phases according to

CM:

1. The period for submitting the knowledge question and writing the project plan

in an RIVM offer (from July until December; in the preceding year; formulation

phase);

2. The period for performing the planned research activities (from January until the

delivery of all agreed knowledge products; production phase);

3. The period after delivery of agreed knowledge products (extension phase).

The project aimed to develop a risk-based approach for clinical trial inspections in the

Netherlands and had to deliver risk models to enable ranking and stratified selection of

clinical trials for inspection. The research activities included the identification of risk

factors and risk indicators and the development of two risk models: one for clinical trials

on medicinal products (1st year) and a second for clinical trials on food substances (3rd

year). We consider the models contributed knowledge products according to CM. To create

these knowledge products, a synthesis of scientific knowledge on risk-ranking, risk models,

medicinal products, and food substances was made and combined with legal and practical

knowledge on clinical trials and inspections. This resulted into a new risk-based approach to

clinical trials. For this project, the RIVM performed desk research in close collaboration with

inspectors of the Inspectorate, without involvement of other organizations.



3

KQ 1 Development model 1

2010200820072006 2009

Delivery risk model 1 Use of risk model 1

Deliveryrisk model 2

(no use)

Implementation risk model 1Development risk model 2

Formulation Phase : formulation of knowledge question (KQ)

Production Phase

Extension Phase

KQ 2 Sleep mode

KQ 3

Project cycle 1

Project cycle 2

Project cycle 3

Figure 3.3 Timeline for the risk model project. Timelines of the Risk Model project visualizing the formulation, production, and extension phase for the three project cycles. In the formulation phases, the knowledge questions (KQ) were: 1) the development of risk model 1 (KQ 1), 2) the implementation of risk model 1 (KQ 2; postponed to project cycle 3), and 3) the implementation of risk model 1 and the development of risk model 2 (KQ 3). In the production phases, the following activities took place: 1) the development of risk model 1 (production phase 1), 2) no activities, project in sleep mode (production phase 2), and 3) the implementation of risk model 1 and the development of risk model 2 (production phase 3). Risk model 1 has been used in extension phase 1, risk model 2 has not been implemented or used.

Development of the first risk model

During the first formulation phase, the RIVM project leader and the contact person of the

Inspectorate agreed to focus on clinical trials on medicinal products (alignment efforts;

Formulation Phase 1).

Although in line with the official policy plan of the Inspectorate, the inspectors emphasized

that it was not the driving force behind their research question; they felt the need for a

tool for risk-based selection of inspection objects because of the overwhelming number of

clinical trials they had to supervise (about 1,800 approved protocols each year). Although the

inspectors were of the opinion that they had the expertise and competencies themselves,

they had no time to develop a method for risk-based selection. They formulated an official

RIVM knowledge question including distinct ideas for the approach of the problem and the

design of a tool.

R1R2R3R4R5R6R7R8R9


54 | Chapter 3

Inspectorate Manager 1: “The knowledge question came up because of the enormous

amount of inspection objects and the small number of inspectors involved. They tried

to find a sensible way for risk-based selection.”

At the RIVM, the origin of the research question was less clear. Some RIVM respondents

thought the Inspectorate’s managers ordered the risk-based approach because of the

official policy.

Researcher 1: “I think that the knowledge question came from higher quarters, because

of the general wish to have more risk-based supervision, just because of the general

principle and to have a model for it.”

After the formal start of the first production phase, it took several consultations to refine

the research question and to agree upon the way to go (alignment efforts; Production

Phase 1). This exploring phase in the production phase took longer than foreseen by the

RIVM investigators in the project plan and caused delay. The data accessibility appeared to

be much more problematic than anticipated, because the Inspectorate could not directly

access the clinical trial database that is in control of another governmental organization,

which largely hindered the development of the risk model.

The research team consisted of three RIVM investigators, who cooperated closely with

three inspectors. During meetings, they discussed the scientific approach of risk ranking,

risk factors, indicators, and draft versions of the risk model (alignment efforts). The project

resulted in an electronic risk model and a report on its development (contributed knowledge

product).

The transfer of the knowledge product by mail was without any project-specific dissemination

activities and six months later than originally planned (Extension Phase 1). After transfer, the

inspectors mentioned to the investigators that their management criticized the risk model.

However, the inspectors internally solved the issue and the nature of the criticism never

became clear to the investigators. In the interviews, the Inspectorate’s managers expressed

some critical remarks on the scientific approach of the risk model but also mentioned

that the inspectors involved were contented with the model and that this was decisive for

accepting the knowledge product.



3

Researcher 1: “And I became aware that manager 2 at the Inspectorate had a negative

opinion and we asked for more explanation, but we never received it.”

Since the risk model was considered an internal confidential inspection tool, the Inspectorate

requested to keep the model confidential. The RIVM agreed although it meant that they

could not further publish it nor use it in scientific discussions on risk-based supervision

(alignment efforts).

On their side, the inspectors did not disseminate the risk model, neither in their external

network due to its confidential status nor within the Inspectorate, because they were of the

opinion that the risk model was too specific for other disciplines within the Inspectorate.

The only occasion they exchanged the risk model was during the discussion with another

governmental organization to get access to the clinical trial database (contributions

through linked utilization). Nevertheless, they shared their experiences with the model in

international discussions on risk-based inspections (contributions through linked utilization).

The Inspectorate also mentioned the existence of the risk models in public presentations, in

publications, such as annual reports, and in a formal answer to questions of the Parliament

(contributions through linked utilization).

Project in ‘sleep mode’

During the second formulation phase (that coincided with the first production phase), the

Inspectorate formulated a new knowledge question: their intention was to incorporate

the risk model in their operating procedures and they asked for RIVM assistance in the

implementation (Formulation Phase 2).

After the official start of the second production phase, it eventually became clear that the

inspectors could not realize the implementation of the risk model due to shortage of staff at

the Inspectorate. Consequently, they did not need RIVM-assistance and the project went into

‘sleep mode’ (Production Phase 2). In fact, the yearly cycle was broken due to unforeseen

circumstances and the Inspectorate officially postponed this knowledge question to the

next (third) year.

R1R2R3R4R5R6R7R8R9


56 | Chapter 3

Implementation and a second risk model

For the third year, the knowledge question included both assistance at the implementation

of the risk model and the development of a second risk model for another category of clinical

trials (Formulation Phase 3). In their offer, the RIVM investigators did not yet define the

category of clinical trials for this second model, but mentioned two options as an example.

They stipulated that the choice for the new category should depend on the availability of

appropriate data after an inventory of available data.

An RIVM investigator assisted at the implementation of the risk model for clinical trials on

medicinal products and adapted the original model in order to facilitate data import into the

model (alignment efforts; Production Phase 3). In June, a short report was presented to the

Inspectorate on the progress, including the adapted electronic risk-ranking tool for clinical

trials on medicinal products (contributed knowledge product).

For the second risk model, the investigators and inspectors initially disagreed on the

category of clinical trials. Due to a serious incident in a Dutch hospital in the previous

year, the inspectors wanted clinical trials on food substances to be the focus of the second

model. The investigators preferred clinical trials on medical devices based on the analogy to

medicinal products, more available data and the match with their own expertise. However,

the Inspectorate’s reality is that they also have to take into account the public and political

circumstances when setting priorities in supervision and they felt the need to develop a risk

model for a category that was less obvious for the researchers. This caused some tension

in the project between investigators and inspectors, which was eventually resolved by

discussion (alignment efforts).

The development of the new risk model for clinical trials on food substances needed extra

unforeseen time and was not delivered according to the project plan but in the following

(fourth) year (contributed knowledge product). During the finalization phase, only the RIVM

project leader was involved and occasionally consulting other investigators. The model was

again confidential and not published on the RIVM website. The second risk-model on clinical

trials with food substances was never used (Extension Phase 3).



3

Interviewer: “What is your opinion about the second model, did you use it?”

Inspector 3: “No, I cannot say anything about that because we did not use it until now.”

Interviewer: “Could you indicate what the reason is?” Inspector 3: “Mainly because of

competition with other tasks or other inspections. There were other issues with priority:

international inspections and the thematic inspection project. And because the priority

of the second category inspection objects decreased, the incident was dealt with and

the interest in these inspection objects diminished.”

Sequel to the project

The investigators and inspectors had differing views on the desired sequel to the project

which uncovered differing views on the ownership of the knowledge products. On one hand,

the investigators regarded the risk models as their own knowledge products or at least co-

creations with co-ownership of both parties. They expected that further development of the

risk models would also evolve in co-creation by RIVM and Inspectorate and that they would

spontaneously receive feedback on user experiences with the model.

Researcher 1: “…I have the opinion that they should exchange this kind of information

spontaneously.”

The inspectors presented their risk ranking experiences in EU expert meetings, but did not

provide feedback to the investigators; they were not aware that international discussions

on a risk-based approach were of high interest to the investigators. They regarded the

risk models as their own products and they choose not to involve the investigators in

the evaluation and further validation of the models. The confidentiality of the products

conflicted with the investigators’ interest to publish the reports and to use the findings in

international discussions. Although both RIVM management and the investigators agreed

with the required confidentiality, these differing expectations were discussed neither at the

start nor at the end of the project causing unarticulated tension.

Appraisal of contributions

All inspectors were rather satisfied with the contributions of the project to their work. In

their opinion, they had used the risk models as much as possible and they expected further

use, validation, and development of the models in the future (alignment efforts). However,

the need for linking the risk models directly to the clinical trial database became gradually

R1R2R3R4R5R6R7R8R9


58 | Chapter 3

evident after transfer, but access to the database remained problematic. Without linkage to

the database, the risk models turned out to be too laborious. Nevertheless, the inspectors

were determined to move on with the models in coming years.

Inspector 2: “Yes, I feel positive about it. Especially inspector 3 used it and in that sense

I have the impression that the worksheets, the Excel sheets, function properly.”

The Inspectorate’s managers were positive about the project afterwards, merely because

the inspectors as users of the models were positive (alignment efforts). However, they

showed some reserve since the risk models could only be in full use if connected to the

clinical trial database. Given that this feature was not yet realized, the risk models had not

taken away the managers’ concerns about the supervision of clinical trials. They still felt

the need to have ways to predict specific, yet unknown risks in order to be able to prevent

incidents and they doubted whether the current feature of the risk models was suitable for

this purpose.

Inspectorate manager 2: “We should think more about what we want to know and

what we want to do with it at the start. We consequently do nothing with the report.

Although this was not a bad example after all, since the inspectors felt contented about

it. But it can be improved.”

On the other hand, the RIVM investigators assessed the contributions of the project to

the Inspectorate’s work as minor. After transfer, they were insufficiently acquainted with

the implementation problems at the Inspectorate. They had expected that the risk models

would be used on a daily basis within two or three years.

Researcher 1: “I expected that we would establish a model able to scan all clinical trials

within a maximum period of two, three years.”

Interviewer: “For all trials?”

Researcher 1: “For all trials, yes, but to achieve that more thinking, more communication

was needed and it had to be done faster. But it did not work out that way…”

At the site of the RIVM, the project contributed to the implementation of their strategy

to extend the research area of the centers specialized in the quality control of medicinal



3

products. The required expertise on risk ranking was on the fringe of the traditional expertise

of the RIVM centers involved and the project offered the opportunity for developing new

expertise (contribution ‘change in involved and linked actors’). However, this contribution

was hindered by the confidentiality of the risk models.

Consultative structure

In this study, we paid special attention to the interaction in all phases of the research project

and we also found that all interviewed persons in this case study recognized the importance

of their interaction to enhance the contributions of the research project. We asked them

to indicate which factors could improve the process and contributions, and they all

spontaneously indicated that the relationship and interaction like meetings and continuous

consultations are crucial for success in a research project.

During the phase of formulation, the investigators and inspectors had only limited

interaction. The formal procedure agreed between the Inspectorate and RIVM does not

facilitate in-depth discussion on the knowledge question and the project plan due to the

very formal deliberations at management level, short time periods, and strict deadlines

within the procedure.

Inspectorate Manager 2: “The more the question is vague, the more RIVM is not able to

make a proper offer. If it is too unarticulated, the RIVM should not accept the question.

What would the offer be based upon, which expertise and how many research hours?

We should have more interaction.”

During the production phases, the consultations went on in a good atmosphere according to

both parties. In meetings, the inspectors and investigators openly discussed the risk factors

and drafts of the risk model (alignment efforts).

Inspector 4: “It was according to an established pattern. We discussed what we as

inspectors thought about the results so far and then the RIVM got the work again to

improve (the drafts).”

The frequency of the consultations between RIVM and the Inspectorate depended much on

proactive organization, availability of persons, and priorities at the Inspectorate (alignment

R1R2R3R4R5R6R7R8R9


60 | Chapter 3

efforts). The interaction between investigators and inspectors often suffered from time

constraints and the inspectors and investigators experienced this in a different way.

The inspectors considered the frequency of interaction overall rather high and satisfactory,

especially in the first year. The investigators emphasized that the lack of time of the

inspectors to give feedback on drafts of the risk model resulted in delay. Although the

inspectors regretted that their own time constraints delayed the evaluation and further

development of the model, they accepted this fact of life.

Inspector 2: “At the start, I think the intensity was very adequate, meetings on fixed

time points. It is my feeling that there was need for it. The meeting dates were planned,

but only used if needed. At a certain point in time, my role changed and I did not

take part in everything anymore and therefore it is difficult for me to assess. It is my

impression that it was less frequent for the extra model; however, this was the second

project and things were clear already. So, although less frequent, it was sufficient in

relation to the need.”

Interviewer: “What is your view on the interaction with the inspectors?”

RIVM Manager 2: “With great difficulty, at least between RIVM and Inspectorate and

I have the impression that the inspectors get so many stuff and small issues on their

desks that they have difficulty to pay attention to structured campaigns and they spent

a lot of time in acting in a reactive way. I suppose…, this is my impression.”

Organizational environment

At the RIVM, the Risk Model project was a rather isolated project that lacked solid

embedding in the research organization because the project was unusual and small in terms

of people involved and research hours available. Colleague investigators were busy with the

regular, large projects and not very interested because of the limited link with their research

activities. The investigators had a few deliberations with their management to monitor

the progress of the project at formalized moments (alignment efforts). The managers

controlled the finances and timely output, but maintained their distance to the project and

only commented on drafts of the final report (alignment efforts). At management level,

meetings between the Inspectorate and the RIVM only focused on general issues and the

Risk Model project was never discussed. The managers discussed neither the contributions



3

of the project nor a way out for the confidential status of the knowledge products in order

to facilitate publication of findings.

Investigator 2: “We worked rather solitary. I experienced it as an isolated project,

detached, isolated. It was not attached to other projects at all. It was not interwoven

with the RIVM network.”

An important finding was the major influence on the project of the established relationship

between both RIVM and the Inspectorate. In the interviews, the inspectors, investigators,

and managers expressed strong ideas of the other organization and we observed divergent

views on the mutual relationship. At management level, the budget system was always an

important issue since the Minister of Health does not allow the Inspectorate to use their

dedicated RIVM-budget for commissioning research to other organizations. The Inspectorate

considered the system therefore as a forced sourcing system without competitors for the

RIVM and without sufficient possibilities to influence the RIVM research proposals. This

view is in contrast with the perception of the RIVM managers and investigators, who

considered their research for the Inspectorate as working on a mutual governmental task

and experienced the Inspectorate’s view as unjustified distrust of their organization.

Inspectorate Manager 1: “There exists a very negative view on the RIVM. Not with

respect to their quality, but because we have this forced sourcing at the RIVM.”

RIVM manager 1: “The Inspectorate relies on RIVM results, but the forced sourcing

reflects the relationship and this causes lack of trust.”

On their side, the RIVM managers experienced that lack of time largely determined the

quality of the Inspectorate’s commissioning role often resulting in unarticulated research

questions. Because the RIVM accepted all research questions, this often led to unsatisfactory

contributions of research projects.

RIVM Manager 1: “So it was a knowledge question and actually, you were told to

elaborate it per definition. So the only choice we had, was to choose the way to deal

with the knowledge question, but not whether you should deal with it at all, that was

no consideration at that time.”

R1R2R3R4R5R6R7R8R9


62 | Chapter 3

Discussion

Relevant alignment efforts

Analyzing the course of the Risk Model project and as indicated in the findings, we observed

alignment efforts made by both RIVM and the Inspectorate; investigators, inspectors, and

managers tried to create alignment within the bounds of their reach, both according to

the institutional procedures at stated moments and according to the circumstances in the

project. The importance of well-organized interaction between the networks RIVM and the

Inspectorate is recognized on an institutional level; the RIVM and the Inspectorate have

agreed upon transparent formal procedures to streamline the yearly commissioning process

and to monitor the course of the agreed research projects. The Risk Model project was

managed according to these formal rules and procedures. De Leeuw et al. [12] described a

classification of 30 different theoretical frameworks for dealing with action on the nexus of

research, policy, and practice. They established three groups of seven categories of theories,

one of them the category Institutional Re-design. This category comprises theories that

acknowledge the importance of interactions between actors from different institutions,

which results in the explicit management of these interactions by maintaining institutional

arrangements and establishing institutional rules. In our view, alignment efforts by the

formal procedures match with strategies in the category Institutional Re-design. For both

RIVM and the Inspectorate, alignment rules and guidelines were top-down established and

explicitly encouraged project leaders and contact persons to interact with their counterpart

at the other organization during formulation and production phase of a research project.

Their management boards jointly discuss and revise regularly the agreed procedures to

improve commissioning process. For the extension phase, the RIVM procedures provided

guidance and rules for the presentation and layout, formal approval by the management

and the transfer of contributed knowledge products.

Despite these institutionalized alignment efforts, we observed difficulties that could be

attributed to lack of alignment at the level of the research project. Based on the course

of the project and the contributions as described in the results section, we inductively

identified six categories of alignment efforts that were relevant for the project.



3

Relevance: discussing the research results and their relevance for the commissioner and

attuning the presentation of the knowledge products to the needs of the commissioner

Our findings show that the investigators and inspectors translated the need for alignment

mainly in alignment efforts on ‘relevance’. The importance of alignment was in fact well

recognized by the investigators and inspectors in the Risk Model project; they developed

the two risk models as much as possible in close collaboration. During the production phase,

alignment efforts primarily focused on discussing the concepts and drafts for the risk models

and on the design of the electronic tool. From the perspective of both investigators and

inspectors, alignment efforts on relevance were crucial for successfully accomplishing the

research project. The investigators had to account for both scientific quality and customer

satisfaction and the inspector’s interest was to get timely knowledge products that fitted

to their needs, which required alignment on the level of the scientific aspects and the

presentation of the results.

Together, they also had to deal with the tension that exists in commissioned RIVM research

projects; the RIVM has an independent status established by law and in principle, the

commissioner cannot determine research methods, influence the outcome of studies or

prevent publication of results [1]. At first sight, the formal independent RIVM position

conflicts with the need to align on ‘relevance’ issues. Here, the concept of backstage and

frontstage work formulated by Goffman is helpful to overcome this contradiction [13].

Bekker et al. [14] described the RIVM as a boundary organization in the context of the Public

Health Status and Forecasts (PHSF) reports and the biannual Dutch Health Care Performance

reports. As a boundary organization, RIVM projects act as intermediary entities into which

organizations delegate actors to contribute to the research and where frontstage and

backstage work are combined in order to enhance the influence of the reports in informing

public health policy. At the frontstage, organizations present themselves in their formal

positions, whereas at the backstage, informal alignment paves the way for the organizations’

accountability at the frontstage. By analogy with the PHSF, the RIVM research projects for

the Inspectorate could also be considered boundary areas where research for the benefit of

the Inspectorate is performed and investigators, inspectors, and their managers collaborate.

From this perspective, the formal agreed procedures represented the frontstage work to

account for the organizations’ roles in the commissioning process. The meetings between

the investigators and inspectors and their informal interaction at a personal level can be

considered the backstage.

R1R2R3R4R5R6R7R8R9


64 | Chapter 3

Consultative structure: agreeing on and acting on the consultative structure

For the Risk Model project, a limited consultative structure existed that mainly focused

on the production phase. Van Egmond et al. [15] described the infrastructure developed

by the RIVM for the above mentioned PHSF report, indicating that the backstage was

well-organized, and presented it as an example for successful boundary work. However,

in the Risk Model project, a backstage area at the start of the new project hardly existed

and backstage work was limited to a few phone calls. Ample opportunity for backstage

interaction, already in an early phase of the research project, will enhance the identification

of possibly influencing factors and trust between actors and thus improve the process and

its contributions. For the third project year, the opportunity existed to discuss the next year’s

project plan during the regular meetings, but this opportunity was not intensively used. This

shows that backstage work is not only a matter of available time, but also of culture and

relationship between organizations. As argued by Wehrens et al. [16], a well-functioning

boundary organization needs “internal room to discuss different perspectives, goals and

expectations (and to find a balance that satisfies everyone involved) while the legitimacy of

the activities, products and projects is not questioned in the broader organizations of the

participants.” For the case of the Risk Model project, the internal room for backstage work

was limited due to priority setting in both organizations and the relationship between RIVM

and the Inspectorate that interfered with open discussion. The transition from the informal

backstage to the formal frontstage was not clearly marked since the project continued for

several years and, each year, new backstage work had to be done. Inspectors had to take

both frontstage and backstage positions at the same time, causing confusion at the side of

the investigators. In fact, the backstage was not very well developed and the organizations

did not sufficiently recognize its importance in relation to the frontstage.

According to Wehrens et al. [16], the backstage and frontstage of organizations have different

functions but are interrelated and cannot exist independently of one another. In most cases,

the frontstage is predetermined by official arrangements that can even be emphasized by

a ceremonial public presentation of the knowledge product to the commissioner. However,

in the Risk Model project, the frontstage was marginalized because of the requested

confidentiality of the risk models. Because of this, the Inspectorate did not have to account

for using the risk models, which distorted the backstage space and made the investigators

confused about the demarcation of backstage and frontstage.



3

Goal and timing: discussing and agreeing on the formulation of the knowledge question,

its origin, the question behind the knowledge question, and the underlying need for the

knowledge products

For three consecutive years, the Inspectorate formulated knowledge questions for the

Risk Model project. The procedure for commissioning research to the RIVM was rather

formal, with strict deadlines, and included several management levels for approval. The

management regularly updated the rules with the intention to improve the research projects

and their contributions. These formal procedures are important for legitimization of the

research projects, but offered limited opportunity for alignment efforts at the investigator

and key user level in the formulation phases. Although the formal alignment efforts with

respect to the goal and timing of the project were in line with the established procedures,

several interviewees mentioned that more informal interaction during this phase of a

research process could improve its contributions. These findings are in line with the critical

key factors identified by De Goede et al. [9] to influence the expectations of research users.

Early extensive alignment efforts between investigators and key users on the formulation of

the research question using previous experience and researchers’ knowledge, exploring the

actual needs behind the question, and taking into account study limitations is important to

manage the expectations on the contributions and achievable timelines at both sides.

Tasks and authority: explicit discussing of the relationship between actors and agreeing upon

input and role of actors, responsibilities, sharing of information and knowledge, authority

over the contributed knowledge products, and the follow-up of the research project

In the Risk Model project, the RIVM project proposals, as presented in the formal offers, were

designed according to the formal guidelines and approved by both the RIVM management

and the Inspectorate. However, the guidelines did not explicitly require consideration of

time investment by the inspectors, the ownership of the knowledge products, the exchange

of confidential information, and the follow-up of the project, whereas these issues caused

friction later in the project. The alignment strategies based on the Institutional Re-design

approach were not able to meet the need for mutual understanding between organizations

at project level. Another category of theoretical frameworks described by De Leeuw et al.

[12] is the Blurring the Boundaries model. In this model, the actors of organizations involved

learn about each other’s world by obtaining access to it. In our study, a notable finding was

that the RIVM and the Inspectorate afterwards differently assessed the contributions of the

project, the interaction during the research process, and the ownership of the knowledge

R1R2R3R4R5R6R7R8R9


66 | Chapter 3

products. Investigators and inspectors had different actor scenarios, but were not aware

of this. This illustrates the need for more alignment strategies based on the Blurring the

Boundaries model. Understanding each other will generate more trust and confidence,

better recognition of perceptions and more open discussions on sensitive issues. Every

actor in a research process has his/her own scenario and perspectives for the future of the

research project contributions. If investigators are aware of the changing circumstances in

the commissioner’s organization and understand the role knowledge and research projects

have in the commissioner’s scenario, they will be better able to promote the research

contributions.

Vertical alignment: sharing information within the own organization and embedding of the

research project within the own organization

In the literature, the importance of horizontal alignment between research and key

users is exhaustively described; horizontal alignment efforts are considered the key for

enhancing research contributions. We observed that also the vertical alignment within the

own organizations of both investigators and inspectors was a very relevant factor for the

research process and its contributions. Some difficulties in a project with respect to tasks

and authority, relevance, and organizational environment cannot be tackled by investigators

and linked actors at a project level and will often ask for a combination of horizontal as

well as vertical alignment efforts at different levels in the organizations. The embedding

of Risk Model in both RIVM and the Inspectorate was limited and the management was

hardly involved during the production process, while their alignment efforts focused on

general issues and not on project-specific issues. We argue that explicit vertical alignment

efforts could bring project-specific issues to the attention of the management level at which

horizontal alignment efforts could really enhance the solutions. The required confidentiality

of the risk models serves as an example of a fundamental issue where vertical alignment

efforts were lacking. The Inspectorate desired confidentiality because publication would also

mean that they had to justify the selection criteria in the risk models to outside stakeholders

and that they could no longer diverge from the selection method. On the other hand, the

RIVM’s task and interest was to deliver scientifically legitimate knowledge products and,

to achieve this, both the necessity of stakeholders’ involvement and publication of the risk

models should have been considered. The inspectors were bound to their own organizations’

policy and the investigators met the requirements of the commissioner. Vertical alignment

efforts on project-specific issues would have identified the different views of RIVM and



3

the Inspectorate at the appropriate organizational level at an early stage and could have

resulted in a more grounded RIVM position on the ownership issue and scientific approach.

Organizational environment: aligning with respect to relevant conditions outside the

research project influencing the relationship between investigators and linked actors and/or

the research project, such as changing priorities, incidents, media-events, and relationship

with other organizations

In the formulation phase, a new research project is dealt with as a well-defined assignment

that can be planned and controlled by the project manager. In reality, the research project

and its actors are part of a complex organizational environment that continuously changes

and influences the course and contributions of the project. In the Risk Model project, we

identified several unexpected influencing factors from the Inspectorate’s organizational

environment, such as data accessibility, time investment by the inspectors, and a serious

incident that influenced the second risk model. At the RIVM site, implicit, not openly discussed

organizational circumstances, such as the project’s embedding in the organization and the

historical relationship between RIVM and the Inspectorate, appeared to be influencing. On

the one hand, both investigators and inspectors did not exactly know and understand the

underlying feelings and motives of the other organization and their alignment efforts focused

mainly on the scientific issues. On the other, the project’s backstage did not sufficiently

facilitate reflection on the project to develop feelers for influencing factors from the project’s

organizational environment. Conscious alignment efforts on the organizational environment

would require more trust and understanding as argued in Blurring the Boundaries models

combined with a well-developed backstage to accommodate discussions.

Reflection on contribution mapping (CM)

The evaluation method of CM aims to offer a practical and realistic approach to evaluate the

contributions of a research project and to analyze the alignment efforts. In CM, both key users

and investigators have an active role by being interviewed and participating in the feedback

session. Due to these characteristics, this method can be considered a fourth generation

evaluation method [17]. The criteria for assessment of contributions and alignment efforts

are not fixed beforehand, but are based on issues that emerge from the interviews. In our

view, this responsive evaluation was useful to unravel a research project such as Risk Model,

the contributions of which remained, on the face of it, unclear. To elucidate our questions

on research contributions, a qualitative approach taking into account the actors’ own issues

R1R2R3R4R5R6R7R8R9


68 | Chapter 3

and their social process was needed to find clues for improvement on both organizational

and project level. By using CM, we indeed gained an illuminating insight in the course of the

Risk Model project. The interviews delivered a rich data set that disclosed the contributions

and alignment efforts. When presenting these findings to inspectors during the feedback

meeting, investigators and managers recognized the analysis and findings, which adds to

the validity of the analysis. The feedback meeting gave opportunity to discuss the differing

views, to exchange experiences, and to formulate concerted actions to solve identified

issues.

We would like to comment that for a small project such as Risk Model, the number of

interviewees can remain limited to get a complete overview. For a major project, we expect

that this would require a proportional number of interviews with a corresponding workload.

Depending on the aim, CM users could consider focusing only on specific key users in order

to conduct the CM with achievable effort. Another possibility for selective application of

CM is to choose a project as a typical example in a specific context. In this case study, we

recognized patterns and issues that are directly applicable to other RIVM research projects

as well.

The concept of contributions was very helpful to identify the (added) value of the project

for the Inspectorate’s work. The project’s attributed value manifested itself not as expected,

but could be demonstrated by the concept of ‘contribution’. The CM method does not focus

on assessment of the contribution category ‘utilization on distance’. In our case, this was not

a drawback because of our focus on contributions to the commissioner’s work to improve

RIVM projects.

Although the first aim of the feedback session was to add information to the collected data

from interviews and document analysis, we experienced that such a feedback session is

an alignment strategy in itself. We drafted an abridged report of the discussion on which

the participants commented. The consensus report included several recommendations for

improvement of the knowledge production process to be taken aboard for future projects

and a proposal for a pilot to test the recommendations in a number of new projects. We sent

the report to managers within both organizations, and we therefore consider the feedback

session as a very useful three-layer part of the case study; the scientific data collection layer

where participants were able to add information or to comment on findings; an alignment



3

strategy in itself since the participants solved issues during the discussion; and a method to

improve the research process.

By joining a systematic evaluation session, participants can learn how to improve the

research process [18]. Although in our case all participants valued the feedback session in this

respect, we experienced that the formulated actions for improvement did not automatically

result in better research processes in actual RIVM projects since the agreed actions were

subject to the same alignment problems as the project and turned out to be more complex

than expected during the feedback session. We argue that a carefully planned and managed

follow-up would be the fourth prerequisite of a successful feedback session.

Study limitations

We have to mention some limitations of this study. The principal researcher of this study

was also involved as an investigator in the Risk Model project. As an RIVM investigator, she

is also involved in other projects for the Inspectorate at present. On one hand, she had the

advantage of having inside knowledge about the research process and the context of the

case. To overcome the disadvantage of being biased by own experiences, the coding of the

interviews was validated by three other researchers not involved in the Risk Model project.

The research team intensively discussed the analysis of the findings. Moreover, the findings

were also discussed with the participants of the feedback session. Combining all feedback,

we conclude that consensus on the analysis exists within the research team.

In this article, we describe the first case in a multiple case study on the contributions of

government-commissioned research projects. For this multiple case study, the different cases

are selected to cover a variety of characteristics that may influence the research process and

its contributions. This first case was a relatively small research project within the specific

context RIVM-Inspectorate. This questions to what extend the findings can be generalized

to other research projects. Our findings were largely in line with our expectations based on

the study framework. According to CM, inspectors, the RIVM project leader and managers,

and the Inspectorate discussed the contribution map in a feedback session. They agreed on

the findings and identified general issues that also emerge from other research projects.

Moreover, we presented the final contribution map in several scientific meetings in the

RIVM. The influence of limited interaction, organizational environment, and relationship

with other organizations is widely recognized, also by investigators operating in other

R1R2R3R4R5R6R7R8R9


70 | Chapter 3

expertise fields and working with other commissioners. Although this case study describes a

specific and small project, we consider the findings relevant for government-commissioned

research projects. However, we will have more insight in the generalizability of this case

after completion of all cases in our multiple case study.

Conclusions and recommendations

By analyzing the course of the Risk Model project and its contributions, we wanted to

establish which alignment efforts are important for the RIVM-Inspectorate context in

order to find clues for improving the contributions of RIVM projects. We identified six

categories of relevant alignment efforts that can be undertaken at three different levels:

at the first level, alignment efforts between investigators and linked actors; at the second

level, alignment efforts related to the organizational environment of the project appeared

also to be important; at the third level, alignment efforts between organizations as part

of their formal and historical relationship. At the first level, investigators and linked actors

can improve alignment by continuous reflection on actor scenarios, awareness of mutual

expectations, and open discussions with commissioners at all levels in the organization. This

will ask for substantial efforts from all actors involved and will take precious (research) time

for interaction on an ongoing basis.

At the second level, more awareness of the importance of the organizational environment

could help to formulate an adequate response to challenges for the research project for

which vertical alignment efforts can be essential. Since the vertical alignment efforts turned

out to be of influence, the focus in a research project should not only be on the external

alignment efforts. Attention for the role of internal interaction, both at the level of the

project as well as management level, and a good institutional embedding could improve the

research process and thus its contributions.

At the third level, alignment efforts are more difficult to deal with; the historical relationship

of the institutes determines their views on each other at all organizational levels and thus

influences the interaction during the research process. This ‘soft side’ of the project can be

hard to manage and therefore a collective approach at all organizational levels should be

developed. It could be useful to openly discuss the historical relationship and its influences

on the interaction within, and even between, organizations. We suggest that both research

organizations and commissioning bodies should invest time and energy in regularly



3

organized reflection sessions where both researchers and managers openly analyze and

discuss the relationship with other organizations. This also implies that maintenance of this

awareness will be important since collective views and memory also influence new actors.

If all actors are continuously aware of the historical context and relational aspects of their

research projects, they could anticipate on the implicit consequences and improve their

alignment efforts. At the strategic organizational level, alignment efforts should focus on

the organizational image to increase trust and legitimacy in order to overcome historically

determined, yet invalid frames.

Based on the findings, we recommend that a research institute should encourage a reflective

attitude towards the social aspects of research projects at all levels in the organization. The

first step for improvement is to validate the importance of the process factors for a new

project; the second step would be to calculate in the need for time for these aspects of the

research process at the start of the project. This will facilitate the third step to anticipate

explicitly on alignment efforts during the research process. With respect to the evaluation

of a research project, we experienced that CM offers an alignment effort in itself, apart

from being a useful instrument to evaluate a research project. The feedback session as part

of CM yielded not only information about the research process and its contributions but

also elicited awareness and preparedness for improving research processes at the side of

the participants. CM can therefore be a useful instrument in reflections on contributions

and research processes. Nevertheless, one should be aware that a feedback session needs

carefully planned follow-up in order to improve the research process; the jointly formulated

actions are in fact new projects themselves and need no less alignment efforts than other

projects.

Organizations, investigators, and all other linked actors should be encouraged to experience

the profits of social investments in research projects and should be rewarded for doing

so. To achieve this, different strategies based on different models should be considered.

Institutional rules incorporated in the organizations ‘quality system’, can facilitate project

teams, but should be coupled with sufficient room for well-organized backstage work

with involvement of actors familiar in both organizations. The challenge is to convince

organizations and their staff to consider alignment efforts as a valuable dimension of

research projects, instead of extra burden, because of its influence on project contributions

and on trust in and legitimacy of the organization in a broader sense.

R1R2R3R4R5R6R7R8R9


72 | Chapter 3

References1. RIVM. Wet op het RIVM (In English: Act on RIVM). Available from: http://wetten.overheid.nl/

BWBR0008289/. Accessed on 16 May 2014.2. Gezondheidsraad. Op weg naar evidence based toezicht. Het onderzoek naar effecten van

toezicht door de Inspectie voor de Gezondheidszorg. The Hague: Health Council of the Netherlands; 2011.

3. Inspectie voor de Gezondheidszorg. Meerjarenbeleidsplan 2008–2011 Voor gerechtvaardigd vertrouwen in verantwoorde zorg. The Hague: Health Care Inspectorate; 2007.

4. Inspectie voor de Gezondheidszorg. Meerjarenbeleidsplan 2012–2015 Voor gerechtvaardigd vertrouwen in verantwoorde zorg (II). The Hague: Health Care Inspectorate; 2011.

5. Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Syst. 2009; 7(Suppl 1):I1.

6. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking(STP)1:what isevidence-informedpolicymaking?HealthResPolicySyst.2009;7(Suppl 1):S1.


8. Demon JMH. Strategic Research RIVM 2011–2014. Project summaries. Bilthoven, Nederland: RIVM; 2011.

9. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriersbetween epidemiological research and local health policy development. Health Res Policy Syst. 2010, 8:26.

10. Yin RK. Case Study Research Design and Methods. 4th rev ed. Thousand Oaks: Sage Publications; 2008.

11. Friese S. Qualitative Data Analysis with ATLAS. Ti. Thousand Oaks: Sage Publications; 2011.12. De Leeuw E, McNess A, Crisp B, Stagnitti K. Theoretical reflections on the nexus between

research, policy and practice. Critical Public Health 2008, 18 (1): 5–20.13. Goffman E. The Presentation of Self in Everyday Life. London: Penguin; 1990.14. Bekker M, van Egmond S, Wehrens R, Putters K, Bal R. Linking research and policy in Dutch

healthcare: infrastructure, innovations and impacts. Evid Policy. 2010; 6 (2):237–253.15. Van Egmond S, Bekker M, Bal R, van der Grinten T. Connecting evidence and policy: bringing

researchers and policy makers together for effective evidence-based health policy in the Netherlands: a case study. Evid Policy 2011; 7 (1):25–39.

16. Wehrens R, Bekker M, Bal R. Coordination of research, policy and practice: a case study of collaboration in the field of public health. Sci Public Policy 2011, 38:755–766.

17. Abma TA. Responsive evaluation: its meaning and special contribution to health promotion. Eval Program Plann. 2005; 28 (3):279–289.

18. Carden F. Issues in assessing the policy influence of research. Int Soc Sci J. 2004; 56:135-151



3

Additional files

Additional file 3.1: Yearly cycle for RIVM research in commission of the Health Care

Inspectorate. Textbox describing the yearly research cycle of the RIVM: the formulation of

the knowledge question by the commissioner, the corresponding RIVM offer, the research

phase, and finally, the delivery of the agreed knowledge product.

Additional file 3.2: Topic list interviews actors Case Study Risk Model. The topic list for

the interviews consists of items regarding the research process, the actors and their

organizational environment, the interaction between actors and the contributions in the

three phases of knowledge production. Electronic supplementary material available at

https://health-policy-systems.biomedcentral.com/articles/10.1186/1478-4505-12-8

Chapter 4

Enhancing the contribution of research to health care

policy-making: a case study of the Dutch Health Care

Performance Report

Published: Journal of Health Services Research & Policy 2016;21(1):29–35

Ingrid Hegger

Lisanne K. Marks

Susan W.J. Janssen

Albertine J. Schuit

Hans A.M. van Oers

R1R2R3R4R5R6R7R8R9


76 | Chapter 4

Abstract

Objectives

The Dutch Health Care Performance Report (DHCPR), issued by the National Institute of

Public Health and the Environment (RIVM), aims to monitor health care performance in The

Netherlands. Both the RIVM and the Ministry of Health wish to increase the contribution

of the DHCPR to health care policy-making. Our aim was to identify ways to achieve that.

Method

We used Contribution Mapping as theoretical framework that recognizes alignment

of research to policy-making as crucial to managing contributions to policy-making. To

investigate which areas need specific Alignment Efforts by researchers and/or policy-makers

to enhance the contributions of the DHCPR, we interviewed RIVM researchers and policy-

makers from the Ministry of Health and assessed the process for developing the 2010

version of DHCPR.

Results

We identified six areas where alignment is specifically relevant for enhancing the

contributions of future versions of the DHCPR: well-balanced information for different

ministerial directorates; backstage work; double role actors; reports of other knowledge

institutes; data collection/generation; and presentation forms.

Conclusion

The contribution of health care performance reporting to policy-making is complex and

requires continuous alignment efforts between researchers and policy-makers. These

efforts should form an inseparable part of health care performance reporting and although

this demands considerable resources, it is worth considering since it may pay back in better

contributions to policy-making.


Case study Dutch Health Care Performance Report | 77

4

Introduction

The Dutch Health Care Performance Report (DHCPR) is developed by the National Institute

of Public Health and the Environment (RIVM) to monitor the performance of the health care

system for the Ministry of Health (MoH) in The Netherlands. Based on a scientific framework,

the DHCPR includes indicators for three important public goals: quality, accessibility and

affordability [1,2]. So far, RIVM has published four editions (2006, 2008, 2010 and 2014).

Their primary function is to contribute to ‘strategic policy-making’ and both RIVM and MoH

wish to strengthen this function [3,4].

Knowledge use in policy-making is a complex matter and no simple way exists to improve it.

Weiss characterized three types of knowledge use in policy, often described as instrumental

(data supply), conceptual (formation of an idea) or symbolic (arguments for a certain

position) [5-8].Sometimes, agenda-setting is mentioned as a fourth way [9].For researchers,

instrumental use of knowledge fits into their logic, yet the principal use for policy-making

is conceptual or symbolic [10,11]. Furthermore, socio-organizational contexts influence

knowledge use; knowledge has to coincide with the users’ needs, to be credible to the user

and to reach the user at the right time [12]. To anticipate users’ needs, Lomas pointed out

that linkage and understanding between the policy and research world is important for

enhancing knowledge utilization [13].

A study on how MoH officials used the 2008 version of the DHCPR focused on its utilization

as a ‘desk book’ (instrumental use) but did not take the developmental process of the

report into account [14]. The study concluded that RIVM’s aim to achieve a clear position

for DHCPR in the policy process had not yet been reached. Interaction of researchers and

policy-makers during the developmental process influences utilization of research products

in policy-making [15]. Four clusters of barriers between research and policy have been

suggested which are promoted or obstructed by interaction: expectations, transfer issues,

acceptance and interpretation. In line with this interaction model, Van Bon-Martens et al

demonstrated that next to content and presentational form, the developmental process is a

decisive factor in the perceived quality of regional public health reporting in The Netherlands

[16]. Moreover, Bekker et al. described how informal interaction between researchers and

policy-makers during this process facilitates acceptance of research, using the concept of

backstage work introduced by Goffman [17, 18].

R1R2R3R4R5R6R7R8R9


78 | Chapter 4

Based on these findings, we decided to take into account the developmental process

and alignment between researchers and policy-makers to investigate the options for

strengthening the DHCPR contribution to policy-making. Contribution Mapping formulated

by Kok and Schuit provides a method to take into account the social context of a research

project [19, 20]. According to its Three Phase Model, researchers and policy-makers form

a hybrid during the co-creation of research, which consists of three phases (Figure 4.1):

formulation (activities to initiate the research process); production (activities to realize the

knowledge products); and extension (activities to make knowledge available to potential

users).

The phases can generate ‘contributions to action’, a concept which encompasses categories

such as ‘change in abilities and actions of involved and linked actors’, ‘contributed

knowledge products’, ‘contributions through linked utilization’ and ‘indications of utilization

at-a-distance’ [19]. Furthermore, contribution mapping recognizes the importance of

‘alignment efforts’. These are specific actions in the research process to enhance the desired

contributions of the research.

Figure 4.1. Three-phase model by Kok and Schuit [19].

Our aim was to study the development of the 2010 DHCPR to identify which areas need

specific alignment efforts by researchers and/or policy-makers to enhance the Report’s

contribution to policy-making [19].



4

Method

A qualitative case study including document analysis (records of the DHCPR process and

policy documents) and interviews [21]. In the semi-structured interviews, we used a

deductive topic list based on the theoretical framework and structured according to the

Three-Phase Process Model (Supplementary File I). An independent researcher (not

employed by RIVM) interviewed: eligible researchers of the RIVM project team (3 project

leaders and 5 researchers); and nine senior policy makers from different MoH directorates

were members of the Ministry’s project group. One additional policy-maker responded by

e-mail.

First, four informal interviews were conducted to explore the DHCPR context. The other 13

interviewees were selected on the basis they had participated in the project group for both

the 2010 and 2014 Reports and/or because they were referred to in the informal interviews.

During the interview period, the 2014 Report was being prepared so we included topics to

investigate whether any changes had been implemented since 2010. Each interview lasted

approximately 60 minutes, was recorded and transcribed verbatim. After importing the data

into Atlas-ti version 7.0, the data were coded according to a deductive code list that was

inductively completed (Supplementary file II). The coding of the interviews started while

data collection was still ongoing. Three researchers independently coded two interviews

and we validated the coding list in a joint session to reach consensus. We reconstructed

the DHCPR2010 process and analyzed the alignment efforts and contributions. We also

performed a member check.

Results

We present our findings by describing noticeable issues for both the contributions of the

2010 Report to policy-making and the alignment efforts during the developmental process

of the DHCPR 2010.

Contributions

Position as information source for policy-makers

The unique selling point of the DHCPR is that it offers an overall picture of the health care

system. This makes it useful for policy directorates and policy-makers who deal with the

health system at a macro level (such as the policy directorate for Market and Consumer

and higher MoH levels). Policy-makers dealing with more specific health domains (such as

R1R2R3R4R5R6R7R8R9


80 | Chapter 4

long-term care or health care insurance) mentioned that, for their directorates, the DHCPR

merely adds to more detailed information from other sources. For them, the DHCPR was

not the distinctive and indispensable information source intended by RIVM. One policy-

maker stated that ‘It is a good product, but as basis for policy-making, it is too broad.’ They

considered the content was too general and the data outdated and preferred the original

(and more up-to-date) data of other sources.

In the light of policy-making dynamics, such as a change of government and the need for

political responses to today’s problems, researchers indicated that strategic timing of DHCPR

is almost impossible. The production period of such a major report is long (now even four

years) and inflexible which makes it difficult to attune to the policy-making cycle.

New communication tools

After the publication of the 2010 Report, RIVM launched a website to improve the accessibility

and topicality of DHCPR data. This publicly accessible website allowed for interim updating,

which proved to be an important reason for the MoH to delay commissioning the next

edition until 2014. Based on website visit numbers, the researchers expected the DMCPR

to have an increased contribution to policy-making. As one researcher mentioned ‘…an

update generates a peak, there are more visits; apparently it is noticed.’ However, interim

website updates depended on availability of new data, making these updates somewhat

arbitrary. Although policy-makers were positive about the update feature, they did not

think that the website contributed to an increase of their DHCPR use, some indicated that

the interrelationship between the trends and conclusions described in the DHCPR and the

randomly updated website was not clear.

In 2012, RIVM intended to renew public attention for the DHCPR in a novel way by publishing

infographics based on already published data. The first infographic on health care costs

generated substantial social media attention and increased visits to the DHCPR website.

Before publication, RIVM did not feel the need for extensive alignment on content and

timing with the MoH. Policy-makers assessed the infographic’s contributions and considered

the infographic particularly informative for the public but not relevant for themselves. Some

were not familiar with the infographic, while other policy-makers entered into a debate with

RIVM because figures presented in the infographic seemed inconsistent with a ministerial

report that was published on the same day. These policy-makers were concerned that any



4

inconsistency between definitions used in the infographic and the ministerial report could

give political trouble. In this context, one policy-maker stated ‘… at least send it first to the

MPG so that we can check the figures, (....) and align the timing’ whereas a researcher

argued ‘It (infographic) contained nothing new, but now you make a graphic of it, they

(policy-makers) suddenly notice things they didn’t notice before.’

Importance as international business card

An additional function of the DHCPR is that of ‘international business card’. Due to its

translation into English, it can serve as a comprehensive description of the Dutch health care

system. Dissemination of the English version contributed to the RIVM researchers’ position

in the international scientific community. In contrast, most policy-makers did not value this

DHCPR function as highly, though it contributed in a practical way to their work by showing

the performance of the Dutch health care system internationally:

‘It (DHCPR) is pretty good for our public relations; internationally it is also valued. (……).

However, if you really look at its impact in the sense of ‘what is its influence on policy and

changes’, it is rather limited.’

Response to research agenda

In every edition of the DHCPR, the RIVM researchers have dedicated one chapter to

knowledge gaps and lacking data on measuring health care system performance, which they

consider a research agenda. Despite this, generation of useful and acceptable qualitative

acceptable data remained a source of concern. So far, the DHCPR project itself had only the

task and budget to collect and integrate data. In 2012, RIVM made a strategic analysis on

data supply for DHCPR and concluded that data on public health and health care require but

lack strategic direction [22]. However, RIVM is not in a position to direct data generation by

other parties. The policy-makers recognized the issue but considered data generation the

responsibility of health care organizations. Nevertheless, some expressed the intention to

improve information management in the health care system, together with RIVM, in the

near future.

Alignment efforts in the DHCPR 2010 process

Aim of DHCPR: choice of indicators

Based on a theoretical framework, RIVM had established the performance indicators for the

first 2006 edition of DHCPR in an interactive process with policy-makers and academics [1,

R1R2R3R4R5R6R7R8R9


82 | Chapter 4

23]. To enable long term monitoring, the indicator selection remained almost unchanged

over the years. For the 2010 edition, the aim of DHCPR was not discussed in the formulation

phase. Only some slight adaptations were made to the indicator set and definitions. However,

we noticed that policy-makers differed in their opinion on the intended users and the choice

of indicators: ‘In fact, all directorates at the Ministry belong to the DHCPR target group.’; ‘In

my opinion, (the intended DHCPR user is) not our Directorate’; ‘Maybe policy-makers should

not be the intended users of this booklet, but people interested in health policy.’

Some members of the MPG changed over the years and for the 2010 edition, the group

included members who had not been involved in 2004. After the 2008 Report, RIVM

had conducted a survey of policy-makers to investigate which indicators they considered

important. Due to the varied results, RIVM decided to stick to the existing indicators.

However, the discussion on appropriate indicators reappeared for the 2010 Report during

the production phase causing divergence of opinion between RIVM and some policy-makers:

‘Several times, we indicated that we preferred other indicators and that those (selected) did

not give a full picture of our policy. (…..) Nevertheless, at a certain moment, RIVM made their

own choice to include these indicators.’

Backstage work: regular meetings and participation in the project group

During the production phase, the RIVM project team had regular meetings with the

Ministry’s project group to discuss progress, methods, indicators and drafts for the DHCPR.

Furthermore, the project leaders had weekly telephone calls with their contact person at

the MoH. Due to these alignment efforts, the Ministry project group members were familiar

with the DHCPR content before release. The RIVM researchers were dedicated to find data

for indicators and to integrate their findings in draft texts. They had incidental interaction

with the project group or individual policy-makers and gained a limited view on the policy-

making dynamics at the MoH.

Each MoH directorate appointed a project group member, mostly senior policy-makers with

a broad overview of their field. The members’ involvement in the group varied according to

their policy domain and personal interest: ‘We intentionally composed a project group where

people from different directorates (…) could give input. Largely, it (active participation) then

also depends on the person.’



4

For the Ministry project group, the final episode of the production phase was the most

lively when they discussed the formulation of the final report with the RIVM project leaders:

‘That is the fascinating ‘game of words’ that you play; how are you going to formulate (the

conclusions) in such a way that we both agree with it.’

Policy-makers particularly focused on checking data relevant for their own directorate to

verify consistency with other data used by the MoH. For them, preventing political turmoil

was an important objective.

Double role actor: important alignment effort discontinued

To pave the way, one project leader had his office at the MoH and maintained a large informal

network of policy-makers which enabled him to identify upcoming policy issues and the

policy-makers’ needs. If policy-makers had questions with respect to the DHCPR, he could

rapidly provide them with an answer. In the interviews, both researchers and policy-makers

stressed that this confidence-building double role was beneficial for the DHCPR process.

In 2010, the person in this double role retired and his position remained unfilled during

the development of the 2014 Report. This was regretted by colleagues and policy-makers:

‘He knew what was going on and who to contact for what. That (retirement) might be a

disadvantage in comparison with previous (DHCPR) processes’.

Extension strategy: gaining ongoing attention for DHCPR

The extension phase started with the release of the Report in 2010 and was going to last for

four years instead of two, due to the extended period between editions. After the formal

presentation to the Minister of Health, the RIVM project team also presented the report to

a wide audience of policy-makers at a well-attended meeting and disseminated hard copies

to the MoH. Dissemination by Ministry project group members in their own department

depended on their interest in the DHCPR, reflecting their position in the group during

the production phase. One policy-maker described her routine as: ‘I send an e-mail like

‘colleagues, this is the publication’ and I also mention that anyone interested could collect a

hard copy at my room. So, I do communicate that a DHCPR is published.’

R1R2R3R4R5R6R7R8R9


84 | Chapter 4

After this initial period, policy-makers’ attention evaporated quickly in the continuous

information overload. As one policy-maker said: ‘Every day, reports arrive and you can be

unlucky that it coincides with something else (….). It is very hard to ‘time’ those things.’

The RIVM project team recognized the need for continuous efforts to gain attention for the

Report during the extended four year period to 2014. They applied new communication

tools (website, infographics) to reach policy-makers after publication, as recommended

in the evaluation report on the 2008 Report [14]. The regularly updated website did not

automatically gain the attention of policy-makers. As one of them stated: ’I guess that these

(updates website) are certainly interesting for colleagues, but they have to know about it

and should not forget it (…). They have to campaign for that.’

Discussion

Opportunities for improving alignment

Alignment with differing policy-makers

RIVM researchers tended to consider the key users of the DHCPR as one group, ‘the policy-

makers at MoH’. Although this may be apparently true from a distance, different needs in

each policy domain existed. To get round interpretation barriers, a watchful attitude with an

eye for policy-makers’ perspectives is indispensable [15]. Some policy-makers felt ignored

because indicators they proposed were not included in the Report. Although the RIVM

project team considered this mismatch inevitable due to the macro level, it nevertheless

adversely influenced the DHCPR image at the MoH. Alignment with policy-makers,

specifically those newly involved, to create acceptance of choices requires continuous

attention during the production process. The project team could develop the backstage

work with direct involvement of researchers, to improve the Report’s alignment with policy

in the early production phase [24].

Alignment in language

Whereas RIVM researchers hoped for discussions with policy-makers on the impact of their

findings, the language used to describe findings turned out to be more important for policy-

makers. Specifically the way RIVM formulated their findings on the (not yet) achieved policy

goals, was of major concern. Policy-makers quite often explained their concern by saying

‘you may equally well write down ‘the glass is half-full’ as write down ‘the glass is half-

empty’’. The policy-makers were concerned that the Report did not present achievements



4

as ‘half-empty’. The boundary between science and policy had to be negotiated. This

was primarily done in the interaction between project leaders and policy-makers in the

backstage. Involving other researchers directly in backstage work can create more feeling

for the impact of language which could improve the presentation of findings.

Alignment by person adopting a double role

A researcher, who is part-time positioned at the MoH and can take the lead in efficient

alignment with policy-makers at the MoH, is regarded as a major alignment effort by Kok

& Schuit [19]. The development of the DHCPR had such a person in a double role and his

contribution to alignment was widely recognized. Despite this, the double role was not filled

for the development of the DHCPR in 2014 and a valuable alignment effort imperceptibly

disappeared.

Alignment with knowledge products from other sources

Policy-makers compared the DHCPR with reports and information from other sources, which

influenced how they valued DHCPR. More alignment with other sources on areas of special

interest and timing of products could strengthen the DHCPR position in the information

flood that policy-makers face.

Alignment on data management

Both policy-makers and researchers felt rather powerless with respect to tackling the data

gap that frustrates filling of indicators with recent figures. Data management is of vital

importance and needs a strategic approach [22]. On this issue, higher management levels

should be involved to create better prospects.

Alignment on the function of products

RIVM initiated two new presentation forms to bring the Report to the policy-makers’

attention – a regularly updated website and infographics. Not all policy-makers were

automatically receptive to these spin-off products. The website’s updates attracted limited

attention while the first infographic remained unnoticed and caused some concern about

the publication of consistent figures. Because the relation between the different DHCPR

presentation forms remained unclear for the policy-makers, they were not convinced about

the usefulness of the new products. Timely alignment on presentation forms and product

coherence are important for policy-makers.

R1R2R3R4R5R6R7R8R9


86 | Chapter 4

Methodological reflections

We used contribution mapping to analyze retrospectively how alignment efforts played

a role in understanding the contributions the Report made to health policy-making. We

used the Three Phase Model to structure the topic list and explained it to the respondents,

combined with the DHCPR development chronology. Although the phases were less distinct

in reality, the model enhanced the respondents’ recollection of the process and proved to

be useful.

Policy-making is a complex process in which many factors are important and linear knowledge

use is secondary [11]. Contribution mapping unraveled leads for better alignment in the

future. Although this method offers an example of nine alignment efforts, it does not restrict

alignment efforts and we were enabled to formulate our own specific areas for alignment

efforts.

An additional benefit of contribution mapping is that the method turned out to be an

alignment effort in itself, initiating discussion between the project leaders of RIVM and

MoH. We informed them on the findings by an extended study report and also put effort

in securing our findings for the RIVM organization by designing a tool for RIVM researchers

to help them to reflect on the research process. Currently, this tool is under development.

Study limitations

Our insights relate to interaction at project level, since we restricted the respondents to

RIVM researchers and directly involved policy-makers. It could be beneficial to conduct

research on the influence of the wider organizational context, as contributions could follow

indirect pathways and alignment at management level also may generate improvement.

Moreover, interviewees may not have fully remembered all relevant information or have

suffered from recollection bias. However, minutes of meetings were consulted, and served

as a triangulation tool for the interviews. The number of interviewees was large enough to

get a general picture of how the process was executed.



4

Conclusion

We identified six areas where specific alignment efforts could help to improve the

contribution of future DHCPR editions to policy-making. These are efforts to:

• include balanced information for all relevant ministerial directorates;

• enhance the participation and commitment of policy-makers and researchers in

backstage work;

• blur the boundaries with the policy-making world by assigning people with double

roles;

• align with relevant reports from other sources;

• approach strategically data collection and enhance data generation for lacking DHCPR

indicators; and

• align the presentation of different DHCPR functions and products with the user’s

needs.

Establishment of structured alignment efforts is just the first step. On the long term, they

need attention, evaluation and maintenance. Due to the repetitive character of the DHCPR,

the definition of a new edition may become routine and minimal alignment efforts could

seem enough in the formulation phase. The pitfall is that this may result in divergent,

implicit expectations and a blurred DHCPR definition. Project groups where researchers and

policy-makers meet to discuss the progress of the DHCPR were important but are not a

guarantee for sufficient alignment with all policy-makers because the Ministry consists of

different policy worlds with their own goals and own policy logic. One valuable alignment

effort, the person with a double role, could disappear unnoticed if not recognized as an

organizational asset.

The contribution of health care performance reporting to policy-making is complex and

requires intensive alignment efforts. Not only is the number of alignment efforts important

but also their frequency. Researchers and policy-makers should undertake continuous

alignment efforts during all phases. Furthermore, both should reflect on existing alignment

efforts and regularly analyze the outcome to adapt to changing circumstances. Although

implementing alignment efforts may demand considerable resources, this investment is

worth considering since it will pay back in better contributions to health care policy-making.

R1R2R3R4R5R6R7R8R9


88 | Chapter 4

References1. van den Berg M, Kringos D, Marks L and Klazinga N. The Dutch health care performance report:

seven years of health care performance assessment in the Netherlands. Health Research Policy and Systems. 2014; 12: 1.

2. Dutch Health Care Performance Report 2006 [Zorgbalans. De prestaties van de Nederlandse gezondheidszorg in 2004]. In: Westert GP, Verkleij H., editors. RIVM rapport 260602002. Bilthoven: RIVM; 2006.

3. van den Berg MJ, Deuning C, Gijsen Rr, Hayen A, Heijink R, Kooistra M, Lambooij M, Limburg LCM. Definitierapport Zorgbalans [Dutch Health Care Performance Report]. RIVM rapport 260612001. Bilthoven: RIVM; 2011.

4. Westert G, Van den Berg M, Zwakhals S, Heijink R, De Jong J, Verkleij H. Dutch Healthcare Performance Report 2010. Bilthoven: RIVM, 2010.

5. Weiss CH. Policy research: data, ideas or arguments? In:Wagner P,Weiss CHW,WittrockB,Wollman H, editors. Social sciences and modern states. Cambridge: Cambridge University Press; 1991. p. 307–332.

6. Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication. 2004; 26 (1): 75-106.

7. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002; 7: 239-44.

8. Lavis J, Robertson D, Woodside J, McLeod C, Abelson J. How Can Research Organizations More EffectivelyTransferResearchKnowledgetoDecisionMakers?MilbankQ.2003;81:221-248.

9. Bekkers V, Fenger M, Homburg V and Putters K. Doorwerking van strategische beleidsadvisering. 2004. Available from: http://hdl.handle.net/1765/1594. Accessed 2013.

10. Lomas J, Brown AD. Research and Advice Giving: A Functional View of Evidence Informed Policy Advice in a Canadian Ministry of Health. Milbank Q. 2009; 87: 903-26.

11. Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14:2.

12. Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Publ Admin Rev. 2003; 63: 192-205.

13. Lomas J. Connecting research and policy. Isuma: Can J Policy Research. 2000;1(1): 140-144.14. Bouwman RJR. Research utilization in policy making. The utilization of the Dutch Health Care

Performance Report (Zorgbalans) in governmental policy making. Master Thesis. Bilthoven: RIVM; 2010.

15. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriersbetween epidemiological research and local health policy development. Health Res Policy Syst. 2010; 8:26.

16. Van Bon-Martens MJH, Achterberg PW, van de Goor IAM, van Oers HAM. Towards quality criteria for regional public health reporting: concept mapping with Dutch experts. Eur J Public Health. 2012; 22(3):337-42.

17. Bekker M, van Egmond S, Wehrens R, Putters K, Bal R. Linking research and policy in Dutch healthcare: infrastructure, innovations and impacts. Evid Policy. 2010; 6:237-253.

18. Goffman E. The presentation of self in everyday life. London: Penguin; 1990.19. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research

to enhance its impact. Health Res Policy Syst. 2012;10:21.



4

20. Hegger I, Janssen S, Keijsers J, Schuit A, van Oers J. Analyzing the contributions of a government-commissioned research project: a case study. Health Res Policy Syst. 2014;12:8.

21. Yin RK. Case Study Research Design and Methods. 4th rev ed. Thousand Oaks: SAGE Publications Inc; 2008.

22. Nugteren R, van den Berg MJ, Verschuuren M, Picavet HSJ, Verkleij HGM, Hoeymans N. Strategische analyse gegevensvoorziening VTV en Zorgbalans : Naar betere informatie over volksgezondheid en zorg. RIVM rapport 270261001. Bilthoven: RIVM; 2012.

23. Bekker M, Van der Grinten TED, Kelley E, Bensing J, Beuse R. Report of the external audit committee; Public Health Status and Forecast and Health Care Performance reports of the National Institute of Public Health and the Environment (RIVM). Bilthoven: RIVM; 2008.

24. Van Egmond S, Bekker M, Bal R ,van der Grinten T. Connecting evidence and policy: bringing researchers and policy makers together for effective evidence-based health policy in the Netherlands: a case study. Evid Policy. 2011;7(1):25-39.

Supplemental material

The online files are available at http://hsr.sagepub.com/supplemental

Chapter 5

Contributions of knowledge products to health policy:

a case study on the Public Health Status and Forecasts

Report 2010

Published: European Journal of Public Health 2016. 2016; 26(6):922-927

Ingrid Hegger

Maarten O. Kok

Susan W.J. Janssen

Albertine J. Schuit

Hans A.M. van Oers

R1R2R3R4R5R6R7R8R9


92 | Chapter 5

Abstract

Background

The Dutch Public Health Status and Forecasts report (PHSF Report) integrates research data

and identifies future trends affecting public health in the Netherlands. To investigate how

PHSF contributions to health policy can be enhanced, we analysed the development process

whereby the PHSF Report for 2010 was produced (PHSF-2010).

Method

To collect data, a case study approach was used along the lines of Contribution Mapping

including analysis of documents from the PHSF-2010 process and interviews with actors

involved. All interviews were recorded and transcribed ad verbatim and coded using an

inductive code list.

Results

The PHSF-2010 process included activities aimed at alignment between researchers and

policy-makers, such as informal meetings. However, we identified three issues that are

easily overlooked in knowledge development, but provide suggestions for enhancing

contributions: awareness of divergent; continuously changing actor scenarios; vertical

alignment within organizations involved and careful timing of draft products to create early

adopters.

Conclusion

To enhance the contributions made by an established public health report, such as the PHSF

Report, it is insufficient to raise the awareness of potential users. The knowledge product

must be geared to policy-makers’ needs and must be introduced into the scenarios of

actors who may be less familiar. The demand for knowledge product adaptations has to

be considered. This requires continuous alignment efforts in all directions: horizontal and

vertical, external and internal. The findings of this study may be useful to researchers who

aim to enhance the contributions of their knowledge products to health policy.


Case study Public Health Status and Forecasts Report | 93

5

Introduction

Public health status and forecasts report

The Dutch National Institute for Public Health and the Environment (Rijksinstituut voor

Volksgezondheid en Milieu, RIVM) has published a Public Health Status and Forecasts Report

(PHSF Report) every four years since 1993, most recently in 2014 [1]. The PHSF Report

integrates research data on public health and identifies future trends in public health in the

Netherlands.

Since the first edition, both the format and the focus of the PHSF report have changed

repeatedly, reflecting developments in public health. An important moment in PHSF history

was the establishment of its official status in the policy cycle by the Dutch Public Health Act

(Wet Publieke gezondheid) in 2002. The PHSF Report provides the policy themes for the next

step in this cycle: the publication of the ‘National Health Memorandum’ by the Public Health

department on behalf of the Minister of Health [2]. (figure 5.1)

RIVM: Public Health Status

and Forecasting (PHSF)

Ministry of Health: National Health Memorandum

Municipalities: Local PHSF and local

memoranda

Health Care Inspectorate:

‘State of Health Care’

Figure 5.1 Public Health Policy Cycle in The Netherlands. The PHSF Report provides the policy themes for the National Health Memorandum of the Minister of Health by law. Local PHSF reports are the basis for health memoranda of the municipalities and are developed by Community Health Services in a comparable process as the national PHSF.

R1R2R3R4R5R6R7R8R9


94 | Chapter 5

Another interesting development is the translation of the national PHSF Report into local

PHSF Reports by Community Health Services since 2006, in line with the decentralization of

health care to municipality level in the Netherlands [3].

Despite its established use for the National Health Memorandum, improvement of PHSF

contributions to health policy-making is still an issue. It remains challenging to use the report

as effectively as possible. Both RIVM and the Ministry of Health (MoH), Welfare and Sport

want the PHSF Reports to serve as a knowledge base for policy-makers; not only for policy-

makers of the PH department acting as the principal, but also for policy-makers of other

MoH departments. For this study, we formulated the following research question: What

improvements need to be made to the PHSF process in order to enhance PHSF contributions

tonationalhealthpolicyinthebroadsense?

Theoretical background

Science-policy relations have been described in a variety of theoretical models. Traditional

‘positivist’ models of knowledge utilization emphasize the importance of bridging the gap

between the policy-making and research domains. Weiss’ typology of knowledge use in

health policy contributed to awareness of the possible functions of knowledge products

in policy-making; as a source of data, ideas or arguments.4 An important notion is that

knowledge uptake is not as rational as is often assumed. In practice, policy-making is

incremental in nature and is a highly complicated process involving many actors and levels

of government. Theories on the complex policy process can help scientists who want to

contribute to policy-making to understand the role of knowledge in that process [5-8]. To

address this complexity, interaction models for knowledge use have been developed that

take into account the policy-makers’ needs and that focus on the interaction between

researchers and policy-makers, as reflected in the model developed by De Goede, e.g. [9].

The ‘constructivist’ perspective goes a step further by considering knowledge as a social

construct produced in a co-creation process [10, 11].

PHSF process

The PHSF Report has been described as an example of a successful ‘boundary object’

developed through co-creation and connecting the science and health policy domains,

while RIVM has been characterised as a ‘boundary organization’ in Guston’s terminology

[11, 12]. A range of actors participate in the PHSF process, including RIVM scientists, policy-



5

makers and scientists working in academia. RIVM scientists repeatedly interact with policy-

makers in the PHSF Policy Advisory Group (PAG) during the development process. Increasing

attention was devoted to alignment with stakeholders in successive PHSF processes. In her

analysis of the PHSF Report as a boundary object, Van Egmond has pointed out the role of

the PAG as a ‘backstage area’ where negotiations about the boundaries between science

and policy-making take place [13]. According to Goffman’s concept of ‘backstage work’ vs.

‘front stage presentation’, these informal negotiations enable the creation of an aligned

knowledge product that is acceptable to both the commissioning client and the research

organization and that can be officially presented to the outside world on ‘the front stage’

[14].

Study approach

To complement the already existing insights, we intended to develop practical

recommendations for researchers to improve the contributions made by the PHSF Report.

De Leeuw has observed that ‘current perspectives (both Knowledge Translation and the

Actor-Network Theory) do not reflect appropriately on actions that can be taken at the

nexus between research, policy and practice in order to facilitate more integration’ [15].

Kok and Schuit have developed the Contribution Mapping (CM) approach for analyzing

‘how’ knowledge is converted into action [16]. Kok and Schuit took the complexity of

knowledge production into account and conceptualized knowledge utilization as so-called

‘contributions to action’. A contribution is made when knowledge is included in the ‘actor

scenario’, representing the actor’s view of the future. Specific actions—so-called ‘alignment

efforts’—can be undertaken to align with the relevant actor scenarios and to enhance

the contributions of knowledge. To facilitate the analysis of alignment efforts, CM uses a

three-phase model of knowledge production: a formulation phase to define the research

question and approach, a production phase to conduct research, and an extension phase to

disseminate the research results [16]. (figure 5.2)

By using the CM approach to analyse the PHSF-2010 process, we expected to gain detailed

insights for the identification of areas that specifically require alignment efforts.

R1R2R3R4R5R6R7R8R9


96 | Chapter 5

Figure 5.2 Three-Phase Model by Kok and Schuit [16]. In the three-phase process model, investigators and linked actors interact during the research process. During the process, contributions to health policy can already be realized. The process narrows when the research approach is decided on. In the production phase, the process narrows again as knowledge products are realized. In the extension phase, the knowledge is disseminated [16]. Published before in reference [16].

Methods

To collect data on the PHSF-2010 process, we used a case study approach that included

analysis of documents related to the PHSF-2010 process and in-depth semi-structured

interviews with involved key actors at both RIVM and MoH who were able to survey

the process at different organizational levels [17]. Two RIVM researchers conducted the

interviews with RIVM experts and managers involved in the PHSF-2010 process in 2011 (n

=10).An independent researcher (notworking at RIVM) conducted the interviewswith

policy-makers and managers at the MoH in 2013 (n =10).All interviewswereconducted

using a list of topics based on the theoretical framework (Supplementary Materials), and

were recorded and transcribed ad verbatim. We coded the interviews in the Atlas-ti 7.1.3

software package using a deductive code list that was based on the theoretical framework

and inductively supplemented based on extensive and iterative analysis of the interviews.

In an open coding session, three researchers discussed the coding of two interviews

until consensus was reached. We identified areas for alignment efforts using a constant

comparative analysis method [18].



5

We operationalized the CM concepts as follows:

1. ‘Actor scenario’: actor’s view of health policy future and/or his/her professional

future

2. ‘Alignment efforts’: opportunities for actions taken by RIVM scientists, policy-

makers and managers of both RIVM and MoH to align the PHSF-2010 Report with

the needs of the MoH

3. ‘Knowledge’: the integration of research data on public health for the purpose

of the PHSF-2010 Report, the PHSF-2010 Reports (knowledge products), and the

exchange of knowledge through interaction between actors

4. ‘Contribution (to action)’: any indication that PHSF knowledge was included in an

actor scenario

5. ‘Formulation phase’: period for defining the PHSF-2010 scope (2006–7)

6. ‘Production phase’: period between agreement on the PHSF- 2010 scope and the

publication of the PHSF-2010 reports (2008–10)

7. ‘Extension phase’: period after publication in 2010, evaluated until 2013

Results

Alignment in the formulation phase

In the Formulation Phase, RIVM aligned with the policy-makers of the Public Health

department about the course to take and agreed to develop a main report and four thematic

reports, all to be delivered in 2010.

Until 2006, the PHSF Report was mainly used by the PH department. Both RIVM and PH

policy-makers expected that co-financing by other departments would be favourable for

involving them more directly in the PHSF process and strengthening PHSF contributions in

the broader health policy domain. For the first time, the Macroeconomic Issues and Labour

Market Department (MILM) provided a budget that was used to develop a thematic report

on the societal benefits of health care [19].

Alignment in the production phase

Once the Production Phase started, the PAG was formed with senior policy-makers of the

relevant MoH departments. (figure 5.3)

R1R2R3R4R5R6R7R8R9


98 | Chapter 5

The RIVM project managers regularly consulted the PAG and experienced that different

departments had varying views and needs. The PH director, who also served as the PAG chair,

challenged RIVM to describe implications for health policy. His opinion was that facts need

interpretation to contribute to policy-making: ‘If RIVM only publishes reports that provide

facts, but do not reflect upon the implications, the policy impact of the document will be

zero and it will be ignored. So it needs to provide a trigger... that is the art of balancing, you

should not cross the line, but you have to provide a trigger and ensure that the message is

not lost.’

However, MILM policy-makers appeared not to be in favour of near-to-policy statements by

RIVM and as new co-financier, they interfered in discussions about the content of the PHSF-

2010 Report. For example, the interpretation of socio-economic health inequalities (SEHIs),

a highly political issue, appeared to be controversial. As one policy-maker mentioned:

‘I argued over that issue (the draft text on SEHIs) with X (RIVM researcher) by phone, since I

believed that the text contained too many political opinions.’

Minister of Health, Welfare and Sport

State Secretary for Health, Welfare and Sport

Directorate General of Public Health Directorate General of Curative

CareDirectorate General of Long-

term Care

Sports Department

Public Health Department (PH)

Healthcare Disciplinary Boards Secretariat Unit (EST)

Secure Youth Care Implementation Unit (UGJ)

Curative Care Department

Pharmaceuticals and Medical Technology Department

Market and Consumer Affairs Department

Long-Term Care Department

Social Support Department

Health Insurance Department

Department represented in the Policy Advisory Group of PHSF-2010

Macroeconomic Issues and Labour Market Department (MILM)

Advisory and support departments

Secretary GeneralDeputy Secretary General

Nutrition, Health Protection and Prevention Department

Figure 5.3 Organization Chart of the MoH, Welfare and Sport in the Netherlands. The departments indicated in italic underlined characters were represented in the PAG of the PHSF-2010 process.



5

Due to time constraints, the thematic reports were completed at the last minute just before

the strict deadlines. The PAG members had hardly any time to read the drafts, hindering

discussion on the content. According to a PAG member, this resulted in a missed opportunity

for alignment: ‘We finally received the thematic reports, which contained hundreds of

pages, at the last minute. In the first place, this gave us the feeling that we could not make

any changes or recommendations, since the process was already completed and the reports

were already finished. In the second place, this also influenced the impact of the thematic

reports later on (....). If the PAG members, the ‘early adopters’, discuss the reports, then it will

sink in properly and that will lead to further dissemination.’ Timely delivery of knowledge

products is a well-known factor for successful contributions. Although the PHSF-2010 Report

was completed in time, we found that timely draft versions are also important to take full

advantage of backstage opportunities in order to create support and enhance contributions.

RIVM professionals recognized that familiarity with the MoH organization is important for

successful alignment. As one researcher said: ‘I think you need to know how things work at

the Ministry of Health. You need to know what they are dealing with. You need to understand

their issues, without fully identifying with them.’ One respondent left RIVM and joined the

Ministry of Health and found out that he did not know the Ministry as well as he thought he

did: ‘When I worked at RIVM, I thought I knew how things worked (at the Ministry). However,

when I myself was employed by the Ministry for four-and-a-half years, I discovered that in

fact I knew very little about it. (...) I had never realized that before, because at RIVM I was

one of the few people who maintained very close contacts with the Ministry. People thought

I knew what went on there.’ This finding underlines the need for continuous attention to

evolving actor-scenarios and policy-makers’ needs.

Another important finding was that effective alignment requires consistent alignment

efforts at every hierarchic level, as well as conscious vertical alignment within the

involved organizations. Since the PHSF is the formal starting point for the National Health

Memorandum, alignment at all hierarchical levels between RIVM and MoH should be a

matter of course. However, coordination was lacking due to limited internal alignment. The

RIVM project managers acted independently in line with the RIVM matrix structure, while

senior management remained distant.

R1R2R3R4R5R6R7R8R9


100 | Chapter 5

Alignment in the extension phase

The ‘front stage’ presentation of the PHSF Report to the Minister took place at an event

where the Health Care Inspectorate’s ‘Staat van de Gezondheidszorg 2010’ Report (State of

Health Care-2010) was also presented. Immediately afterwards, senior RIVM management

misrepresented the PHSF-2010 message to the press by confusing it with the Inspectorate’s

message. This was caused by insufficient familiarity with the PHSF-2010 content as a

result of lacking vertical alignment. Articles appeared in the press claiming that the report

had recommended ‘strong government measures to promote a healthy lifestyle’, and

had concluded that ‘many preventive health campaigns are ineffective’. These messages

conflicted with the intended optimistic PHSF-2010 messages as aligned with the MoH.

The media reports caused inconvenience for MoH policy-makers and they experienced

difficulties in obtaining funding for preventive campaigns for some time. One policy-maker

complained: ‘There was no attention paid to the PHSF messages and for about six months to

a year, we had to repair the situation with Parliament.’

After publication, the focus at the MoH quickly shifted from the PHSF Report to the issues

of the day. Alignment during the production phase is regarded as important for early

knowledge transfer and the creation of ‘early adopters’. However, many of the key actors

(coincidentally) changed in the PHSF-2010 extension phase. This included several PAG

representatives, the PHSF project managers and senior managers (both at RIVM and MoH).

Their successors were less involved in the PHSF-2010 process or not involved at all, which

illustrates the ongoing need for alignment efforts in the extension phase although hardly

any budget was available for these activities. As one policy-maker put it: ‘RIVM should work

together with the PAG and should carefully examine how best to promote the PHSF Report

both within and outside the MoH after the report’s presentation. We should remind people

where and how to find the report, and how it might be valuable to them.’

Contributions to policy-making

The PHSF-2010 Report made a key contribution to the National Health Memorandum

published in 2011 [2], and both PH policy-makers and RIVM project managers were very

pleased with this. As one project leader put it: ‘I am pleased with the contribution to the

Memorandum. Of course, there are always points for improvement. However, the basic

PHSF-2010 issues and ideas were included.’



5

The PHSF-2010 contributions to other health policy domains were less self-evident than

RIVM expected. Some PAG members indicated that they personally appreciated the PHSF-

2010 Report for the overall picture it provided of the public health situation, but that their

department needed information the PHSF-2010 Report did not offer, such as more specific

figures and facts. One policy-maker said: ‘They (RIVM) have all sorts of products and are very

supply-oriented. It’s as if they want to draw attention to their allegedly amazing products

without really knowing who they are talking to. To put it frankly, they imply that you would

be stupid not to use their data and products. However, in my view, you have to approach it

the other way round. I don’t have to use those products at all. I just need access to the right

information, and their task is to help me in whatever way possible.’ Researchers must realize

that actors have their own actor scenarios which may have implications for the process,

the concept, and the format of the knowledge to be produced. This again underlines the

importance of thorough familiarity with the target group, which requires a combination of

horizontal and vertical alignment efforts.

Discussion

Study limitations

Because the interviews were conducted in retrospect, recollection bias may have played a

role. However, we noted that the interview structure and the Three-Phase Model facilitated

recollection. Furthermore, documents, such as minutes of PAG meetings and the PHSF-2010

evaluation report were used for triangulation purposes in combination with the interviews.

The number of interviewees and the representation of all hierarchical levels amongst

respondents were sufficient to gain a good overview of the research process.

Reflections on the CM approach

We used CM to analyze how alignment influenced the PHSF Report’s contributions to health

policy-making. The analysis required a lot of time, making it unfeasible for routine evaluation

of research projects. However, we gained useful insights that are generally applicable, both

in our own organization and in other comparable knowledge institutes. We secured our

findings for the RIVM organization by designing a tool to support researchers in reflecting

on the research process, taking into account the important areas for alignment efforts. A

research article on this tool is currently being prepared.

R1R2R3R4R5R6R7R8R9


102 | Chapter 5

Conclusions

The analysis of the PHSF-2010 process using the CM approach produced several

recommendations for improving the contributions of the PHSF Report to health policy-

making. The PHSF-2010 process included intensive alignment efforts, such as interaction in

PAG meetings. However, we noticed three issues that need careful and specific alignment

efforts:

Actors and their own needs: organizational environment

To be able to decide what needs to be done to enhance contributions, researchers must

explore the actor scenarios of key actors. Since individuals can have a major impact and

actors change all the time, the analysis has to be updated on a regular basis to make sure

new scenarios are included.

Involvement of all hierarchical levels: vertical alignment

To steer the process in the right direction and to create the support needed, monitoring

of adequate vertical alignment is essential. We already noticed this issue in two other

case studies and it is clearly an ongoing challenge to manage vertical alignment for large,

hierarchical organizations like RIVM and MoH [20, 21].

Creation of early adopters: timing of draft products

Policy-makers need sufficient opportunities to comment on draft knowledge products to

enable them to become ‘early adopters’. This requires thorough process management and

researchers should consider spending part of their research budget for this purpose.

Regular actor scenario analyses, ongoing vertical alignment and timely draft products are

issues not exclusively related to the PHSF-2010 process, but also generally applicable to

the process of many other reports for policy support; also at other governance levels and

even in other countries. The general lesson to be learned from this study is that it is not

enough to raise awareness of a well-established knowledge product in order to enhance

its contributions to policy-making. This requires also regular exploration of the actor

scenarios of (sometimes less familiar) actors, and consideration of necessary knowledge

product adaptations. And in turn, this process will require alignment efforts in all directions:

horizontal and vertical, external and internal.



5

References1. Hoeymans N, Van Loon AJM, Van den Berg M, et al. Een gezonder Nederland, kernboodschappen

van de Volksgezondheid Toekomst Verkenning 2014. (Towards a healthier Netherlands. Key findings of the PHSF-2014). Bilthoven: RIVM; 2014.

2. Ministry of Public Health Welfare and Sport. Landelijke Nota Gezondheidsbeleid ‘Gezondheid Nabij’ (National Health Memorandum ‘Health Nearby’). The Hague, the Netherlands: VWS; 2011.

3. De Goede J, Steenkamer B, Treurniet H, Putters K, van Oers H. Public health knowledge utilisation by policy actors: an evaluation study in Midden-Holland, the Netherlands. Evid Policy. 2011; 7(1): 7-24.

4. WeissC.Policyresearch:data,ideasorarguments?In:WagnerP,WeissCH,WittrockB,WollmanH. editors. Social Sciences and Modern States. Cambridge: Cambridge University Press, 1991.

5. Kingdon J. Agendas, Alternatives, and Public Policies. 2nd edn. New York: Longman, 2003.6. Sabatier P. Theories of the Policy Process. 2nd edn. Boulder: Westview Press, 2007.7. Sabatier PA. An advocacy coalition framework of policy change and the role of policy-oriented

learning therein. Policy Sci. 1988;21:129–168.8. De Leeuw E, Clavier C, Breton E. Health policy-why research it and how: health political science.

Health Res Policy Syst. 2014;12:55.9. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriers

between epidemiological research and local health policy development. Health Res Policy Syst 2010;8:26.

10. Wehrens R. Beyond two communities-from research utilization and knowledge translation to co-production?PublicHealth 2014;128:545–551.

11. Bekker M, van Egmond S, Wehrens R, Putters K, Bal R. Linking research and policy in Dutch healthcare: infrastructure, innovations and impacts. Evid Policy 2010;6(2):237–253.

12. Guston DH. Boundary organizations in environmental policy and science: an introduction. Sci Technol Hum Values 2001;26:399–408.

13. Van Egmond S, Bekker M, Bal R, van der Grinten T. Connecting evidence and policy: bringing researchers and policy makers together for effective evidence-based health policy in the Netherlands: a case study. Evid Policy 2011;7:25–39.

14. Goffman E. The Presentation of Self in Everyday Life. London: Penguin; 1990.15. De Leeuw E, McNess A, Crisp B, Stagnitti K. Theoretical reflections on the nexus between

research, policy and practice. Crit Public Health 2008;18:5–20.16. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research

to enhance its impact. Health Res Policy Syst 2012;10:21.17. Yin RK. Case Study Research Design and Methods. 4th rev ed. Thousand Oaks: California, SAGE

Publications Inc, 2008.18. Pope C, Ziebland S, Mays N. Analysing qualitative data. Br Med J 2000;320:114–6.19. Post NAM, Zwakhals SLN, Polder JJ. Maatschappelijke baten. Deelrapport van de VTV 2010 Van

gezond naar beter (Social Benefits. Thematic Report of PHSF-2010 ’Towards better health’). RIVM Report. Bilthoven: RIVM, the Netherlands; 2010.

20. Hegger I, Janssen S, Keijsers J, et al. Analyzing the contributions of a government- commissioned research project: a case study. Health Res Policy Syst 2014;12:8.

21. Hegger I, Marks LK, Janssen SW, et al. Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report. J Health Serv Res Policy 2016;21:29–35.

R1R2R3R4R5R6R7R8R9


104 | Chapter 5

Supplementary data

Supplementary data are available at EURPUB online.

Chapter 6

ResearchforPolicy(R4P):Developmentofareflection

tool for researchers to improve knowledge utilization

Published: Implementation Science 2016; 11(133)

Ingrid Hegger

Lisanne K. Marks

Susan W.J. Janssen

Albertine J. Schuit

Jolanda F.M. Keijsers

Hans A.M. van Oers

R1R2R3R4R5R6R7R8R9


106 | Chapter 6

Abstract

Background

To improve knowledge utilization in policy-making, alignment between researchers and

policy-makers during knowledge production is essential, but difficult to maintain. In three

previously reported case studies, we extensively evaluated complex research projects

commissioned by policy-makers to investigate how alignment is achieved in a research

process and to discover ways to enhance knowledge contributions to health policy. In the

present study, we investigated how the findings of these three research projects could be

integrated into a practical tool for researchers to enhance their contribution to evidence-

based policy.

Methods

A cross-case analysis was conducted to integrate the findings of the evaluation of the three

research projects and to identify important alignment areas in these projects. By means of

an iterative process, we prepared a tool that includes reflection questions for researchers.

The “Research for Policy” tool was tested with input from the project managers of three

new research projects. Based on the findings, the final version of the Research for Policy

tool was prepared.

Results

By cross-case analysis of the three case studies, the following important alignment areas

were identified: the goal, quality, relevance, timing, and presentation of research, the

tasks and authorities of actors, the consultative structure and vertical alignment within

organizations, and the organizational environment. The project managers regarded the

Research for Policy tool as a useful checklist for addressing the important alignment areas in

a research project. Based on their feedback, the illustrative examples from the case studies

were added to the reflection questions. The project managers suggested making the tool

accessible not only to researchers but also to policy-makers. The format of the Research for

Policy tool was further adjusted to users’ needs by adding clickable links.

Conclusions

Alignment between research and policy-making requires continuous efforts and a clear

understanding of process issues in the research project. The Research for Policy tool offers

practical alignment guidance and facilitates reflection on process issues, which supports

researchers in aligning with policy-makers and in acting in a context-sensitive way.


Research for Policy tool | 107

6

Background

Policy-making is a complex process, and the use of scientific knowledge in policy-making, also

referred to as knowledge utilization, is not self-evident [1–3]. Not surprisingly, researchers

want to know why the scientific knowledge uptake into policy-making is so difficult. For

over 40 years, extensive research has been conducted on knowledge utilization, which

resulted into several generations of theoretical models explaining barriers and facilitators

[4–7]. Oliver et al. have pointed out that despite extensive research on knowledge utilization

and many efforts to enhance scientific knowledge uptake in policy, barriers have been

persistently identified in the literature, whereas only “little empirical data analyzing the

processes and evidence use in policy is available” [8]. They argue that researchers should

focus on understanding policy-making processes, the types of evidence used by policy-

makers, and the relationship between research and policy-making.

The first step for scientists is to recognize the characteristics of politics, policy, and

policy-making. As Pielke describes, politics is the “process of bargaining, negotiation, and

compromise” to determine how resources are allocated and under what conditions, whereas

“policy” indicates “the commitment of a group to a particular course of action” aiming at

a desired outcome for solving a specific problem [1]. Policy-making is thus the process to

recognize the problem and to formulate, implement, and evaluate the particular course of

action needed to solve the problem. To understand which scientific knowledge may help

policy-makers and contribute to policy-making and how this can be achieved, scientists

have to acknowledge that policy-making is a dynamic, complex process in which different

sources of knowledge and information are used and where the values and interests of

various stakeholders have to be taken into account [7]. The components of the process, i.e.,

agenda setting of a problem, followed by formulation, implementation, and evaluation of

policy, are often represented as successive stages in a policy cycle. However, policy-making

is shown to be less linear and more iterative in practice [9–11]. In this complex process,

scientific knowledge may have an indirect, not clearly discernible role: besides instrumental

use in a direct and specific way to solve a particular problem, it may also have a conceptual

function as a source of ideas or a symbolic/agenda-setting function when it is used to take

an advocacy position [12–15]. Because of the dynamics of the policy-making process and

the different purposes for which scientific evidence is used, it is generally recognized that

formal and informal interactions between researchers and policy-makers play a key role in

the use of knowledge in policy-making [16]. The constructivist perspective on knowledge

R1R2R3R4R5R6R7R8R9


108 | Chapter 6

utilization even goes a step further. In this perspective, science is a social process during

which scientific evidence is co-created by researchers and other involved actors, including

policy-makers [17]. As Kok and Schuit argue, this constructivist model is helpful in blurring

the boundaries between research and policy-making and discovering ways to align research

with policy-making in order to enhance knowledge contributions to policy-making [17].

Policy and policy-making may relate to various decision-making levels (national, local,

governmental, or private). In this paper, the focus is on research for governmental policy-

making by officials of a ministry or other governmental organizations at a national level. For

scientists working in this context, the different perspectives on knowledge and the insights

on knowledge utilization are particularly relevant. An example of a knowledge institute

working for governmental organizations at national level is the National Institute for Public

Health and the Environment (RIVM), the national public health institute in the Netherlands.

RIVM conducts research and integrates knowledge in the field of public health, health care,

safety, and environmental protection for governmental commissioning organizations such

as the Ministry of Health, Welfare and Sport and the Healthcare Inspectorate [18]. RIVM

and its clients have arranged formalized procedures to ensure agreement on research

proposals, knowledge products, and timelines for commissioned projects. The knowledge

products are important for making RIVM knowledge accessible to the target audience.

They may take a variety of forms, such as reports, scientific papers, fact sheets, websites,

or databases. In the RIVM annual commissioning cycle, RIVM researchers, specifically the

project managers, consult the policy-makers at the commissioning organization. These

policy-makers are their counterparts for a particular research commission and discuss the

articulation of the knowledge question and formulation of the project plan. They have the

task to arrange regular meetings with the commissioning client’s account manager and to

gear their knowledge products to the commissioning client’s needs. RIVM management

regularly monitors the progress of research projects and the timely delivery of products.

Despite these efforts to interact and align with the commissioning organizations, RIVM

researchers have found that the contributions of their knowledge products to policy-making

are not as significant as they expected. At the same time, commissioning organizations still

indicate that they would like to be provided with knowledge products that are optimally

aligned to their specific needs.



6

To investigate how alignment is reached and to discover ways of enhancing the contributions

of RIVM knowledge to policy-making, we conducted a research project with two stages.

In the first stage, we carried out three case studies, for which we used Contribution

Mapping developed by Kok and Schuit [17, 19]. The case studies have been reported in

three published scientific articles [20–22]. In the second stage of our research project, we

translated the findings of the case studies into practical guidance by developing a reflection

tool to support researchers in their alignment efforts. This second stage of the research is

the subject matter of this paper. To provide background, we first summarize the findings of

the first stage of our research project. We then continue with reporting the conduct and

findings of the second stage.

Summary stage 1: three case studies

In the first stage of the research (2011–2013), we extensively evaluated three multi-

annual RIVM research projects: “Development of Risk Model,” “Development of Dutch

Health Care Performance Report 2010,” and “Development of Dutch Public Health Status

and Forecasts Report 2010”. For each project, we analyzed the process in detail using the

Contribution Mapping approach [17]. Based on a constructivist perspective, Contribution

Mapping developed by Kok and Schuit is a method to evaluate the utilization and impact of

knowledge generated by research projects in health policy-making. The method enhances

the understanding of research processes and alignment between researchers and health

policy-makers. It is based on a three-phase model of the research process; a Formulation

Phase to define the research question and plan, a Production Phase to conduct research,

and an Extension Phase to disseminate the research results. (Figure 6.1)

Contribution Mapping conceptualizes the utilization and impact of knowledge through

so-called contributions to action: “Contributions are activities that enable the conversion

of knowledge into an element in decisions and implementation, a part of practices or a

component in innovation” [17]. We consider the concept contribution valuable since it

reflects that knowledge conversion into policy-making can take many, even very subtle,

shapes and forms, already during the research process. The more comprehensive concept

knowledge utilization may suggest that it is about the complete uptake of certain knowledge

in policy-making, which may disguise partial, but meaningful use. To contribute to the work

of actors, i.e., the persons or organizations involved, knowledge has to be included in the

so-called actor scenarios. These are (virtual) scripts implicitly or explicitly formulated by the

R1R2R3R4R5R6R7R8R9


110 | Chapter 6

actors representing their view of the future and their pursuit. Alignment is an important

concept in Contribution Mapping and means that research and policy-making are attuned

and this is reached by reciprocal interaction instead of a one-way interaction from research

to policy-making. To enhance the contributions of knowledge, alignment with actors and

their actor scenarios is necessary. Specific actions can be taken for this purpose, so-called

alignment efforts, defined by Kok and Schuit as “anticipatory efforts that aim to enhance

contributions” [17]. Both researchers and policy-makers can undertake alignment efforts.

Due to some push and pull in this process, both positions may move resulting in knowledge

and knowledge products (better) attuned to the needs of policy-makers and researchers.

Figure 6.1 Kok and Schuit’s three-phase model [17]

In each case study, we described the alignment efforts during the research process and the

contributions that the project made to the policy-making process [20–22]. An outline of the

three case studies is provided in Table 1 below. For further details, we refer to the research

articles on the case studies [20–22].



6

Table 6.1 Outline of case studies

“Development of Risk Model” case study [20]This case study focused on the development of a risk-based approach for clinical trial inspections in the Netherlands and had to deliver risk models to enable ranking and stratified selection of clinical trials for inspection by the Dutch Health Care Inspectorate. These models had to contribute to the Inspectorate’s objective of using scientific knowledge for evidence-based supervision. We found that RIVM and the Inspectorate had divergent views on their collaboration and the ownership of the knowledge product, which resulted in different expectations. Researchers and commissioning inspectors were not aware of these different perceptions. We identified six relevant categories of both horizontal alignment efforts (between investigators and key users) and vertical alignment efforts (within RIVM and the Inspectorate organization) that affected the contributions to the Inspectorate’s work. Relevant alignment efforts became manifest at three levels: the first level directly concerned the project, the second level concerned the organizational environment, and the third level concerned the formal and historical relationship between the organizations.

Case study on Dutch Health Care Performance Report [22]The second case study concerned the Dutch Health Care Performance Report (DHCPR). The DHCPR is published by RIVM and commissioned by the Dutch Ministry of Health, Welfare and Sport in 2006, 2008, 2010, and 2014 [27]. Based on a scientific framework, the DHCPR monitors health care performance in the Netherlands by using indicators for quality, accessibility, and affordability. The aim of the report is to contribute to “strategic policy-making”. We identified six areas where alignment is specifically relevant for enhancing the contributions of future DHCPR editions: well-balanced information for different ministerial directorates, backstage work, double-role actors, reports published by other knowledge institutes, data collection and generation, and presentation formats.

Case study on Public Health Status and Forecasts Report [21]The third case study concerned the Dutch Public Health Status and Forecasts Report (PHSF), which integrates research data and identifies future trends in public health in the Netherlands [28]. The PHSF has a recognized function in connecting the science and policy domains because it is embedded in the national health policy cycle by law. The PHSF provides the policy themes for the National Health Memorandum (NHM), which is published every 4 years by the Minister of Health, Welfare and Sport. The PHSF2010 process included activities aimed at alignment between researchers and policy-makers, such as informal meetings. However, we identified three issues that are easily overlooked in knowledge production, but provide suggestions for enhancing contributions: awareness of divergent, continuously changing actor scenarios; vertical alignment within the organizations involved; and careful timing of draft products to create early adopters.

Purpose of this article

In the first stage of our research project, we used the Contribution Mapping method in three

case studies to gain insights into areas where specific alignment efforts can be helpful to

enhance contributions of RIVM knowledge and knowledge products.

R1R2R3R4R5R6R7R8R9


112 | Chapter 6

In the second stage, we translated the findings of the first stage by developing a tool to

support researchers in their everyday work, since the insights from the case studies did not

directly provide guidelines for improving alignment within future projects. The case studies

had shown that alignment between researchers and policy-makers is more difficult to

maintain in practice than expected. Moreover, the need for alignment efforts continuously

changes, depending on the research project and its current phase. We concluded that

the key to enhancing alignment efforts and therefore the contributions of research is the

researchers’ awareness of key alignment areas, combined with regular, systematic reflection.

This involves taking sufficient time to consider seriously the current situation of the project,

taking into account the alignment areas [20, 22].

We acknowledged that it would not be helpful to offer generic alignment efforts, since every

research project is different. Alignment efforts are difficult to predefine in a tool since they

depend on the specific research process (both the knowledge question and the project’s

context, i.e., the environment of the project that influences the process and outcome of

the project) at a certain point in time, making it impossible to suggest beforehand any

detailed (inter)actions applicable to each research project. Furthermore, we concluded

that the continuously changing context of research projects requires regular reflection

on the research process to identify the need for specific alignment efforts. We found that

researchers find it difficult to analyze regularly the process issues of their project, often due

to unawareness of important alignment issues, time constraints, and their preference for

scientific issues.

In this article, we report the results of the second stage of our research project. Firstly,

we describe how we integrated the findings of the three case studies of the first stage by

conducting a cross-case analysis. Secondly, we describe how we developed the Research for

Policy (R4P) reflection tool to support researchers in creating alignment with policy-makers

and preparing aligned knowledge products. For the formulation of the specifications of the

R4P tool, we took into account the insights acquired in the first stage. The process for the

development of the R4P tool is outlined in Fig. 2.



6

Con

tribu

tion

Map

ping

C

ase

stud

y 2

Ste

p 1

Ref

lect

ion

ques

tions

Form

ulat

ion

Phas

e

Ste

p 2

Test

ing

Feed

back

from

proj

ect m

anag

ers

Alig

nmen

tef

forts

Alig

nmen

tef

forts

Alig

nmen

tef

forts

Eig

ht

alig

nmen

t ar

eas

Con

tribu

tion

Map

ping

C

ase

stud

y 1

Con

tribu

tion

Map

ping

C

ase

stud

y 3

Fina

l R4P

tool

w

ith e

xam

ples

and

cl

icka

ble

links

Step

3R

efle

ctio

nqu

estio

nsP

rodu

ctio

n P

hase

Ste

p 4

Test

ing

Feed

back

from

proj

ect m

anag

ers

Cro

ss-c

ase

anal

ysis

Dev

elop

men

t of R

4P to

ol

Ste

p 5

Fina

l ver

sion

of

the

R4P

tool

Thre

e ca

se S

tudi

es

Figu

re 6

.2 P

roce

ss o

f the

dev

elop

men

t of t

he R

4P to

ol

R1R2R3R4R5R6R7R8R9


114 | Chapter 6

Part A: Cross-case analyses

Methods

We systematically analyzed the alignment areas of the three cases as identified in the first

stage of our research project by a case-oriented approach [23]. In line with the Contribution

Mapping approach, we described the alignment areas in a table for comparison and drew

up a comprehensive list of important alignment areas [17, 19].

Results

The cross-case analysis of the three case studies revealed eight areas where deliberate

alignment efforts could specifically improve alignment between RIVM and the commissioning

Ministry of Health, Welfare and Sport (see Table 2). Based on the barriers indicated by de

Goede et al. and general alignment efforts formulated by Kok and Schuit, the alignment

areas “goal,” “quality,” “consultative structure,” “relevance and timing,” and “presentation”

were anticipated findings [16, 17]. The alignment areas “tasks and authority,” “vertical

alignment”, and “organizational environment” were additional areas.

Table 6.2 Consolidated list of areas for alignment

Area for alignment Topics

Goal The formulation of the knowledge question; exploration of its origin, the “question behind the question,” and the underlying need for the knowledge products

Tasks and authority The input of all involved actors (both researchers and policy-makers); their responsibilities, knowledge and data exchange by actors during the process, and the final authority over the knowledge products

Quality The research method; conceptual framework and data used in the research project

Consultative structure The consultative structure of the project; the sharing of relevant information and the relationships between actors; double-role actors

Vertical alignment Interaction within the organization conducting research and within the commissioning organization; interaction between hierarchical levels and the embedding of the project in the organizations



6

Organizational environment The environment of the research project; awareness of relevant conditions external to the research project influencing the relationship between investigators and linked actors; incidents, media events, relationships with other organizations, changing priorities, and changing actors

Relevance and timing The formulation and wording of the research results and timing of the delivery and presentation of the knowledge products

Presentation The design and structure of knowledge products and the tools for the extension strategy

The alignment areas represent the topics where researchers should reflect on to achieve

sufficient alignment before and during the research project. In this way, they will be able to

determine whether they should put more effort into alignment.

• Alignment area Goal represents the need for full clarity regarding the expectations

about and the purpose of the knowledge product.

• Alignment area Tasks and Authority indicates that a project manager should define

the tasks and responsibilities within his/her research team and should also assure

a clear agreement on the rights and responsibilities of both the commissioning

organization and the research institute within the project to avoid debate at a later

stage.

• Alignment area Quality implies that the scientific models, concepts and definitions

used during the research can have a significant bearing on the findings and need

agreement already at the start. This is to avoid dismissal of an unfavorable research

result by arguing that the approach was flawed.

• Alignment area Consultative Structure focuses on the need to ensure both capacity

and time to allow for adequate interaction during the research process. There is

also attention for planning exchange of knowledge with intended users.

• Alignment area Vertical Alignment stresses the importance of vertical embedding

of the product within the organization for its legitimacy towards the commissioner

and the resources and capacity required. It is essential to refer any problems,

difficulties, or obstacles, which cannot be resolved at the research level to a

higher hierarchical level. Conversely, the higher hierarchical level should inform

researchers on specific issues that may influence there project.

R1R2R3R4R5R6R7R8R9


116 | Chapter 6

• Alignment area Organizational Environment is about identification of the

characteristics of the different involved organizations. The institutional work culture

of the commissioning organization may influence the process and the dynamics of

its context are important to understand the commissioner’s motive. Furthermore,

developments in the own organization should be well known and understood.

• Alignment area Relevance and Timing points out that the commissioner’s

requirements with regard to the purpose, form, and timing of the product may

change over time, even after approval of the original project proposal. Any changes

to the process itself may then be necessary. Furthermore, awareness about any

political sensitivity with respect to wordings and formulation of research results is

important.

• Alignment area Presentation reflects the need to agree timely on the form and

presentation of the knowledge products and their dissemination.

Part B: Development of the R4P Tool

Methods

We developed a reflection tool for researchers called R4P with the purpose to support

researchers in reflecting on their project to identify the need for alignment efforts during the

research process. Based on the findings in the first stage as described in the “Background”

section, we formulated the following specifications:

• The R4P tool should support users in engaging in systematic reflection on the

research process.

• The R4P tool should take into account the alignment areas identified in the case

studies.

• The R4P tool should be generally and easily usable at project team level.

In order to comply with the specifications, we decided to include open-ended questions

that would promote awareness and critical reflection on the process aspects of the project,

while taking into account the alignment pitfalls that we identified in the case studies by

contribution mapping. In this way, we intend to increase researchers’ sensitivity to the

context of their project and to the policy-making process. As they become more context-

sensitive, researchers will be more capable of recognizing and acting on the importance of

alignment with the commissioning and other organizations, in order to enhance the impact

of their work. At the same time, they will be more capable of acknowledging the different



6

roles and responsibilities of researchers and policy-makers, and taking into account the

need to stay independent.

Step 1: Preparing the first part of the R4P tool

We first prepared the part of the R4P tool, which focuses on the Formulation Phase

of a project. In iterative, open sessions, our research team, consisting of the two acting

researchers and their supervisors (two senior researchers and two university professors),

discussed the formulation and categorization of the open-ended reflection questions and

the accompanying explanation for each question until consensus was reached. They were

concisely formulated and we added a brief explanation to each question to provide some

background concerning the question (Additional file 2: Draft R4P tool, part Formulation

Phase).

Step 2: Testing the R4P tool during the Formulation Phase

We asked three experienced project managers who work on policy-orientated research

projects in the field of public health and health care to test the Formulation Phase questions

of the R4P tool for usability in their projects during the upcoming new project cycle (2013/

2014). In August 2013, each project manager received a verbal explanation of the R4P tool.

They were asked to use the tool in the Formulation Phase of new projects for the year 2014.

After 3 months, they were interviewed face to face and asked to provide feedback on the

tool (Additional file 1: Topic list). Each semi-structured interview took approximately 1 h and

was recorded and analyzed by mapping.

Step 3: Developing an extension of the R4P tool for the Production Phase

For use in the Production Phase, we developed an extension of the R4P tool in a similar

process as the development of the Formulation Phase part. Based on the project managers’

feedback on the part for the Formulation Phase, we added examples to the Production

Phase questions. The examples illustrate the rationale of each question and concretize the

rather general wording of the questions (Additional file 2: Draft R4P tool, part Production

Phase).

Step 4: Testing the extension of the R4P tool during the Production Phase

Again, we asked the same three experienced project managers to use the R4P tool in the

production phase of their project during the year 2014. They were interviewed face to

R1R2R3R4R5R6R7R8R9


118 | Chapter 6

face and asked to provide feedback on the tool (see Additional file 1: Topic list). Each semi-

structured interview took approximately 1 h and was recorded and analyzed by mapping.

Step 5: Preparing the final version of the R4P tool for all phases

To complete the R4P tool, we integrated the Formulation Phase and Production Phase

questions and added examples from our case studies to the questions of the Formulation

Phase. Based on the project managers’ feedback, we decided to organize the questions

according to content rather than by project phase. We classified the questions into four

categories: organizational environment of the project (I), goal of the project (II), interaction

during the project (III), and outcome of the project (IV).

In the final version of the R4P tool, we provided an overview to indicate in which phase of

the research project each question could be relevant, and we included clickable links to the

questions. The questions themselves also contain clickable links to explanatory notes to the

questions, as well as corresponding examples (Appendix Chapter 6: Additional file 3: Final

R4P tool).

Results

Step 2: Testing the R4P tool during the Formulation Phase

In step 2, we tested the part for the Formulation Phase, which included ten questions in

five categories (Additional file 2: Draft R4P tool, part Formulation Phase). When providing

their feedback on this part of the draft R4P tool, the project managers regarded the topics

raised in the open-ended questions as the key topics to address during the research process.

However, they indicated that they experienced the questions as rather abstract, which made

it more difficult to link them to their daily practice. Therefore, we added examples to the

question for the Production Phase.

Step 4: Testing the extension to the R4P tool during the Production Phase

The project managers considered the examples provided in the Production Phase questions

to be very helpful in clarifying the questions and in finding an approach for alignment in

comparable situations. They pointed out that they experienced some overlap between

the Formulation Phase and Production Phase questions and stated that they regarded

the tool as already suitable for (anticipating on) the extension phase. They were in favor

of a single integrated R4P tool with questions for all phases. After all, they considered

the R4P tool a useful checklist for addressing the important process topics of a research



6

project. They indicated that the R4P tool should be readily accessible not only to health-

systems researchers but also to researchers in other expertise domains and to RIVM are

commissioning clients, such as policy-makers working at the Ministry of Health, Welfare and

Sport. The project managers suggested using the R4P tool for education of project managers

as part of the regular project management courses. They also suggested making the tool

more user-friendly, for example, by making it available online and adding clickable links.

Step 5: Final version of the R4P tool for all phases

Based on the feedback of the project managers, we decided to merge the two parts of the

R4P tool into one list of questions for all phases. The final version of the R4P tool contains

23 reflection questions and is presented in an additional file (Additional file 3: Final R4P

tool). The list of questions contains clickable links to explanatory notes to the questions and

corresponding examples. The examples have been derived from our case studies and reflect

real-life situations recognizable for RIVM researchers. Most questions are relevant for both

the formulation phase (21 of 23) and the production phase (22 of 23), whereas ten questions

are relevant for the extension phase. To indicate in which phase of the research project each

question can be relevant, the first page provides an overview including clickable links to the

questions.

Discussion

The Contribution Mapping approach in our case studies provided us with useful insights

into important areas for alignment between researchers and policy-makers in the field of

public health and health care. We integrated the alignment areas into one list by a cross-

case analysis. In line with the work of de Goede and of Kok and Schuit, the anticipated

alignment areas “goal,” “quality,” “consultative structure,” “relevance and timing,” and

“presentation” focus on the optimal interaction between researchers and policy-makers [16,

17]. However, the identified additional areas “tasks and authority,” “vertical alignment,” and

“organizational environment” are more connected to the context of the research project.

We found that these context topics strongly influence optimal interaction and alignment

with policy-makers. Researchers were not always aware of the context in relation to their

project and of its influence on their own interaction and alignment with policy-makers.

Therefore, it is important for researchers to comprehend context issues in relation to their

project and to act upon them in aligning with policy-makers.

R1R2R3R4R5R6R7R8R9


120 | Chapter 6

We argue that the R4P tool can be useful for other public health and health systems research

projects, since it focuses on process issues that are relevant to most research projects.

The case studies enabled us to translate the alignment areas into reflection questions and

illustrative examples that can be used in daily practice. Furthermore, we identified several

important topics that relate to improving the contributions of scientific knowledge to policy-

making and that offer ways to facilitate alignment efforts and thus enhance contributions to

evidence-based policy-making. These findings can be of interest for researchers conducting

research in commission of governmental organizations.

All phases of the research process require alignment efforts

For an individual researcher, alignment with policy-makers is the most practical way to

influence the contributions of their research and knowledge [24]. Schut et al. also concluded

that researchers have to address the challenges of the complex dynamics of policy-oriented

research through reflection and context sensitivity [25]. Alignment requires awareness of

emerging issues and continuous efforts at various levels. Both the case studies and the

exploratory pilot project showed that personal interactions between individuals — both

in the organization conducting the research and in the commissioning organization — are

crucial for alignment. In our case studies, personal interactions between researchers and

policy-makers occurred mainly during the production phase of a research process, being

the lengthiest phase. However, we want to point out that informal interactions during

the formulation phase and the extension phase are equally important. In the formulation

phase, interaction can prevent an unarticulated knowledge question, a request for research

being so imprecisely formulated that it does not represent the exact knowledge need and

may have a negative impact on the outcome of the project beforehand. At the end of the

production phase, knowledge is mostly presented in a knowledge product, such as reports,

scientific papers, websites, or databases. In the extension phase, interaction may ensure

that the level of interest in the knowledge products does not decrease quickly.

We acknowledged that project managers have to align on topics relevant for the Extension

Phase already during the Formulation and Production Phase. Although the Three-Phase-

Model proved to be helpful in the analysis of the case studies, we departed from the

separation into three phases for the R4P tool. One list of reflection questions turned out to

be more convenient than a list for each separate phase due to the overlap in topics.



6

Alignment has to be organized

To facilitate better alignment in all phases, knowledge institutes should pay explicit attention

to and recognize the everyday process issues associated with knowledge production. For

knowledge institutes, it can be very useful to analyze their organizational routines and

reserve budget for developing awareness of context issues and interactions with policy-

makers. Individual researchers will be encouraged to pay attention to the process issues

associated with research if they are rewarded for doing so, for example, when sufficient

time and appreciation is devoted not only to the scientific merits of their work [26]. For

commissioning organizations, it is just as important to take the need for organizing alignment

into account.

Vertical alignment is crucial

Our experiences in our research suggest that many researchers prefer to pay attention to

scientific issues rather than spending time to alignment. Thus, the urgency of devoting

attention to alignment must be made clear to researchers, since a single, well-aligned

knowledge product that really offers a contribution will often be preferable to a large

number of knowledge products that are of little practical use. However, this aim cannot

solely be achieved at research level, but requires efforts and collaboration at all hierarchical

levels of both organizations (i.e., the knowledge institute and the Ministry) during the

entire research process. It is crucial that all parties involved are aware of their role in the

research process and assume their responsibilities while taking into account those of the

other organization. Although alignment is essential to enhance contributions to policy-

making, knowledge institutes must simultaneously balance adequate alignment with policy-

makers with sufficient distance from the policy-making domain if they wish to maintain

their independence. Internal vertical alignment between researchers, project managers,

and line management of the knowledge institute is essential to optimize both alignment

and independence and to strike the right balance.

Usability of the R4P tool

During our project, we found that theoretical considerations on alignment and knowledge

contributions have little appeal to health systems researchers. Therefore, the understanding

of important alignment areas and the topics they represent had to be “translated” in order

to be useful to researchers in their everyday work. In the R4P tool, the combination of

reflection questions based on research findings and illustrative examples from the case

studies with situations familiar to the researchers particularly offered added value.

R1R2R3R4R5R6R7R8R9


122 | Chapter 6

Although our reflection tool is based on translation of alignment areas derived from case

studies in our own institute, we argue that the tool could also provide a basis for reflection

in other research projects, both for researchers and policy-makers, since the topics covered

are process-related and the tool can be used in a flexible way. The present tool includes 23

questions that cover all important alignment areas. It is up to users to decide whether they

are relevant to their particular project at a given point in time and whether the answers to

the questions have to result in an alignment effort.

Conclusions

Aligning to the needs of policy-makers offers researchers the opportunity to take influence

into their own hands in an ever-changing context, such as the policy priorities of the

Ministry, the influence of political reality on the relevance of their knowledge products, and

the organization of their own research institute. We found that reaching alignment is not

easy at all and depends on many aspects. In our study, we first identified the most important

aspects for researchers in government-commissioned research. By cross-analysis of the case

studies, we could identify eight key alignment areas: the goal, quality, relevance, timing,

and presentation of research (findings), the tasks and authorities of actors, the consultative

structure and vertical alignment within organizations, and the organizational environment.

The R4P tool is based on these areas. We intended to develop an instrument that supports

researchers in undertaking alignment efforts to enhance the contributions of their work

to policy-making. Researchers recognized the questions — which were illustrated with

examples from case studies— as relevant and to the point. The R4P tool can be deployed

in any health systems research project to reveal the topics that are most important in the

project. Initial experiences in using the R4P tool show that it offers useful alignment guidance

to researchers and facilitates reflection on process issues, which will help researchers to

adopt a more context-sensitive approach in their work. By regular reflection, researchers

will be better able to decide what to do or not.

Since the questions in the tool are intended for inspiring reflection, it is not necessary to

answer completely all questions at the same time. It is up to the users which topic they

want to reflect on depending on the phase and characteristics of a specific research project.

By thinking about the answer to the questions, researchers gain insight into the ongoing

process and become aware of what action is needed and how to anticipate on the policy-

making reality. The tool can be used as a checklist by individuals and as basis for open

discussion in project teams. The R4P tool may also provide a basis for dialogue between



6

researchers and policy-makers, turning it into a shared tool for alignment. For the use of

the R4P tool as shared guidance in the dialogue between researchers and policy-makers,

the next step will be to assess its applicability for policy-makers and any need for adaption.

Finally, the ultimate step will be to investigate the influence of using the R4P tool on actual

knowledge contributions to policy-making.

R1R2R3R4R5R6R7R8R9


124 | Chapter 6

References1. Pielke RA, Jr. The honest broker: Making sense of science in policy and politics. Cambridge:

Cambridge University Press; 2007.2. Bowen S, Erickson, T, Martens P, Crockett, S. More than ‘using research’: the real challenges in

promoting evidence-informed decision-making. Healthc Policy. 2009;4(3):87–102.3. Lomas J. Connecting research and policy. Isuma: Can J Policy Research. 2000;1(1): 140-144.4. Landry R, Amara N, Lamari M. Utilization of social science research knowledge in Canada. Res

Policy. 2001;30(2):333–49.5. Innvaer S, Vis, G, Trommald, M, Oxman, A. Health policy-makers’ perceptions of their use of

evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.6. Strydom WF, Funke N, Nienaber S, Nortje K, Steyn M. Evidence-based policymaking: A review. S

Afr J Sci 2010;106(5/6):249.7. Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and

facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14:2. doi: 10.1186/ 1472-6963-14-2.

8. Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Research Policy and Systems. 2014;12(1):34.

9. Sabatier PA. Theories of the Policy Process. Boulder, CO: Westview Press; 2007.10. Lindblom CE. The science of “muddling through”. Public Adm Rev. 1959; 19 (2): 79-88.11. Smith KE, Katikireddi SV. A glossary of theories for understanding policymaking. J Epidemiol

Community Health. 2013;67(2):198–202.12. Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979;39 (5):426–431.13. Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic

utilization of university research in government agencies. Sci Commun. 2004;26(1):75–10614. Lomas J, Brown AD. Research and advice giving: a functional view of evidence-informed policy

advice in a Canadian Ministry of Health. Milbank Q. 2009;87(4):903–26.15. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations

moreeffectivelytransferresearchknowledgetodecisionmakers?MilbankQ.2003;81(2):221–48.

16. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriersbetween epidemiological research and local health policy development. Health Res Policy Syst. 2010;8:26.


18. RIVM. National Institute for Public Health and the Environment. Available from: http://www.rivm.nl/en. Accessed 21 September 2015.

19. Kok MO, Gyapong JO, Wolffers I, Ofori-Adjei D, Ruitenberg J. Which health research gets used andwhy?Anempiricalanalysisof30cases.HealthResPolicySyst.2016;14(1):36.

20. Hegger I, Janssen S, Keijsers J, Schuit AJ, van Oers, HAM. Analyzing the contributions of a government-commissioned research project: a case study. Health Res Policy Syst. 2014;12(1):8.

21. Hegger I, Kok MO, Janssen SJ, Schuit AJ, van Oers HAM. Contributions of knowledge products to health policy: a case study on the Public Health Status and Forecasts Report 2010. Eur J Pub Health. 2016; 26(6):922-927.

22. Hegger I, Marks LK, Janssen SW, Schuit AJ, van Oers, HAM. Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report. J Health Serv Res Policy. 2016;21(1):29–35.



6

23. Khan S, Van Wynsberghe R. Cultivating the under-mined: cross-case analysis as knowledge mobilization. Forum Qual Soc Res. 2008 9(1). Art. 34.

24. Gold M. Pathways to the use of health services research in policy. Health Serv Res. 2009;44(4):1111–36.

25. Schut M, van Paassen A, Leeuwis C, Klerkx L. Towards dynamic research configurations: a framework for reflection on the contribution of research to policy and innovation processes. Sci Public Policy. 2014;41:207–18.

26. Briggs SV. Integrating policy and science in natural resources: why so difficult? EcologicalManagement & Restoration. 2006;7(1):37–9.

27. Westert GP, van den Berg MJ, Zwakhals SLN, de Jong JD, Verkleij H. Dutch Healthcare Performance Report 2010. Bilthoven: RIVM; 2010.

28. Van der Lucht F, Polder JJ. Van gezond naar beter. Volksgezondheid Toekomst Verkenning 2010. Bilthoven: RIVM; 2010.

R1R2R3R4R5R6R7R8R9


126 | Chapter 6

Additional files

Additional file 1: Topic list.

Additional file 2: Draft R4P tool.

Additional file 3: Final R4P tool

available at DOI 10.1186/s13012-016-0496-1

Appendix Chapter 6

Additional file 3

Final R4P tool



6

Research for PolicyResearch Project

FormulationPhase ProductionPhase Extension Phase

R4P TOOL R4Pquestions R4Pquestions R4Pquestions

Organizationalenvironmentoftheproject

010203040506

010203040506 06

Goal of the project 070809

11

13

07

0910111213 13

Interactionduringtheproject 1415161718

1415161718

141516

18

Outcome of the project 1920212223

1920212223

1920

2223

R1R2R3R4R5R6R7R8R9


130 | Chapter 6

I: Organizational environment of the project

1.Whatorganizationswill be involved in theproject and inwhat capacity?What is the

position of these organizations within the health domain of the project, and what are the

practicalimplicationsofitsposition?Explanation and example 1

2. What are the (current) characteristics of the commissioning organization and its

relationshipwithyourorganization?Howdoyouaddressthesefactorsduringtheprocess?

Explanation and example 2

3. In what (dynamic) context does the commissioner operate during the project, and what

implicationscouldthishavefortheprojectitself?Explanation and example 3

4. In what dynamic context does your institute operate, and what implications could this

haveforyourprojectandthealignmentwiththecommissioner?Explanation and example 4

5. What are the specific characteristics of the researchers on your project team? How

are tasks and responsibilities dividedbetween them, andhow is thework coordinated?


6. Are there any problems which prove impossible to be resolved at project level, whereby

higherorganizationallevelsmightbeofassistance?Explanation and example 6

II Goal of the project

7. How and when should the research product be used according to your organization and

thecommissioningorganization?Whatisitspurpose? Explanation and example 7

8. What is the underlying concern of the research question in relation to the problem that

thecommissionerwishestosolve?Explanation and example 8

9. To what extent is it possible to refine the research question at a later stage of the research

process to bring itmore closely in linewith the commissioner’s need?Explanation and

example 9



6

10. Has the commissioning organization experienced any changes which may affect their

knowledge need since the project proposal was produced? Does the research question

formulatedintheprojectproposalcontinuetoaddressthecurrentknowledgeneed?Isit

necessary(andpossible)toamendtheprojectplan?Explanation and example 10

11.Withinwhattimeframedoesthecommissionerrequiretheproductandwhy?Didyou

agreeontheprojectschedule?Areallphases,milestonesanddeliverablesclearlydefined?


12. Have the scientific models, concepts and definitions to be applied within the project

beenagreedbyallparties?Explanation and example 12

13. Did you explicitly agree on the rights and responsibilities of both the commissioner and

yourinstitutewithintheproject?Explanation and example 13

III Interaction during the project

14.Withwhomshouldyoualignwithinyourownorganizationduringtheprocess?Whose

commitmentmustbesoughtandhow?Explanation and example 14

15. With whom should you align externally during the process and whose commitment

mustbesought?Explanation and example 15

16. What type and frequency of consultation with the commissioner will be most appropriate

totheprocess? Explanation and example 16

17. Did you plan any interim knowledge exchange with the commissioner regarding the

contentoftheresearchproduct?Explanation and example 17

18. How do you achieve interim exchange of knowledge between the various users?


R1R2R3R4R5R6R7R8R9


132 | Chapter 6

IV Outcome of the project

19. What types of user (at various levels within the commissioning organization) can be

identified for the intended product, and how is their diversity to be taken into account in

termsofcontentandprocess?Explanation and example 19

20. How do you monitor the production of (similar) knowledge products by other

organizations, and the influence that such products may have on the commissioner and end

usersduringtheproject?Explanation and example 20

21. Do you have a timely discussion with the commissioner with regard to the form of the

researchproduct? Explanation and example 21

22. Do you envisage to draft a plan for the presentation and dissemination of your research

product, and has this been agreed well in advance with both the commissioner and relevant

internalstaff? Explanation and example 22

23. Have the higher organizational levels within both your institute and the commissioning

organization been made aware of the (expected) research results and the product forms

wellinadvance?Explanation and example 23



6

Explanation Questions

I: Organizational environment of the project

Question 1

Whatorganizationswillbeinvolvedintheprojectandinwhatcapacity?Whatistheposition

of these organizations within the health domain of the project, and what are the practical

implicationsofitsposition?

Explanation question 1

It is important to identify: What are the characteristics of the different organizations

in question? What is their position (in relation to your organization’s network, e.g.

commissioner, project partner, subcontractor). What authority or influence do they possess,

andhowcanthisbeexploitedintheinterestsoftheproject?

Example 1

The issue

An independent governmental organization is in charge of a database containing confidential

health care information and this data is only provided for supervision purposes to the Health

Care Inspectorate. Since your project is commissioned by the Health Care Inspectorate to

support them in their tasks, the Health Care Inspectorate puts a data set from the database

to your disposal. However, the independent governmental organization is not involved and

raises objections to the use for research purposes in a meeting with your manager later in

the process.

The outcome

To resolve the issue, your manager aligns with the organization and the Health Care

Inspectorate. They agree to inform each other in future.

Question 2

What are the (current) characteristics of the commissioning organization and its relationship

withyourorganization?Howdoyouaddressthesefactorsduringtheprocess?

R1R2R3R4R5R6R7R8R9


134 | Chapter 6


It is important to identify: What are the characteristics of the commissioning organization

(e.g. Health Care Inspectorate, Ministry of Health, or other ministry), including its

institutional work culture. Attention should also be devoted to the commissioner’s role

within the research process (e.g. as a member of a steering group or advisory committee)

and the degree of ‘formal’ involvement.

Example 2

The issue

Your commissioner, the Inspectorate, supervises health care practices in your country,

maintains the law and as a result, is often the target of media attention. This clarifies why

inspectors are expected to act both in an authoritative manner and in a very carefull way.

In your own research project, the inspector involved wants to have full control and insists in

being involved in decisions on methods and in publications. According to his workculture,

he persists on his position despite your referring to your institute’s independent position.

The outcome

Since this divergence of views cannot be solved at project level, you ask your manager to

align with the inspector’s manager. They agree on the existing operating procedure ensuring

both sufficient information for the Inspectorate and independence for the researchers.

The inspector involved is informed by his manager on the agreed operating procedure and

takes now another position in the project. In return, you commit to inform him on draft

publications in order to avoid surprises.

Question 3

In what (dynamic) context does the commissioner operate during the project, and what

implicationscouldthishavefortheprojectitself?


To monitor the context in which the commissioner is operating, is essential to understand

any unexpected course of action and to be able to discuss this with a view to understanding

the commissioner’s motives.



6

Example 3

The issue

At the commencement of your research project, the commissioner’s contact person asks

you to be kept fully informed of the decisions and choices made by your research team.

Two months later, it turns out to be impossible to contact her in time, because she does

not answer the phone, responds to emails only after a lengthy delay and, according to her

secretary, has no time to schedule a meeting. Meanwhile, there have been media reports of

an urgent reorganization within the commissioning organization, prompted by the minister’s

desire to show effective leadership following a number of incidents. This clarifies why the

contact person is preoccupied with other matters.

The outcome

You align with the head of department to decide how best to proceed. You agree to delegate

the issue to the account manager. She contacts the commissioning organization and via this

route, your contact is restored.

Question 4

In what dynamic context does your institute operate, and what implications could this have

foryourprojectandthealignmentwiththecommissioner?


External parties generally regard your research institute as a single, integrated organization.

It is however important to devote attention to the activities and developments at the

various levels within your institute and its individual departments in order to ensure

uniformity, avoid overlap and address circumstances which may affect your institute as a

whole. Matters to be considered include strategic plans, alignment between your institute

and commissioning organizations at management level, new tasks and responsibilities for

your organization, and other research projects or reports.

Example 4

The issue

Your institute publishes a report about the adverse effects of cycling in traffic, notably

the inhalation of fine particulate matter from exhaust fumes. A week later, your institute

R1R2R3R4R5R6R7R8R9


136 | Chapter 6

publishes another report in which it advises local authorities to encourage people to cycle

to work as a form of healthy exercise.

The outcome

The institute has unintentionally issued conflicting messages, causing confusion and

attracting adverse media attention. The director of the institute is asked to explain this to

both the Ministry of Health and the Ministry of Environmental protection. He commits to

improve the institute’s internal communication.

Question 5

Whatarethespecificcharacteristicsoftheresearchersonyourprojectteam?Aretasksand

responsibilitiesexplicitlyattributedtotheteammembers?Howistheworkcoordinated?


It is important to consider whether all necessary competences and expertise are represented

within the team. The tasks should be clearly designated, for example to specialists in

communications, media, design, data collection, etc. All team members should be aware of

the purpose and deadlines of the product.

Example 5

The issue

In a major project, the project managers involve all relevant experts with respect to the

complicated public health issues that will be investigated during the project. They carefully

keep guard over the scientific quality of their reports and they present their findings at

many acedemic occassions. Surprisingly, the commissioner expresses some dissatisfaction

about the outcome. It turns out that this is caused by the many delays in the process and

the unwelcome media message that came out of the public presentation of the final report.

The outcome

Apperently, necessary project management competences were not available at an adequate

level. To avoid this type of difficulties in future, it was decided that to add a researcher with

extensive process management competences to the project team and to put her in charge

of managing the process.



6

Question 6

Are there any problems which prove impossible to be resolved at project level, whereby

higherorganizationallevelsmightbeofassistance?


If there are any problems, difficulties or obstacles which cannot be resolved by the project

coordinator and/or the commissioner’s contact person, it is essential to refer the matter to

a higher hierarchical level.

Example 6

The issue

In a project, the researchers are not provided with the data which has been promised by the

commissioner, although the data is crucial to the project. The commissioner’s contact person

says that he is unable to help since the data turn out to be confidential. The researchers

cannot proceed.

The outcome

The project coordinator aligns with the head of department to discuss ways in which this

problem can be resolved. In his turn, the head of department contacts the director, who

discusses the issue with both his counterpart at the commissioning organization and they

agree on the conditions for using the confidential data. After taking some measures, the

researchers receive the data and continue with the project.

II Goal of the project

Question 7

How and when should the research product be used according to your organization and the

commissioningorganization?Whatisitspurpose?


You need full clarity regarding the expectations which the product is to meet. In the case of

a research report for the Ministry of Health, for example, the parties should agree the point

within the policy process at which it is to be used, and the nature of that use.

R1R2R3R4R5R6R7R8R9


138 | Chapter 6

Example 7

The issue

Your two-yearly reports are intended to provide input at different moments in the policy

cycle. It is a major job to collect all data and to integrate them in a report every two years.

However, policy-makers indicate that policy realities constantly change and that they need

more up-to-date data.

The outcome

After a constructive discussion, you agree to start a website to present all data. This website

will be updated four times a year.

Question8

What is the underlying concern of the research question in relation to the problem that the

commissionerwishestosolve?


You need clarity with regard to whether the research question and its interpretation

adequately address the commissioner’s need. Conversely, and depending on the type

of product and the anticipated results, it may also be necessary to examine whether the

commissioner will be able to implement an adequate response based on the research

findings and whether adequate policy instruments are available to do so. You have to be

aware that the commissioner can be confronted with queries or criticism on the research

from the professional field, parliament, the media, etc.

Example 8

The issue

The Ministry of Health submits a research question to investigate prevention of obesity in

children. The question is rather vaguely articulated and for the researchers, it is not clear what

exactly made the commissioner submit the question. Is there need for information on the

trends in the number of obese children, on their geographical distribution, on effectiveness

ofinterventions,oneffectivenessofpubliccampaigns,ontheroleofyouthhealthcare?All

these issues have been addressed already. In a discussion with policy-makers, they indicate

that obesity in children will become a key issue in a new national prevention plan starting



6

soon. The research should explore (policy) options to enhance healthy food uptake and

physical exercise at a local level. The results are awaited within half a year.

The outcome

It is agreed to discuss any findings after three months to align on recommendations. The

results will be presented in an infographic that can be used by local policy-makers.

Question 9

To what extent is it possible to refine the research question at a later stage of the research

processtobringitmorecloselyinlinewiththecommissioner’sneed?


A ‘broad’ research question is not necessarily a bad thing since you will have greater

discretion to define your own focus, which will emphasize the organization’s independent

scientific position. You don’t have to specify all research requirements in advance if you

create opportunities for further refinement and coordination during the research process

(e.g. by means of scheduled consultation meetings).

Example 9a

The issue

The Ministry of Health requests a Public Health Status and Forecasts report for which the

research question is very generally formulated. It is agreed that defining the themes is an

integral part of the project.

The outcome

In the first period of the production phase, the researchers can take certain time to reflect

on the themes and to align with policy-makers and scientists in planned meetings. They

finish the report in time.

Example 9b

The issue

The Inspectorate asks for a model to select inspection objects. At the start of the project,

the researchers need a lot of time to find out what the inspectors exactly expect and need.

R1R2R3R4R5R6R7R8R9


140 | Chapter 6

This extension of the formulation phase was not foreseen in the project plan and causes

delay already at the start of the project. At a later stage, lack of both time and budget hinder

the project.

The outcome

For future projects, the project manager firmly resolves to plan time and budget for further

definition of the project in the production phase to avoid problems at the final stage.

Question 10

Has the commissioning organization experienced any changes which may affect their

knowledge need since the project proposal was produced? Does the research question

formulatedintheprojectproposalcontinuetoaddressthecurrentknowledgeneed?Isit

necessary(andpossible)toamendtheprojectplan?


The commissioner’s requirements with regard to the purpose or form of the end product

may change over time, even after approval of the original project proposal. It is important to

ascertain whether this is the case on a regular basis, and to consider whether any changes

to the process itself are then necessary (and possible).

Example 10

The issue

A commissioner wishes to commission research examining the risks of a new technology

used in hospitals, as a follow-up to an earlier study. You formulate a proposal but before the

project itself starts, a serious incident involving a different type of technology takes place.

This causes significant media attention and the commissioner is subject to severe criticism.

It is now far more important for the commissioner to gain further knowledge about this new

technology than about that covered by your original proposal.

The outcome

In consultation with you, a new focus and a different approach are adopted.



6

Question 11

Withinwhattimeframedoesthecommissionerrequiretheproductandwhy?Didyouagree

ontheprojectschedule?Areallphases,milestonesanddeliverablesclearlydefined?


The exact moment of the product’s finalization and/or presentation can be important in

terms of its relevance and use.

Example 11

The issue

The commissioning Ministry requires an extensive research report to support an important

policy document which is subject to a very strict deadline. Given the deadline, not all

required research can be performed within time, but finalization after the deadline will

make the product far less relevant to the commissioner.

The outcome

The project team agrees on the minimum part of the product to be delivered before the

deadline.

Question 12

Have the scientific models, concepts and definitions to be applied within the project been

agreedbyallparties?


The scientific models, concepts and definitions used during the research can have a significant

bearing on the findings. It is possible to dismiss an unfavourable research result by arguing

that the approach was flawed. To ensure the acceptance and relevance of the product, it

is prudent to create support for the scientific basis among all stakeholders, including the

commissioning organization. It is also important to identify any (political) sensitivities about

the models or definitions.

R1R2R3R4R5R6R7R8R9


142 | Chapter 6

Example 12

The issue

In your project, you have to calculate the cost of a certain type of medical care. You find

that this type of medical care has various definitions in practice, with different interventions

and equipment being grouped under the same general heading. For the purposes of your

research, you opt to apply the definition followed by the majority of health care professionals

and for which reliable data sources are available.

The outcome

When the draft report is completed, the high cost calculation takes the commissioner

unpleasantly by surprise. The commissioner argues that your organization did not apply

the correct definition. It takes a long discussion to convince the policy-makers of the

correct method. To compensate for the commissioner’s objection, you decide to extend the

discussion on the definition used in your report.

Question 13

Did you explicitly agree on the rights and responsibilities of both the commissioner and your

institutewithintheproject?


General rules establish the rights and responsibilities of the parties involved in a project of

your institute. However, it is also important to examine whether any specific agreements are

required with regard to aspects such as the ownership of data, the procedures for providing

feedback on draft versions of the product, and intellectual property rights.

Example13

The issue

A commissioning organization wishes to have three of its staff review a draft report.

However, one of the three is extremely slow to do so. Clear agreements have been made

with regard to the review process and the action to be taken further to the reviewers’

comments (Your institute retains full editorial control and will decide whether to make any

suggested changes). In addition a time limit has been set, i.e. the annotated version must be

returned within two working weeks.



6

The outcome

The research coordinator decides to present the commissioner’s contact person with two

options: either the completion deadline remains unaltered and the third reviewer’s input

is disregarded altogether, or the deadline will be deferred to allow time for this reviewer’s

comments to be taken into consideration. The new agreement is recorded in writing.

III Interaction during the project

Question 14

Withwhom should you align within your own organization during the process?Whose

commitmentmustbesoughtandhow?


Vertical imbedding of the product within your organization is important for its legitimacy

towards the commissioner, the resources and capacity required (determined in part by the

desired quality and form of the product) and own future use or further research.

Example 14

The issue

During the production phase of a major project, the project team informs the director on

the proceedings in regular meetings. In the final stage of a theme report, a director at the

Ministry of Health is dissatisfied with its publication form, a flyer instead of a formal report

and he immediately contacts the director of the research institute.

The outcome

Being well informed, the director of the research institute is able to respond adequately on

the complaint. The directors agree on a compromise, a formalized brochure and additional

budget to complete a research report.

R1R2R3R4R5R6R7R8R9


144 | Chapter 6

Question 15

With whom should you align externally during the process and whose commitment must

besought?


In advance of the research process, you should identify the persons with whom alignment

is required (also with a view to the intended recipient of the product). You should also

determine how contact can best be established and maintained.

Example 15

The issue

Your institute is asked to conduct a major research project on public health. Two other

research institutes are very disappointed that they were not charged with this project.

The outcome

Since you need their expertise and wish to maintain a good relationship, you decide to

invite them for a stakeholder meeting before the project starts. You agree to include several

experts from both institutes in the project team and to organize a management meeting

twice a year. All organizations are satisfied with this outcome.

Question 16

What type and frequency of consultation with the commissioner will be most appropriate

totheprocess?


You should ensure that both capacity and the time schedule allow for adequate interaction

and consultation. Agree the form that such consultation will take (e.g. face-to-face, via social

media, email etc.) with all parties concerned.

Example 16

The issue

The contact persons of the commissioning organization are very busy people and it is difficult

to contact them. At the start of the project, you agree on different communication forms.

Four meetings are planned for the whole year. Furthermore, you start an online community



6

to discuss upcoming issues in the meantime. Any interim questions will be solved by e-mail

or by phone.

The outcome

After several months, it turns out that the online community is hardly used. However, the

planned meetings prove to be very important for alignment with the commissioner. In the

next meeting with the commissioner, you agree to plan an extra meeting and to change the

function of the online community into a repository of background information.

Question 17

Did you plan any interim knowledge exchange with the commissioner regarding the content

oftheresearchproduct?


The intended effect of the research product can be increased or achieved somewhat sooner

by scheduling discussions of the interim results (before completion of the project and its

final report). It is important to discuss possible research outcomes with the commissioner

at the earliest possible opportunity, in order to allow all parties to be adequately prepared

for the implications of a particular finding.

Example 17

The issue

You foresee that your research results will not be politically expedient for the Ministry of

Health. In an early stage, you share the results with your contact person and with your

manager.

The outcome

Although the Ministry is not in the positin to influence your research results, timely

discussion with policy-makers give them the opportunity to prepare an adequate response

before your report is published. It must be remembered that such discussion relates solely

to alignment and anticipation. It does not entail the commissioner’s ‘approval’ of the results.

Your institute maintains its scientific independence and integrity by ensuring appropriate

internal alignment between all hierarchical levels of the organization.

R1R2R3R4R5R6R7R8R9


146 | Chapter 6

Question18

Howdoyouachieveinterimexchangeofknowledgebetweenthevarioususers?


The impact of the research product will be enhanced when there is interim discussion and

an exchange of knowledge between the various end users during the course of the project.

It is therefore useful for the project coordinator to know who alignst with those end users

on behalf of the commissioner, and what activities he or she undertakes in order to do so.

If the commisioner’s alignment efforts are not enough to ensure knowledge exchange with

the end users, it may be necessary for the project team to take action, involving the account

manager and his/her staff.

Example 18

The issue

A project has a steering group on which various directorates of the Ministry of Health are

represented. The project group collates and processes the steering group’s comments on the

draft versions of the final research report. Some directorates offer considerable feedback,

others little or none at all.

The outcome

After publication of your report, the project evaluation reveals that the directorates which

have provided least feedback also show least awareness of the report’s findings, yet are

the most critical with regard to its practical value. A steering group member who shows

considerable involvement and engagement in the process tends to engender greater

satisfaction. The project team decides that any follow-up project will involve closer alignment

with the commissioner’s contact person in order to determine the exact membership of the

steering group and the degree of involvement its members are expected to show.

IV Outcome of the project

Question 19

What types of user (at various levels within the commissioning organization) can be

identified for the intended product, and how is their diversity to be taken into account in

termsofcontentandprocess?



6


There may well be several types of user within the commissioning organization (e.g. different

policy directorates, departments or hierarchical levels). Their diverse interests may demand

differentiation in terms of the presentation of the product, both in form and timing.

Example 19

The issue

For your project, you regard the policy-makers of the Ministry of Health as the intended users

of the knowledge products resulting from your project. All directorates are represented in

the advisory board of the project and you expect that your knowledge products will be

widely supported within the Ministry.

The outcome

In practice, it turns out that behind the scenes, your first report is not supported unanimously

within the Ministry and some policy-makers question whether your research should be

continued. You find that this mismatch can be attributed to the characteristics of the policy

directorates: one directorate acts more on general or system level whereas another is more

specialized in a specific health domain. This causes some directorates to feel less committed

than others, although during the advisory board meetings, the representatives do not

clearly express their discontent. You decide to spend frequently a workday at the Ministry

to enable informal meetings with representatives of the different directorates and to gather

information on their specific needs.

Question 20

How do you monitor the production of (similar) knowledge products by other organizations,

and the influence that such products may have on the commissioner and end users during

theproject?


Throughout the production phase, it will be useful to ascertain whether other research

organizations are working on similar or related products. It is also important to determine

the likely value of such products to the commissioner, and whether they will affect the

(perceived) relevance of your product. This will enable you to take these influences into

account in your project and in the alignment with the commissioner.

R1R2R3R4R5R6R7R8R9


148 | Chapter 6

Example 20

The issue

Another research organization intends to present a report at a large event just days before

you are due to publish a product relating to similar subject matter. This will inevitably distract

attention from your product. Moreover, inside information suggests that the conclusions of

the other report directly contradict your own findings. This could place the commissioner in

a particularly difficult situation.

The outcome

You decide to contact both the commissioner and the higher hierarchical levels within your

institute to align on the potential problems and the action required.

Question 21

Do you have a timely discussion with the commissioner with regard to the form of the

researchproduct?


Much research culminates in the publication of a report. However, it is possible that a different

form of product will be more useful to the commissioner. Examples of alternative products

include a summary of a scientific publication to support a policy decision, a presentation, or

an instrument such as a simulation model or a survey questionnaire. Knowledge presented

during a round-table discussion is also a ‘product’.

Example 21

The issue

A commissioner faces a particularly complex policy issue and requests your institute to

provide input for its internal deliberations. Out of habit, your commissioner asks for an official

report on the subject. However, you realize that a report will not meet the commissioner’s

knowledge need for their own exploration of the problem and cannot be delivered on time.

The outcome

Following consultation with the commissioner, it is agreed to prepare and host a

brainstorming session for the policy-makers concerned. Based on literature study, your



6

team will also produce a background document and a presentation. On the outcomes of the

brainstorming session, the commissioner is able to base further action and decisions, and

to formulate terms of reference for a research project on the same topic to be conducted

the following year.

Question 22

Do you envisage to draft a plan for the presentation and dissemination of your research

product, and has this been agreed well in advance with both the commissioner and relevant

internalstaff?


Working to a set plan for the presentation and dissemination of the research product will

increase its outreach and hence the likelihood that the findings are acted upon in practice.

It is useful to plan the relevant activities during the course of the project itself, since there

may be neither time nor capacity once the product has been completed

Example 22

The issue

Despite some delays in a project, the deadline can still be met if staff works overtime. The

research product is completed just before the Christmas holiday, much to the relief of the

project team. When work resumes in January, the team is disbanded and its members

assigned to new projects. They have no time left for extended presentation or dissemination

of the former project’s results since these activities had not been planned and no budget is

available.

The outcome

During the new commissioning cycle by the Ministry, the researchers realize that their

previous research efforts did little to raise awareness of their findings. The policy-makers

show scant interest in the research findings and the Ministry commissions only limited

further research. The researchers discuss this disappointing result in their team and agree

to plan more time and capacity for dissemination in a new project they are working on.

R1R2R3R4R5R6R7R8R9


150 | Chapter 6

Question 23

Have the higher organizational levels within both your institute and the commissioning

organization been made aware of the (expected) research results and the product forms

wellinadvance?


The higher organizational levels can enhance the effectiveness and assimilation of a

product within the commissioning organization by drawing their counterpart’s attention to

the forthcoming product at the earliest possible opportunity. Moreover, the discussion of

relevant RIVM results at the senior management level supports the profiling of RIVM. It is

important that the higher organizational levels on both sides are aware of the forthcoming

results and conclusions so that no one is taken by surprise. The proactive transfer of

information about RIVM research results to the higher management levels is therefore

extremely worthwhile.

Example 23

The issue

Your institute is preparing a research report which has been commissioned by a specific

policy directorate within the Ministry of Health Your director reports on progress and the

expected results at a meeting of the Ministry’s Executive Council, whereupon a director at

the Ministry realizes that the report may well be relevant to his directorate and the Minister,

which he then informs accordingly.

The outcome

The publication of the final report prompts an unexpected discussion between various

field parties, a debate which inevitably attracts media attention. However, because your

director already informed the Minister of Health about the publication and the contents

of the report, the latter is able to field press questions when arriving at her next public

engagement.

Chapter 7

Discussion and Conclusion

R1R2R3R4R5R6R7R8R9


152 | Chapter 7


Discussion and conclusion | 153

7

Introduction

This chapter starts with a short description of the study context, design and findings.

Following this short overview, we present a reflection on the findings in relation to the

research questions. Next, we reflect on the methodology used in this study. Finally, we

provide some perspectives for further research and we end with the conclusions of this

study.

Short description of the study context, design and findings

Context

National Public Health Institutes (NPHIs) have the task to support governments in

safeguarding public and environmental health by monitoring the health status of the

inhabitants, by conducting research and providing advice based on scientific knowledge. It

is part of the NPHIs’ job to contribute with scientific knowledge to policy and practice, which

appears to be difficult.

The National Institute for Public Health and the Environment (RIVM) in the Netherlands

describes its mission as follows on its website [1]:

‘How can we keep ourselves and our environment healthy? That is the challenge facing

government authorities at all levels, from the local to the international. The Dutch National

Institute for Public Health and the Environment (RIVM) carries out independent research and

provides policy advice to assist them in this task.’ Furthermore, the website also mentions

that RIVM places its knowledge ‘at the disposal of policy-makers, researchers, regulatory

authorities and the general public’.

Behind this concise description of RIVM on the website, a complex reality hides in which

researchers often experience that just producing and offering knowledge appears to be

insufficient for contributing to policy and practice. The complexity of a research institute’s

task to provide policy-makers effectively with scientific knowledge is well-recognized [2-4].

Research findings often find their way with difficulty to practice, which is certainly true for

policy-making. How to enhance knowledge utilization is a major issue in public health and

covers an extended research field nowadays [5-7]. This study aims to gain more insights in

ways for enhancing the contributions of government commissioned research in the NPHI

context.

R1R2R3R4R5R6R7R8R9


154 | Chapter 7

For our study on improving knowledge utilization, we defined two main research questions

(Chapter 1):

1. What alignment areas are important for enhancing the contributions to health

policy of government commissioned research conducted by National Public

Health Institutes?

2. Based on the empirical findings gained in this study, what is a practical approach

to enhance knowledge contributions from NPHIs to evidence-informed health

policy-making at national level?

Theoretical framework

Since we intended to find empirical insights at the level of the research project in our study,

we used the method Contribution Mapping developed by Kok and Schuit proceeding from

a constructivist perspective [8] (Chapter 1 and 2). According to Contribution Mapping,

knowledge has only impact if people put a meaning to it, which they will be more inclined

to do if knowledge is aligned with their own views, beliefs, ideas and concerns. Contribution

Mapping considers alignment between researchers and knowledge users important and

provides a method to investigate alignment efforts in relation to knowledge contributions,

which may show into options for improving knowledge utilization.

Study design

We gathered data by a qualitative case study approach [9]. We conducted three case

studies and analyzed these in line with the method Contribution Mapping. After this stage,

we conducted a cross-case analysis to identify areas where specific attention for alignment

is needed in a research project. Based on these alignment efforts and the findings from

the case studies, we developed a reflection tool for researchers at an NPHI [10]. Figure 7.1

provides a schematic representation of the study.



7

Con

tribu

tion

Map

ping

C

ase

stud

y 2

Ste

p 1

Ref

lect

ion

ques

tions

Form

ulat

ion

Phas

e

Ste

p 2

Test

ing

Feed

back

from

proj

ect m

anag

ers

Alig

nmen

tef

forts

Alig

nmen

tef

forts

Alig

nmen

tef

forts

Eig

ht

alig

nmen

t ar

eas

Con

tribu

tion

Map

ping

C

ase

stud

y 1

Con

tribu

tion

Map

ping

C

ase

stud

y 3

Fina

l R4P

tool

w

ith e

xam

ples

and

cl

icka

ble

links

Step

3R

efle

ctio

nqu

estio

nsP

rodu

ctio

n P

hase

Ste

p 4

Test

ing

Feed

back

from

proj

ect m

anag

ers

Cro

ss-c

ase

anal

ysis

Dev

elop

men

t of R

4P to

ol

Ste

p 5

Fina

l ver

sion

of

the

R4P

tool

Thre

e ca

se S

tudi

es

Figu

re 7

.1 S

chem

atic

repr

esen

tatio

n of

the

stud

y Fr

om: H

egge

r et

al;I

mpl

emen

tatio

n Sc

ienc

e. 2

016;

11(1

33)[

10]

R1R2R3R4R5R6R7R8R9


156 | Chapter 7

Findings

Three case studies

In the first stage of our research (2011–2013), we conducted three empirical, qualitative

case studies (Chapter 3, 4 and 5) [11-13]. We gathered data by document analysis and by

semi-structured interviews with researchers, policy-makers and managers involved in the

projects. By coding and analysis, we identified alignment areas that were important for the

contributions of the project.

Three multi-annual RIVM projects from the Public Health and Health Care domain with

different characteristics and all completed in 2010 were selected as cases: “Development

of a Risk Model” (chapter 3); “Development of the Dutch Health Care Performance Report

2010” (chapter 4); and “Development of the Dutch Public Health Status and Forecasts

Report 2010” (chapter 5). These three cases had different characteristics, with the most

prominent differences their size and the increasing formal status of the knowledge products

in policy-making.

In the first case study “Development of a Risk Model”, the development of a risk-based

approach for clinical trial inspections in the Netherlands is described (Chapter 3) [11]. The

knowledge product was a risk model acting as an initial impetus for risk-based selection of

inspection objects. This knowledge product had to be further developed to become part of

the formal inspection policy at a later stage. It therefore retained a confidential, unpublished

status. We found that researchers and commissioning inspectors had divergent views on

their collaboration and the ownership of the knowledge product, but were not aware of

these different perceptions. Areas relevant for alignment in the project concerned not only

the project level, but also the organizational environment of the project and the formal and

historical relationship between the organizations. Although the project was distant from

the strategical level of both organizations and although both researchers and inspectors

involved were not (fully) aware of the historical relationship between the organizations, the

organizational environment and history proved important also at project level.

The second case study described the development of the Dutch Health Care Performance

Report 2010 (DHCPR) (Chapter 4) [12]. The DHCPR monitors the health care performance in

the Netherlands by using indicators for quality, accessibility, and affordability in commission

of the Ministry of Health, Welfare and Sport and aims to contribute to strategic policy-



7

making. In this study, influence of the environmental complexity on alignment emerged.

Important alignment areas were familiarity with the commissioning organization and the

need for alignment with different ministerial directorates, but also with other knowledge

institutes. Furthermore, launching new product types, such as an infographic, also requires

specific alignment, such as on the purpose of the new product and its consistency with other

knowledge products. In this project, we observed that after retirement of a highly valued

double-role-actor, both policy-makers and researchers considered the lack of a successor as

a noticeable loss in the alignment repertoire for the DHCPR.

The third case study concerned the Dutch Public Health Status and Forecasts Report 2010

(PHSF 2010) (Chapter 5) [13]. The PHSF report is embedded in the national health policy

cycle by law and provides the policy themes for the National Health Memorandum. Its

contribution to health–policy making is thus formalized and direct. The PHSF2010 process

included established alignment efforts between researchers and policy-makers based

on about 20 years of experience with alignment from previous PHSF editions. However,

researchers still felt that more contributions could come out of this major project. We

identified areas for specific alignment that could help in enhancing contributions of the

PHSF: creating more awareness of the divergent, continuously changing actor scenarios

of stakeholders and more specifically of policy-makers at different places and levels in the

Ministry; working on vertical alignment within the organizations involved; and careful timing

of draft products to create early adopters.

Cross-case study

To find the most important alignment efforts, we conducted a cross-case study and analyzed

the findings of the three cases in conjunction (Chapter 6) [10]. We identified eight areas

for which alignment efforts are particularly relevant, which are summarized in Table 7.1.

The issues reflect which problems can be avoided by alignment, the topics indicate where

alignment efforts could focus on, for example by discussing them in joint meetings of

researchers and commissioners (Table 7.1).

R1R2R3R4R5R6R7R8R9


158 | Chapter 7

Table 7.1 Eight areas for alignment

Area for alignment Issue Topics to address

Goal Need for full clarity regarding theexpectationsaboutandthe purpose of the knowledge product

Theformulationoftheknowledgequestion;explorationofitsorigin,the‘questionbehindthequestion’andtheunderlyingneedforthe knowledge products

Tasks and authority Needtodefinetasksandresponsibilitieswithintheresearch team and to have clear agreement on the rights andresponsibilitiesofinvolvedorganizations

The input of all involved actors (both researchers and policy-makers); their responsibilities,knowledgeanddataexchange by actors during the process and thefinalauthorityovertheknowledgeproducts

Quality Needtosharethescientificmodels, concepts and definitionsusedduringtheresearch to avoid dismissal of an unfavorable research result

The research method; conceptual framework, and data used in the research project

Consultativestructure

Need to ensure both capacity andtimetoallowforadequateinteractionduringtheresearchprocess

Theconsultativestructureoftheproject;thesharingofrelevantinformationandtherelationshipsbetweenactors;double-roleactors

Verticalalignment Needforverticalembeddingof project and knowledge productwithinorganizationstoassuresufficientlegitimacyand resources

Interactionwithintheorganizationconductingresearchandwithinthecommissioningorganization;interactionbetween hierarchical levels and the embedding of the project in the organizations

Organizationalenvironment

Need to be familiar with thecharacteristicsofinvolvedorganizations,suchasinstitutionalworkculture, dynamics of the commissioner’s context and developments in the own organization.

The environment of the research project; awarenessofrelevantconditionsexternaltotheresearchprojectinfluencingtherelationshipbetweeninvestigatorsandlinked actors; incidents, media events, relationshipswithotherorganizations,changingprioritiesandchangingactors

Relevance and timing

Needtobecontextsensitiveand to be aware of any changes in the commissioner’s needs and requirements with regard to the purpose, form, andtimingoftheproduct.

Theformulationandwordingoftheresearchresultsandtimingofthedeliveryandpresentationoftheknowledgeproducts

Presentation Needtoagreetimelyontheformandpresentationoftheknowledge products and their dissemination.

The design and structure of knowledge products and the tools for the extension strategy

From: Hegger et al;Implementation Science. 2016;11(133) [10]



7

Development of the reflection tool Research for Policy (R4P)

In the next phase of our study, we formulated a practical approach to translate our findings

for researchers working in government commissioned projects (Chapter 6).

Based on the eight alignment areas identified, we developed a reflection tool for researchers.

In this tool, we translated the important alignment areas into open concise questions to

support researchers in their reflection on the research process. In a pilot study, we asked

three project managers to test the two drafts of the R4P tool in their ongoing projects. They

indicated that the topics raised were in fact the key topics they experienced in their research

projects. They suggested making the tool more user-friendly, for example by adding clickable

links, and considered the examples from the case studies to illustrate the questions very

helpful. Taking account for their feedback, we finalized the Research for Policy tool.

The final R4P tool has a clickable format and all questions are illustrated with an example

based on the case studies (Annex 1). When using the tool for reflection, researchers can

decide themselves which questions they consider relevant at a certain moment. They can

also use it as a checklist to prevent overlooking important issues.

Reflection on the findings

In scientific literature, it is well recognized that knowledge push without adequate

alignment is usually insufficient to realize contributions [14-17]. For example, interaction

between researchers and policy-makers is widely considered an important precondition for

knowledge utilization in policy-making [15]. As we described in the introduction, this can be

understood by considering the different logic of policy-making in comparison to research

and science. Alignment efforts and moving towards each other are necessary to find the the

right balance between science and policy-making. In the case studies, we observed that this

common understanding was put into practice by formal and more informal alignment efforts

for coming to agreement with the commissioners. In the interviews, everyone stressed the

importance of meetings, knowing each other and understanding each other’s interests and

needs.

Nevertheless, another common feeling was that there is room for improvement, raising the

question ‘which buttons should be pushed to achieve this’. We initiated the current study

to find practical ways for improving knowledge contributions to health policy in addition to

already implemented approaches.

R1R2R3R4R5R6R7R8R9


160 | Chapter 7

The findings of the three case studies and the cross-case analysis provided us an answer to

the first research question:

What alignment areas are important for enhancing the contributions to health policy of

government commissioned research conducted by National Public Health Institutes?

From the case studies, we identified eight alignment areas that need specific attention to

prevent unaligned situations during the research project (chapter 6). The areas showed to

be consistent with many critical factors for knowledge utilization mentioned in scientific

literature [4, 14, 15, 18-23]. In addition, we consider the difficulty of reaching alignment in

practice also a crucial finding. As the saying goes, the devil is in the detail. We observed that

effective alignment is about ongoing attention to details, which makes reaching alignment

more complicated than often envisaged. Apart from procedures – such as organizing a

meeting-, effective alignment asks for awareness of the critical factors, for knowledge about

the other stakeholders’ concerns and for political sensitivity. For example, it is common

practice to consult the commissioner at the start of a project. However, if researchers are

insufficiently aware of the fact that behind the initial research question often other, actual

questions hide or that the commissioning organization may quite differently perceive the

concern or problem behind the research topic, they could unjustly assume that alignment

on the research question exists after a meeting. Wordings or professional jargon can have

a different meaning for researchers and policy-makers, causing different interpretations of

agreed objectives and terms.

Furthermore, the difficulty of reaching alignment is also caused by some burdensome

features of alignment efforts: they often ask much more time than anticipated and require

full attention. In the case studies and the pilot study for the R4P tool, some researchers

indicated that although they acknowledge the importance of alignment and undertaking

alignment efforts, they consider the scientific issues the main priority of the project and

the process issues as an additional task needing precious research time. We noticed that

researchers look at the process as the vehicle for bringing knowledge to the commissioner;

it should smoothly function and preferably cause no difficulties, extra costs or additional

time.

The case studies demonstrated the need for continuous awareness for non-aligned

situations teaching us that lack of alignment is often unexpected or even unnoticed. The

entire process needs alignment efforts, but what type of alignment effort is needed at what



7

point in time can hardly be predefined. Researchers need more than little enthusiasm for

process issues in order to sustain in conducting alignment efforts apart from the agreed

consultation structure during the entire project.

These findings bring us to the second research question:

Based on the empirical findings gained in this study, what is a practical approach to enhance

knowledge contributions from NPHIs to evidence-informed health policy-making at national

level?

From the case studies, we learned that developing a set of generally applicable alignment

actions would not be the best solution for a practical approach to reach effective alignment.

Instead of operating rules, effective alignment needs continuous reflection on the process.

Researchers working on projects for contributing to health policy have to be aware of

the context of their research, while remaining independent at the same time [18]. Schut

et al affirm the importance of reflection among researchers and the promotion of more

embedded, context-sensitive and flexible research strategies [24].

These considerations made us decide to develop a reflection tool for researchers. Such a

tool would make our findings available in a practical way and can be applied in research

projects, regardless their characteristics such as subject, commissioner, phase, context,

seize, resources and political sensitivity (Annex 1). It is up to the researchers (and other

users) to determine the type and intensity of the alignment efforts needed.

However, this reflection tool can only be effective in an organizational framework that

meets some preconditions. As mentioned before, attention for process issues is difficult to

sustain and researchers need support by facilitating reflection and alignment efforts. The

most important preconditions are (1) a facilitating organization, (2) alignment as a two-

way process and (3) balance between scientific independency and alignment with the

commissioner’s needs. As we will explain, these preconditions also interrelate making the

challenge of enhancing knowledge contributions even more complex.

Facilitating organization

In their interaction, policy-makers and researchers have to take into account and have to deal

with the complex context, in which they operate. Strydom et al pointed out the influence

R1R2R3R4R5R6R7R8R9


162 | Chapter 7

of organizational factors on the uptake of evidence by policy-makers in the decision-

making process [18]. Our case studies have shown that the professional environment of

the NPHI also influences the level at which researchers (are able to) undertake alignment

efforts to enhance knowledge uptake by their commissioning counterpart [18]. Because

of their position at the nexus of science and policy-making, knowledge institutes such as

NPHIs are accountable to both the government and the scientific community and the first

organizational challenge manifests itself. This also implies that they have to meet both the

need for producing aligned commissioned knowledge products to serve the commissioning

organization and the need for publishing in scientific journals as rewarded by the scientific

community. As we mentioned before, we observed that the researchers’ focus was often

primarily on the scientific issues and the content of the knowledge products, whereas

the process issues and alignment efforts, although considered quite important, were

secondary. For researchers, a publication in a peer-reviewed scientific journal is a valuable

reward contributing to their scientific status. A reward for the backstage process tasks is

often less pronounced since many aspects are rather invisible and not quantifiable. NPHIs

should therefore consider whether they reward backstage alignment in sufficient balance

compared to rewarding scientific output and the quantifiable performance tasks, such as

timely delivery or number of reports [25]. Rewarding the often-invisible work on process

issues and alignment efforts as such, regardless any actual contributions to health policy,

emphasizes its importance and will motivate researchers to reflect on alignment.

Knowledge utilization will benefit from sufficient flexibility in interactions with policy-makers.

To be creative in alignment efforts and to interact with the final users of knowledge products,

researchers need a certain level of room for manoeuvre, such as time for informal meetings

and a mandate to make flexible arrangements. On the other side, organizations have to

assure the quality of their knowledge products while facing the risk of external criticism and

negative publicity. Most NPHIs will have an accredited quality system with formal operating

procedures in place for managing the risks related to quality and communication [26]. The

NPHI’s need to be flexible and creative may conflict with the need for formal procedures

to assure the quality of knowledge products from time to time. For stimulating alignment

efforts at every organizational level, we recommend NPHIs to assess regularly their internal

organizational control system, such as operating procedures and accountability rules,

specifically regarding their impact on the margin of manoeuvre for undertaking alignment

efforts by the different organizational levels.



7

Related to this topic of the organizational control system, the case studies revealed that

interaction of researchers with managers and higher hierarchical levels both within the

research organization and within the commissioning organization was often limited or even

absent. Being governmental organizations, most NPHIs will be organized in a hierarchical

way with different organizational layers having a decreasing order of control from top to

bottom [27]. For each organizational layer, the mandate for taking decisions on issues will

be different as well as the importance of these issues and the considerations they attach

to them. For example, the case studies demonstrated that the researchers’ mandate for

taking decisions on data exchange with other research institutions or the strategy for

data collection was limited, which resulted in unfulfilled needs of the commissioner. The

researchers could not solve these problems on their own level, but also experienced that

it was difficult to lift the problem to a higher organizational layer [11, 12]. This lack of what

we call vertical alignment clearly hinders contributions. It was often not obvious or easy

for researchers to gain sufficient insight into the project’s context and the policy-maker’s

reality due to the information gap between the different organizational layers. Vice versa,

managers of higher organizational levels were often unaware of alignment issues at project

level. In each project, issues may arise that can better be solved in a joint endeavor across

the organization, which asks for internal, vertical alignment.

We also would like to emphasize that vertical interaction is not only relevant in case of already

existing problems. For example, we found that the way representatives of the commissioner

participate in the advisory commission of the project was indicative for their organization’s

support and use of the knowledge products afterwards (Chapter 4). Committed members

proved to be favorable for achieving contributions afterwards and we argue that vertical

alignment could also be applied for influencing the appointment of advisory-commission-

members on forehand. The need for vertical alignment during the entire process will be

obvious for major knowledge products with a presentation at the highest national level,

for example in the cases of the DHCPR and PHSF report in our study (Chapter 4 and 5). In

such cases, lack of vertical alignment may undermine alignment reached at project level,

as was the case for the PHSF report when presented to the Minister of Health. The key

message communicated by the highest management was not consistent with the message

as formulated by the researchers and aligned with the policy-makers at project level [13].

R1R2R3R4R5R6R7R8R9


164 | Chapter 7

Vertical alignment may hardly seem necessary in a project with a very specific (technical)

scope and a limited number of stakeholders as in the case study ‘Development of a Risk

Model’ (Chapter 3). However, as part of expectation management, it can be wise to verify

whether the other stakeholders also involve their management in the project outcome. Our

case studies showed that lack of vertical alignment unexpectedly led to misunderstandings

and problems in the final stage of the project.

In addition to vertical alignment on project specific issues, we argue that organizations

should also reflect on the topics alignment and alignment efforts in a systematic way on a

management level. Successful alignment efforts and valuable practices should be identified,

assured and implemented in other organization parts. For example, the double-role-actor

in the DHCPR case can be regarded as a valuable alignment practice that was not assured

by the organization. Both policy-making organizations and NPHIs should consider how to

implement a certain level of alignment assurance, for example by designating for example

by designating special alignment experts who develop alignment strategies, support their

colleagues in implementing them and who keep the organization active on alignment issues.

The two-way process

To reach alignment, efforts only from the researchers’ side will not be sufficient. We observed

that the commissioner’s efforts put into the process determine the success of a research

project to a considerable extent. Without time for consultation or possibilities to share

information, alignment with the policy-makers’ needs is very difficult to reach. In all case

studies, contact persons of the commissioning organization were very busy policy-makers

for which the commissioning a research project was an additional task to their regular

policy-making tasks. Embedding a knowledge product in a formal procedure, as is the case

for the Dutch PHSF in the public health policy cycle, increases the policy-makers’ interest

and commitment, thus stimulating their involvement in the process and their preparedness

to spend time on consultations. Furthermore, researchers can have peace of mind to some

extent about the contributions of their knowledge product to health policy since this is

largely determined by its embedding in the policy cycle. However, such a formal status is

only achievable in limited cases and for other cases, not even desirable. For example, as we

observed in the case Risk Model (chapter 3), smaller projects with experimental approaches

generating intermediate knowledge products do not serve as NPHI flagship and will have

modest contributions to policy-making. Such research projects do not easily fit into a



7

predefined framework and benefit from a high degree of flexibility in the project plan. On

this level, alignment efforts nevertheless remain crucial for ascertaining commissioner’s

satisfaction since a comparative disadvantage of a less formal (and often less important)

knowledge product can be the limited commissioner’s interest and preparedness to put

efforts into the process. To overcome such situations, it is imperative that both the NPHI

and the commissioning organization commit to invest in alignment efforts. Even in case

of small projects with lesser strategic importance for the NPHI, flexible vertical alignment

should be obvious and possible. Although a modest knowledge product may not be the

most important from the NPHI’s point of view, it nevertheless contributes to its image on

the long term. For example, in the case study Risk Model (chapter 3), we found that every

(little) experience contributes to the commissioner’s view on the NPHI and may influence

future commissioning.

Next to alignment efforts, we argue that commitment to respect each other’s role in the

process is also crucial in a two-way process. On one hand, alignment with the commissioner’s

needs is necessary to get optimal contributions to policy-making. On the other hand, more

involvement of a commissioner also implies more influence on the knowledge produced

which leads us to the sensitive issue of scientific independence.

Scientific independence

To be influential in public health, a NPHI needs public trust and authoritative expertise.

Although these preconditions may seem obvious and fundamental, they are also difficult

to combine with sufficient alignment in an era of declining public trust in institutions such

as government, politics and science [28]. Most NPHIs are directly financed by their national

government, which is often also their main commissioner for several public health tasks

including research and policy advice [29]. Nowadays in the (social) media, people frequently

refer to governmental knowledge institutes as being insufficiently independent. The

authority of governmental organizations is not accepted by definition anymore and people

question the NPHI’s advices. As put forward in public discussions, some people believe that

a NPHI paid by the government will -despite of other evidence- support the governmental

position with respect to controversial public health issues such as childhood vaccination,

pollution risks and prevention measures. More than ever before, NPHIs have to balance very

sensitively and act very carefully to maintain public trust and authority [30].

R1R2R3R4R5R6R7R8R9


166 | Chapter 7

At the same time, the need to contribute to health policy is evident. NPHIs need a significant

budget raised from taxpayers, an investment which should lead to societal benefits.

Furthermore, the government considers evidence-informed (health) policy essential and

needs support to succeed in this ambition. From this perspective, the plea for better

alignment to the commissioner’s needs is fully understandable. However, the commissioner’s

involvement and interference should always be subordinate to the overriding NPHI’s need to

safeguard its scientific independence and integrity. Although no governmental organization

will contest this statement in general, the difficulty in practice lies in determining where

policy and opinions ends and science starts. Co-creation of knowledge in which researchers,

policy-makers and/or other stakeholders closely cooperate in a research project is often put

forward as desired approach to optimize alignment and to initiate knowledge use already

during the research process. However, researchers and policy-makers should be aware

that this approach may have the drawback of blurring responsibilities and jeopardizing

scientific independence. We observed that policy-makers oftentimes attempted to interfere

with scientific issues, such as the choice of methodology or definitions. The concepts

of ‘front stage work’ and ‘backstage work’ as put forward by Goffman can be helpful in

understanding and respecting each other’s role in the process [28, 31, 32]. Backstage work is

very important for alignment, as we could observe in the case studies. However, researchers

and policy-makers have to remain sufficiently autonomous and aware of their role to avoid

harm at the front stage. The challenge for researchers is to align with policy-makers and to

remain aligned while persevering in their role and discussing openly any arising tensions.

NPHIs should consider developing actively a professional attitude of their researchers in

this respect. For example, RIVM acknowledged the need to provide continuing education of

its staff in this respect by initiating the so-called ‘RIVM Academy’ in 2013, which focusses

on training researchers in the specific competences they need as experts working in a NPHI

to support the government in safeguarding public and environmental health [33]. In the

context of the Academy, they learn about the policy-making process, the commissioning

organizations and the NPHI position in society while exchanging their own experiences with

all these aspects of their work.

One should bear in mind that the NPHI’s scientific independence and integrity are

highly important: both for the NPHI itself, but also for the commissioning governmental

organizations. As mentioned before, the preconditions for reaching effective alignment and

stimulating reflection interrelate. In an organization facilitating reflection, both researchers



7

and policy-makers will be able to align with each other while preserving their independence.

In the case of doubts, flexible vertical alignment supports them to find the right balance.

A delicate balance

All these considerations make clear that conducting NPHI research and producing knowledge

products in commission of governmental organizations to serve evidence-informed health

policy, is a delicate balancing act between alignment and independence, an art in itself.

One of the key tasks for a NPHI is to serve the public good by producing knowledge that

meets the needs of its commissioners. Next to ensuring a clear research question in relation

to the knowledge needs, application of a valid method and a founded interpretation of the

findings, it is the NPHI’s responsibility to present its work in a serviceable form to policy-

makers and other intended users [34]. This overall package of responsibilities, including

alignment efforts, characterizes the job of NPHI researchers. Producing knowledge to support

evidence-informed health policy in commission of governmental organizations entails an

inherent paradox making the job not that easy [3, 35]. On one hand, the commissioner asks

for answers to knowledge questions, which demands significant research efforts. At the

same time, scientific knowledge is not the only thing that matters for policy-makers; they

might disregard the knowledge at the end, despite scientific quality and all alignment efforts

put into the process [2, 35].

Furthermore, scientific knowledge will only contribute in some way to policy-making if

it fits and is included in the actor scenario of policy-makers. However, it is impossible to

know fully someone else’s actor scenario; moreover, most people are not even aware of

(all aspects of) their own actor scenario. Since actor scenarios also change continuously,

complete adaptation to someone else’s actor scenario is impossible. However, alignment is

about finding an acceptable convergence, not a pursuit of perfection. Sometimes, alignment

is even impossible despite all efforts.

Awareness of these inevitable paradoxes could be a consolation for researchers, who

experience not yielding the expected contributions to health policy. Instead, they might

consider the existence of their organization as a basic contribution to policy-making.

R1R2R3R4R5R6R7R8R9


168 | Chapter 7

In fact, the RIVM ambition to be the trusted advisor on public health and the environment is

a perfect concise formulation to describe the ideal role of NPHI researchers in policy-making

[36]. Establishing and maintaining this valuable position is more complex than one could

envisage at first sight and needs continuous efforts and evaluation of processes.

Reflection on the methodology

Reflection on the use of case studies

In this study, we conducted an evaluation of three cases from RIVM. In fact, the study can be

considered a self-evaluation, which entails both advantages and disadvantages.

Due to our existing knowledge of the organizations involved, the procedures and important

issues, we were able to conduct in-depth examinations of the cases. This proved to be very

valuable for our goal to find a practical approach for enhancing knowledge contributions.

All detailed information and insights in the different layers of a case helped us to find the

heart of the problem in alignment: effective alignment is very difficult to reach and it highly

depends on details, ongoing efforts and organizational environment.

Being aware that our bias could influence the findings, we decided to deploy independent

investigators for most interviews. Interviewees were very open and willing to provide the

interviewers with insight information, which was of great value. The processes investigated

were completed sometime before, which made us being alert on any recollection bias.

Although we cannot exclude this bias, the interviews offered so much details from

different perspectives that we were able to reconstruct a consistent picture for each case.

Furthermore, data saturation could be obtained and general issues clearly came up to the

surface, showing that many alignment areas are generally applicable. Our findings are in line

with barriers and enhancers for knowledge utilization as described in scientific literature [4,

14, 15, 18, 19, 20-23]. We therefore argue that the alignment areas defined will be relevant

for NPHIs and other knowledge institutes in the field of public health and the environment.

Since the need for a two-way process is also an important precondition, policy-makers may

also profit from the findings in their alignment efforts with research organizations.

Reflection on Contribution Mapping

In the case studies, we used Contribution Mapping as theoretical framework for our analysis.

In this paragraph, we reflect on this method.



7

The Three Phase Model

In the analysis of the case studies, we considered the division of a project in three phases

very functional for our purpose to unravel alignment efforts during the project. However,

the original graphic representation of the Three Phase Model of Contribution Mapping

showing the course of a research project looks rather complicated at first sight [8]. (Figure

7.2) We experienced that the complete model is rather hard to explain to an outsider

who is not familiar with the constructivist theory. In the interviews, we used a simplified

version (without the arrows and actors) as we noticed that it was helpful for respondents in

recollecting the course of events in the project.

Figure 7.2 The Three Phase Model (Kok & Schuit, 2012 [8])

Although complicated, the model still reduces significantly the complexity of real life. In

practice, the phases of a project are not as strictly separated as the model suggests. This

specifically emerged from the feedback of the project managers during the development

of the R4P tool. Alignment needs both anticipation on coming phases and repetition of

efforts during the process. This means that most reflection questions of the R4P tool can

be relevant at any point in time. Although a division into phases is useful for retrospective

analysis, it is clearly not for reflection on alignment in practice. We decided to let go the idea

of developing separate R4P-parts in the final R4P tool. In the R4P tool, a table now indicates

the phase(s) for which every reflection question can be relevant (Chapter 6) [10].

R1R2R3R4R5R6R7R8R9


170 | Chapter 7

Analysis by Contribution Mapping

The Three Phase Model only represents the course of one project. In practice, most

researchers work simultaneously on several projects, which differ in phase, timing, size,

commissioner and structure. The same situation exists on part of the policy-makers for

whom commissioning research is often just a (minor) part of their tasks. Working on different

projects and tasks also influences alignment, if only by causing time constraints and being

engaged in inequitable issues. One could argue that the evolving actor scenarios represent

this real-life-complexity in the Three Phase Model. Nevertheless, Contribution Mapping

does not address alignment constraints related to interference of other projects or tasks.

The Contribution Mapping approach of conducting case studies, analyzing them and

providing feedback to stakeholders is laborious and asks for considerable throughput time

and research capacity. In the meantime, the projects evaluated are finished or if continued,

staffed with new researchers and policy-makers. Furthermore, the context is continuously

changing, due to reorganizations, new managers, politicians and budget considerations.

Therefore, we do not consider Contribution Mapping as an appropriate method for providing

an immediate solution to an urgent problem.

Particularly, Contribution Mapping is a valuable method for evaluating the ‘alignment

system’ of a knowledge institute. For this purpose, the method could gain strength if a

specific comparison between the intended outcome of formalized institutional procedures

and the actual outcome of all alignment efforts is also included. For example, in our case

studies, a recurrent problem reported was the gap between the formalized requirement for

researchers to align their project plan with the commissioning organization and the feasibility

to meet this requirement in the allocated period. Analysis by Contribution Mapping made

clear what the consequences of this existing gap were for the research projects. Knowledge

institute can use such general findings to optimize their internal procedures. Based on the

analysis of three cases, we were able to identify general applicable alignment areas that

are important for a NPHI, regardless the project, commissioner or people involved. By

translating these areas into the R4P tool, we made the insights of this study accessible for

interested researchers and retained them for practical use.

An important notion for applying Contribution Mapping is that it is an intervention in itself.

One should be aware that it also has the effect of an alignment effort and may change the



7

constellation around the case investigated. In fact, most alignment areas mentioned in the

R4P tool also apply to a contribution-mapping project.

Reflection on the R4P tool

In this study, we developed a practical reflection tool based on our findings and intended

to support researchers in their alignment efforts. To get this tool further developed and

used by the intended users, appeared to be a case of knowledge utilization within a study

on knowledge utilization. In turn, we experienced that dissemination of our findings and

the R4P tool was likewise difficult to achieve in an effective way needing attention and

alignment efforts.

In our study, we found that alignment within the own organization is also essential. The

need for alignment exists on many levels and can hardly be underestimated. However, we

also observed that sharing of alignment experiences occurred only to a limited extent within

the organizations involved in the case studies. Again, we would like to advocate systematical

reflection and open information exchange with respect to alignment with commissioning

organizations and it would be great if this would include a certain degree of transparency

about difficult situations and mistakes to illustrate the unruly reality of reaching alignment.

Perspectives for further research

Further development of the R4P tool

In this study, we developed the R4P reflection tool for supporting NPHI researchers in

their alignment with commission governmental organizations. The tool is based on our

findings in three case studies in the health domain. In the pilot study, we demonstrated

the tool’s applicability for the health domain, but its effectiveness on actual alignment with

the commissioner needs further research. We propose to develop further the R4P tool

while using it at the same time. For this purpose, prospective action research will be most

suitable, in which researchers systematically use the tool in their project, in accordance with

a study protocol. In different cycles, in which also policy-makers should be involved and

interviewed, the researchers could implement adaptations of the tool and/or procedures

based on the findings.

As researchers suggested during the pilot study, the tool will probably be useful for

commissioning organizations too. In an implementation study, it is worthwhile to investigate

R1R2R3R4R5R6R7R8R9


172 | Chapter 7

whether applying the tool in joint meetings of researchers and commissioners could provide

a method to make implicit assumptions more explicit. If this could be achieved, alignment

and mutual trust will improve.

We hypothesize that the R4P tool will also be applicable in other research domains based

on the recognition of the R4P issues shown in different groups of researchers. Nevertheless,

a point for further development is the implementation of the R4P tool in research domains

that are linked to other commissioning ministries and/or other stakeholders than the health

domain. Such an applicability study will also show which adaptations may be needed for use

by other NPHIs.

European differences in alignment

A major challenge for NPHIs is to manage the balance between the scientific focus and the

process issues in the long term. As the case studies demonstrated and we argued before,

alignment asks for ongoing efforts and attention from researchers. The remaining question

is how to persist effectively in attention for alignment. We suggest creating an enabling

and learning organization that supports its staff in alignment efforts and process issues.

Based on the experience gained in the case studies, we expect that this will require an

integrated, long-term approach. This challenge could benefit from research on optimal

ways to preserve the motivation for process-issues and the systematic transfer of good

practices within a knowledge institute. Such research should focus on the influence of work

culture, but also on other cultural aspects in alignment. For example, in the Netherlands,

the work culture is often relatively informal in the way people interact and communicate.

The advantages of this informality are easy interaction and access to each other, but may

cover up the still existing formal hierarchy, sometimes resulting in unexpected lack of

vertical alignment. In other countries, other characteristics can be influential and it will be

useful to map these differences in future research. In the first place, this will provide insights

for enhancing contributions at a national level in the countries concerned. In the second

place, knowledge about international differences in alignment processes is highly relevant

to enhance contributions to European policy-making. It is already complex to understand

and optimize alignment processes within one country, an international context is even more

complex [37]. To complete the Dutch example, the tendency to avoid hierarchical thinking

within organizations is often reflected in the international representation of knowledge

institutes, for example in European working parties. Where the Netherlands considers



7

representation by a lower hierarchical level appropriate, other member states will be more

inclined to delegate higher hierarchical levels from knowledge institutes [38], which creates

a different starting position in the consultation structure. To be aware of the implications of

these differences can be very useful for effective alignment with other member states for

realizing contributions to European policy-making.

Enhancing contributions in an era of skepticism

For this study, we focused on the level of the research project and specifically on the

researchers’ possibilities of exercising influence on their contributions to policy-making.

Our starting point was the situation where a commissioning organization has a knowledge

question and asks for scientific evidence to support the policy-making process. Even in this

situation, implying a favorable attitude towards scientific evidence, we found that alignment

is more difficult than expected, since it is about the details, ongoing attention and sticking

to one’s role. In fact, researchers have to be well acquainted with the policy-making process

and organization (and vice versa).

We also mentioned the importance of the NPHI’s scientific authority and the fact that

this authority is not as obvious anymore for the public as it used to be some decades ago.

Discussions in social media reflect public skepticism about and even denial of scientific

evidence, for example in the case of childhood vaccination and environmental change. NPHIs

already try to anticipate on this development by investigating effective risk communication

and effective participation in the public debate. However, it is also conceivable that skepticism

and denial become manifest in policies, which will be challenging to both policy-makers and

researchers, rocking the established principles in their profession [39]. Research is needed

to investigate the possible impact of this trend on enhancing knowledge contributions to

health policy and should include the level of the research project as we did in our study. The

insights gained should offer guidance for collaboration and maintaining scientific integrity,

both to researchers and to policy-makers.

Conclusions

Next to the scientific outcome, the knowledge contributions of a NPHI research project to

health policy also depend on the process, specifically the interaction between the NPHI

and the commissioning governmental organization, both at management level and at

project level. Reaching alignment with the commissioner is an important way to influence

R1R2R3R4R5R6R7R8R9


174 | Chapter 7

knowledge utilization making alignment efforts a never-ending, crucial part of a research

project. To enhance contributions to health policy, national public health institutes should

pay specific and ongoing attention to eight alignment areas: goal, tasks and authority,

quality, consultative structure, vertical alignment, organizational environment, relevance

and timing and presentation (Table 7.1). Each area represents issues that may arise in

government commissioned research. Researchers may enhance knowledge contributions

to health policy by focusing specific alignment efforts on these issues.

To identify the need for any specific alignment effort, a general rule or procedure is not

achievable since alignment depends on situation-specific details. Every research project

has its own characteristics and is constantly evolving in time. To identify the need for

specific alignment efforts, we advise researchers to reflect systematically and regularly on

the process. The reflection tool R4P, based on the important alignment areas, supports

researchers in this systematic reflection.

Nevertheless, knowledge institutes have to acknowledge that reaching alignment is much

more difficult than often believed. Furthermore, they need to balance carefully alignment

with the commissioner’s need against the institute’s scientific independence and integrity.

By organizing the institute in a way that facilitates and supports this balancing act, the NPHI

may optimize its contributions to health policy.



7

References1. RIVM. About RIVM. Available from: http://www.rivm.nl/en/About_RIVM. Accessed 1 October

2016.2. Jasanoff, S. The Fifth Branch: Science Advisers as Policymakers. Cambridge, MA: Harvard

University Press; 1990.3. Pielke RA, Jr. The honest broker: Making sense of science in policy and politics. Cambridge:

Cambridge University Press; 2007.4. Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979; 39(5): 426-431.5. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations

moreeffectivelytransferresearchknowledgetodecisionmakers?MilbankQ.2003;81(2):221–48

6. Innvaer S, Vis, G, Trommald, M, Oxman, A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.

7. Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14:2.

8. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Pol Syst. 2012; 10:21.

9. Yin RK. Case Study Research Design and Methods. 4th rev ed. Thousand Oaks: SAGE Publications Inc; 2008

10. Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFEM, van Oers HAM. Research for Policy (R4P): a reflection tool for researchers to enhance contributions to policymaking. Implement Sci. 2016; 11(133).

11. Hegger I, Janssen SWJ, Keijsers JEFM, Schuit AJ, van Oers, HAM. Analyzing the contributions of a government-commissioned research project: a case study. Health Res Policy Syst. 2014;12(1):8

12. Hegger I, Marks LK, Janssen SW, Schuit AJ, van Oers, HAM. Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report. J Health Serv Res Policy. 2016;21(1):29–35.

13. Hegger I, Kok MO, Janssen SJ, Schuit AJ, van Oers HAM. Contributions of knowledge products to health policy: a case study on the Public Health Status and Forecasts Report 2010. Eur J Pub Health. 2016; 26(6):922-927.

14. De Leeuw E, McNess A, Crisp B, Stagnitti K. Theoretical reflections on the nexus between research, policy and practice. Crit Public Health 2008;18:5–20

15. DeGoedeJ,PuttersK,vanderGrintenT,vanOersHA.Knowledgeinprocess?Exploringbarriersbetween epidemiological research and local health policy development. Health Res Policy Syst 2010;8:26

16. Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Syst. 2009; 7(Suppl 1):I1

17. Wehrens R. Beyond two communities -from research utilization and knowledge translation to co-production?PublicHealth.,2014;128(6):545-551.

18. Strydom WF, Funke N, Nienaber S, Nortje K, Steyn M. Evidence-based policymaking: A review. S Afr J Sci 2010;106(5/6):249.

19. Sedlačko M, Staroňová K. An Overview of Discourses on Knowledge in Policy: ThinkingKnowledge, Policy and Conflict Together. Cent Eur J Public Health. 2016; 9(2): 10-31.

20. Keijsers JFEM, Paulussen TGWM, Peters LWH, Fleuren MAH, Lammers F. Better use of knowledge: Information behaviour of national policy makers. (In Dutch: Kennis beter benutten: Informatiegedrag van nationale beleidsmakers). Woerden: NIGZ and TNO; 2005.

R1R2R3R4R5R6R7R8R9


176 | Chapter 7

21. Lomas J. Improving Research Dissemination and Uptake in the Health Sector: Beyond the Sound of One Hand Clapping. McMaster University Centre for Health Economics and Policy Analysis. Policy Commentary C97-1. Hamilton, Ontario; 1997.

22. Haynes AS, Derrick GE, Chapman S, Redman S, Hall WD, Gillespie J, Sturk, H. From “our world” to the “real world”: Exploring the views and behaviour of policy-influential Australian public health researchers. Soc Sci Med. 2011; 72(7):1047-1055.

23. Kothari A, Birch S, Charles C. “Interaction” and research utilisation in health policies and programs:doesitwork?HealthPolicy.2005;71(1):117-125.

24. Schut M, van Paassen A, Leeuwis C, Klerkx L. Towards dynamic research configurations: a framework for reflection on the contribution of research to policy and innovation processes. Sci Public Policy. 2014;41:207–18

25. Frey BS, Homberg F, Osterloh M. Organizational Control Systems and Pay-for-Performance in the Public Service. Organization Studies. 2013; 34(7): 949-972.

26. Viso AC, Wanoyoike S, Dusenbury C, Mably S, Sethi M, Desenclos JC. Peer-to-Peer Evaluation Initiative for National Public Health Institutes. In: A Series of Technical and Policy Briefs. 2014; IANPHI: St-Maurice, France; Atlanta, USA.

27. IANPHI. Organizational Charts. Available from: http://www.ianphi.org/resources/toolkit/organizationalcharts.html. Accessed: 26 February 2017.

28. Bijker WE, Bal RA, Hendriks R. The paradox of scientific authority: the role of scientific advice in democracies. Cambridge, MA: MIT Press; 2009.

29. IANPHI. The International Association of Public Health Institutes. Available from: http://www.ianphi.org/. Accessed: 15 January 2017.

30. Assmuth T, Lyytimäki J. Co-constructing inclusive knowledge within converging fields: Environmental governance and health care. Environ. Sci. Policy 2015 51: 338-350.

31. Goffman E. The presentation of self in everyday life. London: Penguin; 1990.32. Bekker M, van Egmond S, Wehrens R, Putters K, Bal R. Linking research and policy in Dutch

healthcare: infrastructure, innovations and impacts. Evid Policy 2010;6(2):237–253.33. RIVM. Annual Report 2013: RIVM Academy encourages learning. Bilthoven: RIVM. Available

from: http://www.rivmjaarverslag2013.nl/en/People_and_organisation/RIVM_Academy_encourages_learning. Accessed: 26 February 2017

34. WHO. The 10 Essential Public Health Operations. Available from: http://www.euro.who.int/en/health-topics/Health-systems/public-health-services/policy/the-10-essential-public-health-operations. Accessed: August 2016.

35. Stone DA. Policy paradox: The art of political decision making. New York Norton & Company Ltd; 2002

36. RIVM. About RIVM; Mission and strategy. Available from: http://www.rivm.nl/en/About_RIVM/Mission_and_strategy. Accessed: 12 January 2017.

37. van de Goor I, Hamalainen, RM, Syed A, Juel Lau C, Sandu P, Spitters H, Eklund Karlsson L, Dulf D, Valente A, Castellani T, Aro AR; REPOPA consortium. Determinants of evidence use in public health policy making: Results from a study across six EU countries. Health Policy. 2017; 121(3):273-281.

38. Nederlandse END-vereniging. Handleiding voor Nederlandse Gedetacheerde Nationale Deskundigen (In English: Manual for seconded national experts). 2013. Available from: http://eu.nlvertegenwoordiging.org/binaries/content/assets/postenweb/b/belgie/permanente-vertegenwoordiging-van-nederland-bij-de-eu-in-brussel-pv-eu/import/europese_instellingen/end/handleiding_end/handleiding---update---end-najaar-2013.pdf. Accessed: 15 January 2017.

39. Lewandowsky S, Mann ME, Brown NJL, Friedman H. Science and the Public: Debate, Denial, and Skepticism. Journal of Social and Political Psychology, North America, 4, Aug 2016. Available at: http://jspp.psychopen.eu/article/view/604. Date accessed: 12 March 2017.

Summary

Samenvatting

Dankwoord

About the author

ListofPublications(2011-2017)

R1R2R3R4R5R6R7R8R9


178 | Summary


Summary | 179

S

Summary

Chapter 1. Introduction

In this thesis, we describe an empirical study on how contributions of knowledge and

knowledge products to health policy-making are achieved. We have investigated how these

contributions can be enhanced by focusing on the processes involved in the relationship

between research and health policy-making and by taking into account the continuously

changing context of the research process.

We specifically addressed the use of knowledge produced by knowledge institutes, such as

National Public Health Institutes (NPHIs), in public policy by governmental organizations.

These knowledge institutes weigh and translate relevant information for the purpose of

policy-making and often present the knowledge in a way that meets the needs of the

policy-makers, in the form of reports, tools, websites, infographics, or other knowledge

products. In practice, NPHI researchers experience that knowledge contributions to policy-

making are not self-evident and despite taking into account the policy-makers’ needs,

contributions of knowledge products to health policy are difficult to achieve. In the pursuit

of evidence-informed health policy-making, decision-making is well informed by the best

available research evidence and scientific knowledge. However, one should bear in mind

that other types of information, such as ideas and interests, are also relevant and influential

in addressing the wicked problems in public health and health care. When NPHI researchers

intend to contribute effectively to policy-making, they will need some insights on knowledge

utilization in policy-making and support in the form of practical tools next to their specific

scientific expertise.

The aim of this study was to obtain insights in how knowledge institutes such as NPHIs can

enhance the contributions to evidence-informed health policy of knowledge produced in

commission of governmental organizations. We addressed the following research questions:

1. What alignment areas are important for enhancing the contributions to health

policy of government commissioned research commissioned by National Public

HealthInstitutes?

2. Based on the empirical findings gained in this study, what is a practical approach to

enhance knowledge contributions from NPHIs to evidence-informed health policy-

makingatnationallevel?

R1R2R3R4R5R6R7R8R9


180 | Summary

For this study, Contribution Mapping provided the theoretical framework. We used a case

study approach and mapped in detail the process of three projects of the Dutch National

Institute for Public Health and the Environment (RIVM), focusing on the alignment efforts in

the projects and the projects’ contributions to health policy-making by the commissioner at

national level. Based on the findings from the case studies, we identified the most decisive

alignment areas for optimal knowledge contributions to health policy-making. Finally, we

prepared and evaluated a draft tool based on the theoretical framework and empirical

findings for supporting researchers in alignment.

Chapter 2. The complex relationship between research and health policy: a concise

overview of theoretical approaches

It often appears difficult to achieve integration of scientific knowledge and research

evidence into health policy; a problem which has been subject of extended scientific

research in social sciences for several decades. Chapter 2 provides a concise overview of the

theoretical background for the complex relationship between research and health policy,

which describes an influential typology of knowledge use and successive theoretical models

on knowledge utilization.

Knowledge can be used in different ways and Weiss provided an influential typology of

knowledge use, which makes a division between instrumental, conceptual and symbolic use

of knowledge. Instrumental use means that technical, objective data are utilized in a direct,

specific way to solve a particular problem. Conceptual use emerges when research delivers

ideas to policy-makers, which changes their perception of a problem. When decision makers

use research as an argument to take an advocacy position, it influences policy in a symbolic

way.

To handle the complexity of knowledge utilization, models have been developed to explain

actual knowledge use and to identify barriers and enhancers to improve the knowledge

utilization. The rational positivist models regard knowledge as a product for use in policy-

making at a certain point in time. These models presume two different worlds of science

and policy-making and hardly reflect the complex and incremental nature of policy-making.

Interactive models of science-policy relations also consider the context, communication

and interaction between researchers and policymakers for bridging the gap between the

two communities. Constructivist models emphasize the co-creation of knowledge in a


Summary | 181

S

social process and consider boundaries between research and policy as the outcome of

negotiation processes between researchers and policy-makers. The idea of separate

worlds is replaced by the concept of ‘hybrids’ where researchers and policy-makers co-

create knowledge in mutual interaction. In this concept, boundary organizations having

characteristics of both science and politics, and involving both scientists and policy-makers

in knowledge production, can play an important role. In a negotiation process including

informal interaction, researchers and policy-makers search for consensus backstage behind

the scenes. This boundary work creates trust and makes knowledge (products) ‘acceptable’

at the formal frontstage, where a clear division of tasks and responsibilities is presented to

the public.

All models offer some part of understanding about what is going on in the relationship

between science and policy and the different theoretical approaches resulted in different

practices to improve knowledge utilization. The positivist approach is reflected in knowledge

translation activities such as research priority setting, knowledge distribution and evaluation

of uptake. From the interaction perspective, knowledge brokers linking researchers and

knowledge users act as valuable intermediates to bridge the gap between the two worlds. In

this study, we used the Contribution Mapping approach by Kok and Schuit, a constructivist

approach for monitoring research projects and for understanding of how to improve

alignment between researchers and knowledge users. Contribution Mapping introduces the

concept of alignment efforts, which are efforts undertaken by researchers and other involved

actors, such as commissioner and knowledge users, to reach alignment in the research

process. Alignment efforts are the starting point for enhancing research contributions to

policy-making and have a central role in this study. Furthermore, the term actor scenario

indicates the involved actors’ scenario of the future. According to Contribution Mapping,

knowledge has to feature in the actor scenario of (intended) knowledge users for achieving

actual contributions.

Chapter 3. Analyzing the contributions of a government-commissioned research project:

a case study

Chapter 3 describes the first case study ‘Development of a Risk Model’, in which we analyzed

the process of an RIVM project. In this project, risk-ranking models for clinical trials were

developed in commission of the Health Care Inspectorate. The aim of our case study was

to identify the alignment efforts that influenced contributions of the Risk-Model-project to

R1R2R3R4R5R6R7R8R9


182 | Summary

the Inspectorate’s work. According to the Contribution Mapping approach, we conducted

an analysis of relevant documents such as minutes of project meetings and we interviewed

actorsfromRIVMandInspectorateinvolvedintheproject(n=10):theRIVMinvestigators,

the key users of the risk models (four inspectors) and managers from both RIVM and the

Inspectorate at two different organization levels. In a feedback session with inspectors,

RIVM investigators and their managers, we discussed our findings for finalization. Based on

the findings, we inductively identified six categories of relevant alignment efforts.

We found that both organizations, RIVM and Inspectorate, had divergent views on the

ownership of the project’s knowledge products and on the mutual relationship of their

organizations, which resulted in different expectations with respect to the project. The

RIVM researchers considered the use of the risk models to be disappointing, whereas

the inspectors had a positive opinion about their contributions. However, all interviewed

actors (investigators, inspectors, and managers) were unaware of these remarkably

different perceptions. We identified six relevant categories of alignment efforts: relevance,

consultative structure, goal and timing, tasks and authority, vertical alignment and

organizational environment. These six categories of alignment efforts include both horizontal

alignment efforts between the investigators and inspectors as well as vertical alignment

efforts between investigators or inspectors and their managers within the own organization.

Furthermore, it became manifest that alignment efforts can be relevant at three levels: at

the first level, to align between investigators and linked actors; at the second level, to achieve

alignment with the organizational environment of the project; at the third level, to align

between the involved organizations as part of their formal and historical relationship (existing

irrespective of the project). Researchers can improve first-level alignment by continuous

reflection on the process, awareness of mutual expectations, and open discussions with the

commissioner. At the second level, vertical alignment is essential to formulate an adequate

response to challenges for the research project from the organizational environment.

Alignment at the third level is more difficult to influence and requires a collective approach

at all organizational levels.

Based on the findings in this first case study, the importance of a reflective attitude towards

the social aspects of research projects at all levels of the organization became clear.


Summary | 183

S

Chapter 4. Enhancing the contributions of research to health care policy-making: a case

study on the Dutch Health Care Performance Report

In a second case study, we analyzed the developmental process of the 2010-edition of the

Dutch Health Care Performance Report (DHCPR). The DHCPR is a major RIVM knowledge

product, in which the health care performance in the Netherlands is monitored by using

indicators for quality, accessibility and affordability of health care. Both RIVM and the

Ministry of Health wishes to increase the contribution of the DHCPR to health care policy-

making and our aim was to identify ways to achieve that. This case study focused specifically

on the process at project level, where RIVM researchers and policy-makers of the Ministry

of Health are the main involved actors. To investigate which areas need specific alignment

efforts by researchers and/or policy-makers to enhance the contributions of the DHCPR, we

interviewedRIVMresearchers(n=8)andpolicy-makersfromtheMinistryofHealth(n=10).

We integrated the interview data with findings from the document analysis of relevant

(policy) documents and reports and assessed the process for developing the 2010 version of

DHCPR according to the Contribution Mapping approach.

We identified six areas where alignment is specifically relevant for enhancing the

contributions of future versions of the DHCPR. These areas are:

1. the DHCPR content for presenting well-balanced information for different

ministerial directorates;

2. backstage work for negotiating the border between science and policy-making and

for reaching consensus on the formulation of findings and recommendations;

3. availability of double role actors working both at the knowledge institute and the

Ministry for enhancing information exchange and mutual understanding;

4. (expected) reports of other knowledge institutes for avoiding overlap, contradictions

and poor timing;

5. data collection and generation for assuring data completeness and continuity of

data supply.;

6. the form in which project results are presented for being aligned to the needs of

the end users and the commissioner

We found that achieving the contribution of healthcare-performance-reporting to policy-

making is complicated and requires continuous alignment efforts between researchers and

policy-makers. These efforts should therefore form an inseparable part of the healthcare-

R1R2R3R4R5R6R7R8R9


184 | Summary

performance-reporting. This demands substantial resources, but it is worth considering

since it may pay back in better contributions to policy-making.

Chapter 5. Contributions of knowledge products to health policy: a case study on the

Public Health Status and Forecasts Report 2010

Chapter 5 describes the case study on the 2010-edition of the Dutch Public Health Status

and Forecasts report (PHSF 2010). The PHSF Report integrates research data on public

health and identifies future trends affecting public health in the Netherlands. The PHSF has

a formalized position in the national public-health policy cycle and is issued by RIVM every

four years. RIVM considers all policy-makers at the Ministry of Health, Welfare and Sport its

key users, but RIVM-researchers experience that far from all policy-makers are familiar with

the PHSF and that the group of users could considerably be extended.

To investigate how PHSF contributions to national health policy can be enhanced, we

analyzed the development process of the PHSF 2010. As in the previous two case studies,

we used Contribution Mapping in this case. For collecting data, this study included the

analysis of documents from the PHSF-2010 process and in-depth semi-structured interviews

with key actors from RIVM and the Ministry of Health who were involved in VTV 2010.

Interviewers independent from RIVM and the Ministry of Health conducted the interviews

withRIVM researchers andmanagers at three levels (n=10) andwith policy-makers and

managersatthreelevelsattheMinistryofHealth(n=10).Finally,weidentifiedimportant

areas for alignment efforts using a constant comparative analysis method.

The PHSF report is considered an example of successful boundary work in literature. Indeed,

the PHSF-2010 process included several established activities aiming at alignment between

researchers and policy-makers, such as informal meetings. Nevertheless, we identified

three issues that are easily overlooked in knowledge development, but provide suggestions

for enhancing contributions to health policy. The first issue is awareness of divergent and

continuously changing actor scenarios; the second issue is attention for effective vertical

alignment within the organizations involved; and the third issue is attention for careful

timing of draft-knowledge-products to create early adopters. We found that it is insufficient

to raise the awareness of potential new users to enhance the contributions made by an

established public health report, such as the PHSF Report. This is because the knowledge

product must also be geared to the needs of new users and must be introduced into the


Summary | 185

S

scenarios of new actors who may be less familiar to the researchers. However, in order to

keep established users on board, the demand for knowledge product adaptations has to be

carefully considered. This requires continuous alignment efforts in all directions: horizontal

and vertical, external and internal.

Chapter6.ResearchforPolicy(R4P):Developmentofareflectiontoolforresearchersto

improve knowledge utilization

In the three previous chapters, we extensively evaluated complex research projects

commissioned by the Ministry of Health to investigate how alignment is achieved in a

research process and to discover ways to enhance knowledge contributions to health

policy. We found that alignment between researchers and policy-makers during knowledge

production is essential, but difficult to maintain. In this chapter, we describe how the

findings of the three case studies were integrated into a practical tool for researchers to

enhance their contribution to evidence-informed policy.

By cross-case analysis of the three case studies, we identified the following eight important

alignment areas:

1. Goal of the project;

2. Tasks and authorities of involved actors;

3. Quality of the research method;

4. Consultative structure in the project;

5. Vertical alignment within organizations;

6. Organizational environment of the project;

7. Relevance and timing of knowledge products;

8. Presentation of research findings.

Since we found that alignment between research and policy-making requires continuous

efforts and a clear understanding of process issues in the research project, we prepared a

tool for supporting researchers in their alignment efforts. Our starting point was that we

want to encourage researchers to reflect on process issues in order to become aware of any

need for alignment.

In an iterative process, we developed a tool that includes practical reflection questions for

researchers taking into account the eight alignment areas. This Research for Policy (R4P) tool

R1R2R3R4R5R6R7R8R9


186 | Summary

was tested with input from the RIVM project managers of three new research projects in

2015. They regarded the R4P tool as a useful checklist for alignment in a research project.

Based on their feedback, illustrative examples from the case studies were added to the

reflection questions. The project managers suggested making the tool accessible not only

to researchers but also to policy-makers. We adjusted the format of the R4P tool further to

the users’ needs by adding clickable links. In this way, the R4P tool offers practical alignment

guidance and facilitates reflection on process issues, which supports researchers in aligning

with policy-makers and in acting in a context-sensitive way.

Chapter 7. Discussion and Conclusion

In this chapter, the study as a whole is discussed including a reflection on the findings in

relation to the research questions and on the methods applied. Finally, it provides some

perspectives for further research and ends with the conclusion of this study.

The findings of the three case studies (chapter 3, 4 and 5) and the cross-case analysis

(chapter 6) provided an answer to the first research question:

What alignment areas are important for enhancing the contributions to health policy of

government commissioned research commissioned by National Public Health Institutes?

From the case studies, we identified eight alignment areas that need specific attention

to prevent unaligned situations during the research project (chapter 6). These areas are

consistent with critical factors for knowledge utilization mentioned in scientific literature.

However, another crucial finding is that reaching alignment in practice turns out to be very

difficult. It requires ongoing attention to details and awareness of the critical factors, for

knowledge about the other stakeholders’ concerns and for political sensitivity. This causes

alignment efforts to be much more time-consuming than anticipated and researchers need,

apart from enthusiasm for scientific issues, enthusiasm for process issues in order to sustain

in conducting alignment efforts.


Summary | 187

S

The second research question was:

Based on the empirical findings gained in this study, what is a practical approach to enhance

knowledge contributions from NPHIs to evidence-informed health policy-making at national

level?

From the case studies, we learned that effective alignment needs continuous reflection

on the process. We developed a reflection tool, called R4P tool, for making the findings

available for researchers in a practical way and offering them support in determining

the type and intensity of the alignment efforts needed (chapter 6). Nevertheless, we

argue that the R4P tool can only be effectively used in an organizational framework that

meets some preconditions. Since continuous attention for process issues is difficult to

sustain, researchers need support in reflection and alignment efforts. The most important

preconditions for sufficient alignment are (1) a facilitating organization, (2) alignment as a

two-way process and (3) the right balance between scientific independency and alignment

with the commissioner’s needs.

A facilitating organization of a knowledge institute rewards not only scientific achievements,

but also alignment efforts for ensuring a smooth process, since this motivates researchers

to spend time and energy to this often-invisible part of their job. Furthermore, alignment

benefits from flexibility in interactions of researchers with policy-makers, which implies

that the margin of manoeuvre for undertaking alignment efforts has to be sufficient for all

organizational levels. For knowledge institutes, vertical alignment should be an important

point of concern since we found that lack of it clearly hinders knowledge contributions. In

addition, both policy-making organizations and NPHIs should consider how to implement

a certain level of alignment assurance by their organization, for example by designating

special alignment experts who develop alignment strategies, who support their colleagues

in implementing them and who keep the organization active on alignment issues.

Reaching alignment needs a two-way process. For an optimal outcome, both the NPHI and

the commissioning organization should spend time and energy to the process and commit

to invest in alignment efforts. We observed that for a knowledge product embedded in a

formal procedure, as is the case for the PHSF report, commitment to the process increases

at both the NPHI and the governmental commissioning organization. Nevertheless, small

projects with lesser strategic importance also need two-way alignment efforts at different

organizational levels since they all contribute to the relationship between NPHI and

commissioning organization in the long term.

R1R2R3R4R5R6R7R8R9


188 | Summary

Alignment with the commissioner’s needs to get optimal contributions to policy-making

may apparently conflict with the NPHI’s scientific independence. Nowadays, the authority

of governmental organizations is not accepted by definition anymore and people publicly

question the NPHI’s advices, for example on vaccination. The challenge for researchers is

to align with policy-makers and to remain aligned while persevering in their independent

scientific role and not creating any alleged conflict of interest. NPHIs should therefore

support their staff in developing the specific skills needed for working at this nexus of

science and health policy.

In this study, we used a qualitative case study approach that enabled an in-depth examination

of the cases. For data analysis, the Contribution Mapping approach offered a functional

three-phase model to unravel alignment efforts. During the development of the R4P tool,

it became clear that the reflection questions for the three different phases showed overlap

and needed to be largely identical. The final R4P tool is therefore organized by process topics

instead by the three phases of the research process. We consider Contribution Mapping

a valuable method for evaluating the alignment system of a knowledge institute, since it

reveals institutional patterns to be tackled. Since the method requires much time and effort,

using it for the evaluation of a small project will be less obvious.

Perspectives for further research include further development of the R4P tool, European

differences in alignment and the implications of skepticism towards science. The R4P tool

has to be validated for alignment-outcome and can be extended to other research domains.

For operating in European and international networks, it is useful to investigate whether

cultural differences between countries affect alignment and alignment efforts. These

insights can be used to improve the contribution of scientific knowledge to health policy at

international level.

Based on our study, we argue that knowledge institutes have to acknowledge that reaching

alignment needs much more effort, time and resources than often presumed. Conducting

research at an NPHI and generating knowledge products in commission of governmental

organizations to serve evidence-informed health policy is a delicate balancing act between

alignment and independence. By organizing the institute in a way that facilitates and

supports this balancing act, the NPHI may optimize its contributions to health policy.


Samenvatting | 189

S

Samenvatting

Hoofdstuk 1. Inleiding

In dit proefschrift beschrijven we een empirische studie over hoe bijdragen van

wetenschappelijke kennis aan gezondheidsbeleid tot stand komen. We hebben onderzocht

hoe deze bijdragen kunnen worden versterkt door gerichte aandacht voor de processen die

plaatsvinden in de relatie tussen onderzoek en beleidsvorming en door rekening te houden

met de steeds veranderende context van het onderzoeksproces.

We hebben ons specifiek gericht op het gebruik van kennis die wordt geproduceerd ten

behoeve van het nationale overheidsbeleid door kennisinstituten, zoals de nationale

volksgezondheidsinstituten (NPHIs; National Public Health Institutes in het Engels). Dergelijke

kennisinstituten integreren en duiden relevante informatie voor beleidsontwikkeling

en presenteren vervolgens deze kennis meestal op een wijze die tegemoet komt aan de

behoeften van de beleidsmakers, in de vorm van rapporten, instrumenten, websites,

infographics, of andere kennisproducten. In de praktijk ervaren de onderzoekers echter dat

het gebruik van de door hun geleverde kennis bij de beleidsvorming niet vanzelfsprekend

is, ondanks dat rekening is gehouden met de specifieke kennisbehoeften van de

beleidsmakers. De overheid streeft weliswaar naar zogenaamde ‘evidence-informed health

policy-making’, waarbij in de besluitvorming de beste beschikbare onderzoeksgegevens

en wetenschappelijke kennis worden betrokken, maar het is daarnaast belangrijk om in

gedachten te houden dat ook andere soorten informatie dan wetenschappelijke kennis,

zoals ideeën, belangen en praktijkervaring, relevant en invloedrijk zijn bij het aanpakken

van complexe en vaak hardnekkige problemen in de volksgezondheid en gezondheidszorg.

Wanneer NPHI-onderzoekers effectief willen bijdragen aan beleidsontwikkeling, hebben

ze, naast hun specifieke wetenschappelijke expertise, ook inzicht in kennisgebruik bij

beleidsvorming nodig en ondersteuning in de vorm van praktische hulpmiddelen.

Het doel van deze studie was om inzicht te verkrijgen in hoe kennisinstituten, zoals NPHIs, de

bijdragen van de kennis geproduceerd in opdracht van overheidsorganisaties aan evidence-

informed beleid kunnen versterken. Hiervoor hebben we de volgende onderzoeksvragen

geformuleerd:

1. Welke afstemmingsgebieden zijn belangrijk om de bijdragen aan gezondheidsbeleid te

versterken van projecten uitgevoerd door NPHIs in opdracht van de overheid?

R1R2R3R4R5R6R7R8R9


190 | Samenvatting

2. Op basis van de empirische bevindingen van deze studie, wat is een praktische aanpak om

kennisbijdragen van NPHI’s aan evidence-informed gezondheidsbeleid op nationaal niveau

te verbeteren?

Voor deze studie leverde Contribution Mapping het theoretische kader. We hebben

casestudies uitgevoerd, waarin we gedetailleerd het proces van drie projecten van het

Nederlandse Nationaal Instituut voor Volksgezondheid en Milieu (RIVM) in kaart hebben

gebracht en vooral aandacht hebben besteed aan de inspanningen om af te stemmen

(alignment efforts) in de projecten en de bijdragen (contributions) van de projecten aan

het gezondheidsbeleid op nationaal niveau. Op basis van de bevindingen uit de casestudies

hebben we vastgesteld wat de belangrijkste onderwerpen zijn om over af te stemmen als

je een optimale bijdrage van kennis aan gezondheidsbeleid wilt bereiken. Op basis van

het theoretische kader en de empirische bevindingen, hebben we tot slot een concept-

instrument dat tot doel heeft onderzoekers te ondersteunen in het afstemmen, ontwikkeld

en geëvalueerd.

Hoofdstuk 2. De complexe relatie tussen onderzoek en gezondheidsbeleid: een beknopt

overzicht van theoretische benaderingen

Het blijkt vaak moeilijk om te bewerkstelligen dat wetenschappelijke kennis en bewijs

uit onderzoek daadwerkelijk in gezondheidsbeleid verwerkt worden: een probleem dat

al jarenlang het onderwerp is van uitgebreid wetenschappelijk onderzoek in de sociale

wetenschappen. Hoofdstuk 2 geeft een beknopt overzicht van de theoretische achtergrond

voor de complexe relatie tussen onderzoek en gezondheidsbeleid, waarin we een invloedrijke

typologie van kennisgebruik en opeenvolgende theoretische modellen over kennisgebruik

beschrijven.

Kennis kan op verschillende manieren gebruikt worden. Weiss heeft een invloedrijke

typologie van kennisgebruik geformuleerd, waarin instrumenteel, conceptueel en symbolisch

gebruik van kennis worden onderscheiden. Instrumenteel gebruik betekent dat technische,

objectieve gegevens op een directe, specifieke manier worden gebruikt om een probleem

aan te pakken. Conceptueel gebruik komt tot stand wanneer onderzoek aanleiding geeft tot

nieuwe denkbeelden bij beleidsmakers en zo hun perceptie van een probleem verandert.

Wanneer bij een beslissing of in een discussie onderzoek gebruikt wordt als argument voor

een bepaald standpunt, beïnvloedt onderzoek het beleid op een symbolische manier.


Samenvatting | 191

S

Om de complexiteit van kennisgebruik te kunnen hanteren, zijn modellen ontwikkeld om

het daadwerkelijke kennisgebruik te verklaren en barrières en bevorderende factoren voor

kennisgebruik te identificeren. De rationele positivistische modellen beschouwen kennis als

een product voor gebruik in beleidsontwikkeling op een specifiek moment. Deze modellen

veronderstellen twee verschillende werelden van wetenschap en beleidsontwikkeling en

weerspiegelen nauwelijks de complexe en incrementele aard van beleidsontwikkeling.

Interactieve modellen van de relatie wetenschap-beleid houden echter ook rekening met

de context, communicatie en interactie tussen onderzoekers en beleidsmakers om de kloof

tussen de twee gemeenschappen te overbruggen. Constructivistische modellen benadrukken

de co-creatie van kennis in een sociaal proces en beschouwen de grens tussen onderzoek

en beleid als de uitkomst van de onderhandeling die plaatsvindt tussen onderzoekers en

beleidsmakers. Het idee van afzonderlijke werelden is hierbij vervangen door hybride

ruimtes waarin onderzoekers en beleidsmakers in wederzijdse interactie kennis creëren.

In dit concept kunnen zogenaamde boundary organizations, die zowel wetenschappelijke

als beleidsmatige kenmerken hebben en zowel wetenschappers als beleidsmakers in de

kennisproductie betrekken, een belangrijke rol spelen. Achter de schermen (backstage)

wordt in een onderhandelingsproces met informele contacten tussen onderzoekers en

beleidsmakers gezocht naar consensus, ook wel boundary work genoemd. Zo wordt

vertrouwen gecreëerd en worden kennis en kennisproducten acceptabel gemaakt voor het

formele openbare podium (front stage) waar een duidelijk onderscheid bestaat in taken en

verantwoordelijkheden van onderzoek en beleid.

Alle modellen bieden op een andere manier inzicht in de relatie tussen wetenschap en

beleid en de verschillende theoretische benaderingen hebben geleid tot verschillende

benaderingen om kennisgebruik te verbeteren. De positivistische aanpak wordt weerspiegeld

in kennistranslatie-activiteiten zoals het opstellen van onderzoeksagenda’s, disseminatie

van kennis en evaluatie van gebruik en toepassing. Vanuit het interactieperspectief fungeren

kennismakelaars die onderzoekers en kennisgebruikers met elkaar in contact brengen

en verbinden, als waardevolle tussenpersonen om de kloof tussen de twee werelden te

overbruggen. In deze studie hebben we Contribution Mapping van Kok en Schuit gebruikt,

een constructivistische benadering om het proces van onderzoeksprojecten te analyseren

en inzicht te krijgen in de mogelijkheden om de afstemming tussen onderzoekers en

kennisgebruikers te verbeteren. Contribution Mapping gebruikt de term alignment

efforts voor de inspanningen van betrokkenen om tot onderlinge afstemming in het

R1R2R3R4R5R6R7R8R9


192 | Samenvatting

onderzoeksproces te komen. Alignment efforts is ook de term die we in deze Nederlandstalige

samenvatting hanteren: ze vormen het aanknopingspunt om de bijdragen van onderzoek

aan beleidsontwikkeling te verbeteren en hebben een centrale plaats in deze studie. Verder

wordt de term actorscenario gebruikt om het toekomstscenario van betrokkenen (actoren)

aan te duiden. Volgens Contribution Mapping kan kennis pas daadwerkelijk een bijdrage

leveren als het een plek krijgt in het actorscenario van (beoogde) kennisgebruikers.

Hoofdstuk 3. Analyse van de bijdragen van een onderzoeksproject uitgevoerd in opdracht

van de overheid: een casestudie

Hoofdstuk 3 beschrijft de eerste casestudie waarin we het onderzoeksproces van het RIVM-

project ‘Ontwikkeling van een risicomodel’ analyseren. In dit risicomodelproject werden in

opdracht van de Inspectie voor de Gezondheidszorg modellen ontwikkeld voor het scoren

van risico’s van klinische studies. Het doel van onze casestudie was om de alignment efforts

te identificeren die de bijdrage van het risicomodelproject aan de werkzaamheden van de

inspectie hebben beïnvloed. In lijn met de Contribution Mapping werkwijze hebben we een

analyse uitgevoerd van relevante documenten zoals de verslagen van projectbijeenkomsten.

Vervolgens hebben we interviews gehouden met actoren van RIVM en inspectie, die

betrokkenwarenbijhetproject(n=10):ditwarendeRIVM-onderzoekers,debelangrijkste

gebruikers van de risicomodellen (vier inspecteurs) en de managers van zowel RIVM als de

inspectie op twee verschillende organisatieniveaus. In een feedbacksessie met inspecteurs,

RIVM-onderzoekers en hun managers hebben we onze bevindingen besproken om de

casestudie af te ronden. Op basis van de bevindingen hebben we op inductieve wijze zes

categorieën relevante alignment efforts geïdentificeerd.

We stelden vast dat beide organisaties, RIVM en inspectie, een andere visie hadden op het

eigenaarschap van de kennisproducten van het risicomodelproject en op de onderlinge

relatie tussen hun organisaties, hetgeen resulteerde in uiteenlopende verwachtingen over

het project. De RIVM onderzoekers vonden het gebruik van de risicomodellen teleurstellend,

terwijl de inspecteurs juist positief waren over de bijdragen van de risicomodellen. Alle

geïnterviewde betrokkenen (onderzoekers, inspecteurs en hun managers) waren zich

echter van deze opmerkelijke verschillen in beleving niet bewust. We vonden zes relevante

categorieën alignment efforts: relevantie, overlegstructuur, doel en timing, taken en

zeggenschap, verticale afstemming en organisatorische omgeving. Deze belangrijke

categorieën alignment efforts omvatten zowel horizontale alignment efforts tussen de


Samenvatting | 193

S

onderzoekers en de inspecteurs, als verticale alignment efforts tussen onderzoekers dan

wel inspecteurs en hun managers binnen de eigen organisatie.

Het werd bovendien duidelijk dat alignment efforts van belang zijn op drie niveaus.

Op het eerste niveau gaat het om afstemming tussen de onderzoekers en andere direct

betrokkenen bij het project; op het tweede niveau om afstemming vanuit het project op de

organisatorische omgeving; op het derde niveau gaat het om afstemming tussen de twee

betrokken organisaties als onderdeel van hun formele en historische relatie (die ongeacht het

project bestaat). Onderzoekers kunnen de afstemming op het eerste niveau zelf verbeteren

door regelmatig te reflecteren op het proces, door zich bewust te zijn van de wederzijdse

verwachtingen en door open gesprekken met de opdrachtgevers. Op het tweede niveau is

verticale afstemming essentieel om adequaat te kunnen reageren op uitdagingen voor het

onderzoeksproject vanuit de organisatorische omgeving. Afstemming op het derde niveau is

lastiger te beïnvloeden en vereist een collectieve aanpak van alle organisatorische niveaus.

Uit de bevindingen van deze eerste casestudie werd het belang duidelijk van een

reflecterende houding op alle niveaus van de organisatie ten opzichte van de sociale kant

van onderzoeksprojecten.

Hoofdstuk 4. Verbeteren van de bijdrage van onderzoek aan gezondheidszorgbeleid: een

casestudie over de Nederlandse Zorgbalans

In een tweede casestudie hebben we het ontwikkelingsproces geanalyseerd van de

2010-editie van de Zorgbalans. De Zorgbalans is een belangrijk RIVM kennisproduct dat

rapporteert over de gezondheidszorgprestaties in Nederland op basis van indicatoren voor

de kwaliteit, de bereikbaarheid en de betaalbaarheid van de gezondheidszorg. Zowel het

RIVM als het Ministerie van Volksgezondheid, Welzijn en Sport (VWS) wil de bijdrage van de

Zorgbalans aan het gezondheidszorgbeleid versterken en het doel van deze casestudie was

manieren te identificeren om dit te kunnen bereiken. De casestudie richtte zich specifiek op

het proces dat op projectniveau plaatsvindt, waarin RIVM-onderzoekers en beleidsmakers

van het Ministerie van VWS de belangrijkste betrokkenen zijn. Om te onderzoek welke

aandachtsgebieden specifieke alignment efforts van onderzoekers en / of beleidsmakers

nodig hebben om de Zorgbalans-bijdrage aan beleid te versterken, hebben we RIVM-

onderzoekers(n=8)enbeleidsmakersvanhetMinisterievanVWS(n=10)geïnterviewd.De

interviewgegevens hebben we vervolgens geïntegreerd met bevindingen uit de analyse van

R1R2R3R4R5R6R7R8R9


194 | Samenvatting

relevante (beleids)documenten. Daarna hebben we het ontwikkelingsproces van Zorgbalans

2010 geanalyseerd volgens de Contribution Mapping aanpak.

We hebben zes aandachtsgebieden geïdentificeerd waar afstemming specifiek van belang

is om voor nieuwe edities van de Zorgbalans de bijdrage aan beleid te verbeteren. Dit zijn:

1. de inhoud van de zorgmonitor om de informatie goed uit te balanceren voor de

verschillende directoraten op het Ministerie van VWS;

2. het zogenaamde ‘backstage’-werk, om achter de schermen over de grens tussen

wetenschap en beleid te onderhandelen en informeel te kunnen zoeken naar

consensus;

3. de beschikbaarheid van medewerkers die zowel in het onderzoeksinstituut als het

ministerie werken (double-role-actors) om zo de uitwisseling van informatie en het

wederzijds begrip te versterken;

4. de (te verwachten) rapporten van andere kennisinstellingen om overlap,

tegenspraak en ongelukkige timing te voorkomen;

5. dataverzameling en datageneratie zodat de volledigheid van data en de continuïteit

van datatoelevering geborgd zijn;

6. de wijze waarop de projectresultaten worden gepresenteerd zodat het aansluit bij

de behoeften van de eindgebruikers en de opdrachtgever.

We hebben geconstateerd dat de totstandkoming van de bijdrage aan beleid van een

zorgmonitor ingewikkeld is en dat onderzoekers en beleidsmakers daarvoor voortdurend

moeten afstemmen. Alignment efforts moeten daarom een onlosmakelijk onderdeel

vormen van de ontwikkeling van een zorgmonitor. Dit vergt wel een substantieel deel van

het budget, maar is het overwegen waard omdat het zich terugbetaalt in een betere bijdrage

aan beleid.

Hoofdstuk 5. De bijdrage van kennisproducten aan gezondheidsbeleid: een casestudie

over de Volksgezondheid Toekomst Verkenning 2010

Hoofdstuk 5 beschrijft de casestudie over de Volksgezondheid Toekomst Verkenning (VTV)

van 2010. De VTV integreert onderzoeksgegevens over publieke gezondheid en identificeert

toekomstige trends die de volksgezondheid in Nederland beïnvloeden. De VTV heeft een

vaste plaats in de nationale beleidscyclus voor de publieke gezondheid en wordt iedere vier

jaar uitgebracht door het RIVM. Het RIVM beschouwt alle beleidsmakers van het ministerie


Samenvatting | 195

S

van VWS als belangrijkste beoogde gebruikers, maar RIVM-onderzoekers ervaren dat verre

van alle beleidsmakers bekend zijn met de VTV en dat de groep gebruikers nog flink kan

worden uitgebreid.

Om te onderzoeken hoe de VTV-bijdrage aan het nationale gezondheidsbeleid kan worden

versterkt, hebben we het ontwikkelingsproces van de VTV 2010 geanalyseerd. Daarbij

hebben we net als in de twee voorgaande casestudies Contribution Mapping toegepast. Om

data te verzamelen omvatte deze studie de analyse van documenten uit het VTV-2010 proces

en semigestructureerde diepte-interviews met belangrijke spelers, die vanuit RIVM en het

Ministerie van VWS betrokken waren bij de VTV 2010. Interviewers, die onafhankelijk waren

van RIVM en het Ministerie van VWS, voerden de interviews uit met de RIVM-onderzoekers

enhunmanagersopdrieniveaus(n=10)enmetdeVWSbeleidsmakersenhunmanagers

opdrieniveaus(n=10).Vervolgensidentificeerdenwebelangrijkegebiedenvooralignment

efforts met behulp van een constante vergelijkende analysemethode.

De VTV wordt in de literatuur beschouwd als voorbeeld van succesvol ‘boundary work’

(grenswerk). Het VTV-2010-proces bevatte inderdaad verschillende vaste activiteiten gericht

op de afstemming tussen onderzoekers en beleidsmakers, zoals informele vergaderingen.

Toch hebben we drie onderwerpen geïdentificeerd, die gemakkelijk over het hoofd worden

gezien in kennisontwikkeling, maar wel een aanknopingspunt bieden voor een betere

bijdrage aan beleid. Het eerste onderwerp is aandacht voor de verschillende en voortdurend

veranderende actorscenario’s; het tweede onderwerp is aandacht voor effectieve verticale

afstemming binnen de betrokken organisaties; en het derde onderwerp is aandacht voor

zorgvuldige timing van commentaarrondes voor concept-kennisproducten om zo ‘early

adopters’ van kennisproducten te creëren. We hebben geconstateerd dat voor het vergroten

van de bijdrage aan beleid van een bestaand volksgezondheidsrapport als de VTV, het

onvoldoende is om van potentiële nieuwe gebruikers aandacht te vragen voor het bestaande

product. Het kennisproduct moet namelijk ook afgestemd worden op de behoefte van de

nieuwe gebruikers en moet worden opgenomen in het actorscenario van deze nieuwe, voor

de onderzoekers minder bekende gebruikers. Om bestaande gebruikers ook binnenboord te

houden, moet de noodzaak voor het aanpassen van een bestaand kennisproduct echter wel

zorgvuldig worden overwogen. Dit vereist voortdurende alignment efforts in alle richtingen:

horizontaal en verticaal, extern en intern.

R1R2R3R4R5R6R7R8R9


196 | Samenvatting

Hoofdstuk 6. Research for Policy (R4P): ontwikkeling van een reflectietool voor

onderzoekers om kennisgebruik te verbeteren

In de drie vorige hoofdstukken hebben we uitgebreid drie complexe onderzoeksprojecten, die

waren uitgevoerd in opdracht van het Ministerie van VWS, geëvalueerd om te onderzoeken

hoe de afstemming in een onderzoeksproces bereikt wordt en om mogelijkheden te

ontdekken voor het verbeteren van de kennisbijdrage aan het gezondheidsbeleid. We

constateerden dat afstemming tussen onderzoekers en beleidsmakers tijdens de productie

van kennis weliswaar essentieel, maar ook moeilijk vast en vol te houden is. In dit hoofdstuk

beschrijven we hoe we de bevindingen van de drie casestudies integreerden in een praktische

tool voor onderzoekers om hun bijdrage aan evidence-informed beleid te versterken.

Door middel van een cross-case analyse van de drie casestudies identificeerden we de

volgende acht belangrijke gebieden voor afstemming:

1. het doel van het project,

2. de taken en verantwoordelijkheden van de betrokkenen,

3. de kwaliteit van de onderzoeksmethode,

4. de overlegstructuur van het project,

5. de verticale afstemming binnen organisaties,

6. de organisatorische omgeving van het project,

7. de relevantie en de timing van kennisproducten,

8. de presentatie van de onderzoeksresultaten.

Aangezien we hebben vastgesteld dat afstemming tussen onderzoek en beleid

voortdurende inspanningen (alignment efforts) vereist met daarbij inzicht in het proces

van het onderzoeksproject, hebben we een instrument ontwikkeld om onderzoekers te

ondersteunen in hun alignment efforts. Ons uitgangspunt daarbij was dat we onderzoekers

willen aanmoedigen om te reflecteren op het proces om zo bewust te worden van een

eventuele noodzaak tot afstemmen.

Rekening houdend met de acht geïdentificeerde afstemmingsgebieden hebben we in een

iteratief proces een hulpmiddel (tool) met praktische reflectievragen voor de onderzoeker

ontwikkeld. Deze Research for Policy tool (R4P-tool) werd getest door de RIVM projectleiders

van drie nieuwe onderzoeksprojecten in 2015. Zij vonden de R4P-tool nuttig als checklist voor

afstemming in een onderzoeksproject. Op basis van hun feedback werden verhelderende


Samenvatting | 197

S

voorbeelden uit de casestudies toegevoegd aan de reflectievragen. De projectleiders stelden

voor om de R4P-tool niet alleen toegankelijk te maken voor onderzoekers maar ook voor

beleidsmakers. We hebben het format van de R4P-tool verder aangepast aan de behoefte

van de gebruikers door aanklikbare links toe te voegen. De R4P-tool biedt zo een praktische

richtlijn voor de benodigde afstemming en hulp bij reflectie op het proces, wat onderzoekers

ondersteunt bij het afstemmen met beleidsmakers en bij het omgevingsbewust handelen.

Hoofdstuk 7. Discussie en conclusie

In dit hoofdstuk wordt de studie als geheel besproken, inclusief een reflectie op de

onderzoeksbevindingen in relatie tot de onderzoeksvragen en een reflectie op de toegepaste

methoden. Tot slot worden mogelijke invalshoeken voor verder onderzoek benoemd en

wordt het hoofdstuk afgerond met de conclusie van deze studie.

De onderzoeksbevindingen van de drie casestudies (hoofdstukken 3, 4 en 5) en de cross-

case analyse van de drie casestudies (hoofdstuk 6) leverden een antwoord op de eerste

onderzoeksvraag:

Welke afstemmingsgebieden zijn belangrijk om de bijdragen aan gezondheidsbeleid te

versterken van projecten uitgevoerd door NPHIs in opdracht van de overheid?

Uit de casestudies hebben we acht afstemmingsgebieden geïdentificeerd waarvoor

specifieke aandacht nodig is om te voorkomen dat niet-afgestemde situaties tijdens het

onderzoek ontstaan (hoofdstuk 6). Deze gebieden zijn in overeenstemming met kritische

factoren voor kennisgebruik zoals genoemd in de wetenschappelijke literatuur. Een andere

belangrijke bevinding is echter dat het bereiken van afstemming in de praktijk erg moeilijk

is. Afstemming vereist voortdurend aandacht voor details, bewustzijn van kritische factoren,

kennis over de belangen en zorgen van de andere stakeholders en politieke sensitiviteit. Dit

zorgt ervoor dat alignment efforts veel meer tijd kosten dan voorzien en dat onderzoekers

naast interesse in wetenschappelijke problemen ook interesse in het proces nodig hebben

om te kunnen volharden in alignment efforts.

R1R2R3R4R5R6R7R8R9


198 | Samenvatting

De tweede onderzoeksvraag was:

Op basis van de empirische bevindingen van deze studie, wat is een praktische aanpak om

kennisbijdragen van NPHI’s aan evidence-informed gezondheidsbeleid op nationaal niveau

te verbeteren?

In de case-studies zagen we dat voor goede afstemming voortdurende reflectie op het

proces nodig is. We ontwikkelden de reflectie-tool R4P om onze bevindingen op een

praktische manier beschikbaar te maken voor onderzoekers en hen ondersteuning te

bieden bij het bepalen van het benodigde type en de benodigde omvang van alignment

efforts (hoofdstuk 6). Toch willen we benadrukken dat de R4P-tool alleen effectief gebruikt

kan worden in een organisatorische omgeving die aan bepaalde voorwaarden voldoet.

Aangezien continue aandacht voor het proces lastig is vol te houden, hebben onderzoekers

ondersteuning nodig bij reflectie en alignment efforts. De belangrijkste randvoorwaarden

voor voldoende afstemming zijn (1) een faciliterende organisatie, (2) afstemming als een

tweerichtingsproces en (3) een juiste balans in de wetenschappelijke onafhankelijkheid en

het afstemmen met de behoeften van de opdrachtgever.

Een faciliterende organisatie van een kennisinstituut beloont niet alleen wetenschappelijke

prestaties maar ook de alignment efforts om het proces goed te laten verlopen, zodat

onderzoekers gemotiveerd worden om tijd en energie te besteden aan dit vaak onzichtbare

deel van hun baan. Verder heeft afstemming baat bij enige flexibiliteit in de interacties van

onderzoekers met beleidsmakers, hetgeen betekent dat er voldoende manoeuvreerruimte

op alle organisatorische niveaus moet zijn voor alignment efforts. Verticale afstemming

zou een belangrijk aandachtspunt voor kennisinstituten moeten zijn, aangezien we zagen

dat te weinig verticale afstemming duidelijk de kennisbijdrage belemmert. Ook zouden

zowel de beleids-makende organisaties als NPHIs kunnen overwegen om een vorm van

afstemmingsborging te implementeren, bijvoorbeeld door speciale afstemmingsexperts aan

te stellen, die afstemmingsstrategieën ontwikkelen, die hun collega’s ondersteunen om die

strategieën te gebruiken en die de organisatie alert houden voor afstemmingsvraagstukken.

Om afstemming te bereiken is een tweerichtingsproces nodig. Voor een optimaal resultaat

moeten zowel het NPHI als de opdrachtgever tijd en energie te steken in het proces en zich

committeren om te investeren in alignment efforts. We zagen dat bij een kennisproduct dat

is ingebed in een formele procedure, zoals bij de VTV, de inspanningen om het proces goed te

laten verlopen toenemen bij zowel het NPHI als de opdrachtgevende overheidsorganisatie.


Samenvatting | 199

S

Toch zijn alignment efforts op verschillende organisatorische niveaus in twee richtingen

ook nodig voor de kleine projecten van minder strategisch belang, aangezien alle projecten

bijdragen aan de relatie tussen de NPHI en de opdrachtgever op lange termijn.

Afstemming met de behoefte van de opdrachtgever om tot een optimale bijdrage aan beleid

te komen, kan in strijd lijken met de wetenschappelijke onafhankelijkheid van het NPHI.

Tegenwoordig wordt het gezag van overheidsorganisaties niet zonder meer geaccepteerd

en stellen mensen de NPHI adviezen, bijvoorbeeld over vaccinatie, openlijk ter discussie.

De uitdaging voor onderzoekers is om af te stemmen met beleidsmakers en afgestemd te

blijven, terwijl ze tegelijk hun onafhankelijke wetenschappelijke rol vasthouden en daarbij

geen schijnbare belangenverstrengeling creëren. NPHIs moeten daarom hun medewerkers

ondersteunen bij het ontwikkelen van de specifieke vaardigheden die nodig zijn voor het

werken in de nexus van wetenschap en gezondheidsbeleid.

In deze studie hebben we een kwalitatieve casestudie benadering gebruikt die een

diepgaand onderzoek van de projecten mogelijk maakte. Voor de data analyse bood

Contribution Mapping een functioneel driefase-model om de alignment efforts te

ontrafelen. Tijdens de ontwikkeling van de R4P-tool werd duidelijk dat de reflectievragen

voor de drie verschillende fasen elkaar overlappen en grotendeels identiek moesten zijn. De

uiteindelijke R4P-tool is daarom ingedeeld op proces-onderwerpen in plaats van op de drie

fasen van het onderzoeksproces. We beschouwen Contribution Mapping een waardevolle

methode voor het evalueren van het alignment-systeem van een kennisinstituut, omdat het

de institutionele patronen blootlegt waarvoor actie vereist is. Aangezien de methode veel

tijd en moeite vergt, ligt toepassing voor de evaluatie van een klein project minder voor de

hand.

Mogelijke invalshoeken voor verder onderzoek zijn de verdere ontwikkeling van de R4P-tool,

Europese verschillen in afstemming en alignment efforts en de implicaties van activistisch

scepticisme ten opzichte van wetenschap. De R4P-tool moet gevalideerd worden op

effectiviteit (daadwerkelijke meer afstemming) en kan uitgebreid worden naar andere

onderzoeksdomeinen. Voor werken in Europees verband en in internationale netwerken

is het nuttig om te onderzoeken of culturele verschillen tussen landen invloed hebben

op afstemming en alignment efforts. Deze inzichten kunnen worden gebruikt om ook op

internationaal niveau de bijdrage van wetenschappelijke kennis aan gezondheidsbeleid te

verbeteren.

R1R2R3R4R5R6R7R8R9


200 | Samenvatting

Op basis van onze studie stellen we dat kennisinstituten moeten erkennen dat het

bereiken van afstemming veel meer inspanning, tijd en middelen nodig heeft dan vaak

wordt verondersteld. Het uitvoeren van onderzoek bij een NPHI en het genereren van

kennisproducten in opdracht van overheidsorganisaties ten behoeve van evidence-

informed gezondheidsbeleid vraagt de nodige behendigheid om de juiste balans te houden

tussen afstemming en onafhankelijkheid. Door het instituut te organiseren op een wijze

die dit balanceren vergemakkelijkt en ondersteunt, kan het NPHI zijn bijdragen aan het

gezondheidsbeleid optimaliseren.


Dankwoord | 201

D

Dankwoord

Mijn promotietraject was een lang en leerzaam proces, waarin ik veel hulp, morele steun en

vertrouwen van de mensen om mij heen heb ondervonden. Daar ben ik bijzonder dankbaar

voor!

Allereerst wil ik mijn promotores Hans van Oers en Jantine Schuit bedanken. Zij hebben

steeds ruim tijd voor mijn begeleiding vrij gemaakt. In onze IKU-overleggen was de sfeer

bijzonder goed en hebben we ook veel gelachen.

Hans, jij hebt mij steeds weer teruggebracht naar de kern van het probleem en mij geholpen

door de bomen het bos te blijven zien. Je vertrouwen in mij heeft mij enorm geholpen in de

afgelopen jaren.

Jantine, jouw visie en inbreng over de context van ons werk zijn heel waardevol voor mij

geweest. Je verlaat spijtig genoeg het RIVM, tegelijk is het ook mooi dat je de decaan bent

geworden van de faculteit waar ik mijn proefschrift mag verdedigen.

In de dagelijkse praktijk werd ik in mijn onderzoek gesteund door mijn co-promotor en

leidinggevende Susan Janssen. Susan, naast je inhoudelijke bijdrage aan mijn proefschrift,

heb je mij steeds de ruimte en gelegenheid gegeven om te werken aan dit onderzoek en mij

aangemoedigd vol te houden. Daar wil ik je heel graag voor bedanken!

Zonder de directe medewerking van een aantal mensen had ik het onderzoek niet kunnen

uitvoeren. Jolanda Keijsers was als meedenker van het eerste uur betrokken bij de opzet

van de studie en twee artikelen. Joyce de Goede heeft mij op weg geholpen na haar eigen

promotietraject en was steeds beschikbaar voor advies. Lisanne Marks heeft aanvankelijk

als student en later als collega meegewerkt aan de studie naar de Zorgbalans 2010 en

de ontwikkeling van de R4P-tool. Tijdens haar studie is Lisanne vanuit iBMG, Erasmus

Universiteit begeleid door Roland Bal, die ook een bijdrage heeft geleverd aan het overzicht

van theoretische benaderingen van kennis voor beleid. Maarten Kok heeft mij geïnspireerd

met zijn theoretische beschouwingen en een aandeel geleverd in de studie naar de VTV

2010. Peter Schouten was als student betrokken in het eerste onderdeel van de studie naar

VTV 2010 bij de start van het onderzoek. Jolanda, Joyce, Lisanne, Roland, Maarten en Peter,

bedankt voor jullie waardevolle bijdrage aan mijn onderzoek.

R1R2R3R4R5R6R7R8R9


202 | Dankwoord

De leden van de leescommissie bedank ik van harte voor alle tijd en moeite die zij hebben

besteed aan het beoordelen van mijn proefschrift. Geachte leden van de leescommissie, ik

stel het bijzonder op prijs dat u allen mij zo verder heeft willen helpen.

Als senior RIVM onderzoeker kreeg ik de mooie kans om alsnog te promoveren. Ik bedank

mijn werkgever RIVM, vertegenwoordigd in de personen Annemieke de Vries, Xandra

Gravestein, Henriëtte Treurniet, Rogier Bos, Annemiek van Bolhuis en André van der Zande,

dat ik hiertoe in de gelegenheid ben gesteld.

Mijn naaste collega’s zijn altijd belangrijk voor mij, maar hun onmisbaarheid voelde ik

zeker tijdens mijn promotietraject. Dank jullie wel voor de feedback die specifiek op mijn

proefschrift was gericht, maar ook voor alle collegialiteit, steun en gezelligheid die jullie mij

op het werk schenken! Gelukkig houdt onze samenwerking ook na de afronding van mijn

proefschrift niet op.

Terwijl ik mijn promotieonderzoek verrichtte, ging het leven natuurlijk gewoon door en

bracht naast fijne gebeurtenissen ook turbulenties met moeilijke momenten. Het was voor

mij belangrijk dat ik op die momenten zo veel mensen om mij heen had die mij op een of

andere manier hebben gesteund en mij aanmoedigden de draad van het proefschrift vast

te houden. Ik bedank mijn vriendinnen en vrienden voor hun interesse, begrip en voor alle

vrolijke ontspanning, tijdens gezellige etentjes, de repetities van het WP-koor, de NoWaBi-

wandelingen, de koffie na gedane fitness-inspanning, de maandelijkse Vrijmibo-gesprekken

en de uitjes met de Cult-club.

Mijn familie van Hegger-, Dämmler- en Bakker-kant vormt een liefdevolle kring van

dierbare mensen om mij heen. Ik bedank mijn familieleden allemaal voor de interesse in

de vorderingen van mijn ‘boek’, de aanmoedigingen en het vertrouwen in de goede afloop

ondanks mijn lange doorlooptijd om dit af te ronden.

Een speciaal woordje richt ik tot mijn moeder. Mama, jij hebt mij met papa altijd door dik en

dun gesteund en jullie hebben altijd onvoorwaardelijk van mij gehouden, daarvoor hoefde

ik gelukkig niet te promoveren. Ik hoop nog heel lang samen met jou dingen te kunnen

ondernemen!


Dankwoord | 203

D

Mijn kinderen hebben mij steeds geïnspireerd om door te zetten, zoals zij dat tegelijkertijd

ook deden in hun eigen studies. Matthijs, Thomas en Esther, in de jaren dat dit proefschrift

tot stand kwam hebben jullie je ontwikkeld tot prachtige, liefdevolle volwassenen met een

eigen leven en lieve, leuke partners. Ik vind het fijn dat Sephora, Anouk en Richard ook een

onderdeel van mijn leven zijn geworden.

Tot slot bedank ik mijn levensmaatje. Lieve Herman, mijn promotietraject en alles

wat verder in die periode gebeurd is, hebben we samen goed doorstaan. Je hebt me

steeds aangemoedigd bij het schrijven van mijn proefschrift en mij geholpen met jouw

vanzelfsprekende zorgzaamheid, zoals het brengen van thee naar zolder en het eten klaar

hebben op een tijdstip dat handig was voor mijn schrijf-flow. Nu mijn proefschrift is afgerond,

verheug ik me op de zee aan vrije tijd samen.

R1R2R3R4R5R6R7R8R9


204 | About the author


About the author | 205

A

About the author

Ingrid Hegger was born on November 11th, 1960 in The Hague, the Netherlands. She

studied Pharmacy at the University of Leiden, the Netherlands from 1979 until 1986 and

obtained the professional title of pharmacist at the University of Utrecht, the Netherlands

in November 1987.

Since February 1988, she works at the National Institute for Public Health and the

Environment in Bilthoven, the Netherlands. She became an expert in the quality of biological

and biotechnological medicinal products and in the European regulation of medicinal

products. In her professional life, she has been involved in many national and international

activities related to the quality of biological medicinal products: the EU control authority

batch release of immunological medicinal products and plasma derived products; the

scientific assessment of biological medicinal products for marketing authorization and the

standardization of vaccines in the European Pharmacopoeia.

From 1999 onwards, the focus in her work shifted from the product quality of biologicals

towards “close-to-policy” projects in the field of medicinal products, pharmaceutical care

and the regulatory system for medicinal products. She has been involved in projects on

the regulation of medicinal products, investigational medicinal products for clinical trials,

advanced therapies, pharmaco-economics, orphan diseases, the influence of internet on

medicines use and the organization of pharmaceutical care. Currently, she is working on

personalized medicine, new technologies and the Dutch care system.

In 2011, she started as principal researcher in the RIVM research project “Improving

knowledge utilization”, granted by the RIVM strategic research programme. The study

conducted in this project is presented in this thesis.

Ingrid is married to Herman Bakker and together, they have three, now grown-up, children,

Matthijs, Thomas and Esther.

R1R2R3R4R5R6R7R8R9


206 | List of Publications (2011-2017)


List of Publications (2011-2017) | 207

L

List of Publications

(2011-2017)

Publishedjournalarticles(2011-2017)

Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFEM, van Oers HAM. Research for Policy

(R4P): a reflection tool for researchers to enhance contributions to policymaking. Implement

Sci. 2016; 11(133)

Hegger I, Kok MO, Janssen SWJ, Schuit AJ, van Oers HAM. Contributions of knowledge

products to health policy: a case study on the Public Health Status and Forecasts Report

2010. Eur JPublic Health. 2016; 26(6): 922-927.

Hegger I, Marks LK, Janssen SWJ, Schuit AJ, van Oers HAM. Enhancing the contribution of

research to health care policy-making: a case study of the Dutch Health Care Performance

Report. J Health Serv Res Policy. 2016; 21(1): 29-35.

Hegger I, Janssen SWJ, Keijsers JFEM, Schuit AJ, van Oers, HAM. Analyzing the contributions

of a government-commissioned research project: a case study. Health Res Policy Syst. 2014;

12(1): 8.

Jongen PM, van den Bogert CA, van de Laar CW, Notenboom K, Hille ET, Hegger I. Risk

indicator taxonomy for supervision of clinical trials on medicinal products. Curr Med Res

Opin. 2016; 32 (7):1269-1276.

Venhuis BJ, Keizers PHJ, Klausmann R, Hegger I. Operation resistance: A snapshot of falsified

antibiotics and biopharmaceutical injectables in Europe. Drug Test Anal. 2016; 8(3-4):398-

401.

Van Kerkhof LWM, van de Laar CWE, de Jong C, Weda M, Hegger I. Characterization of apps

and other e-tools for medication use: insights into possible benefits and risks. JMIR MHealth

UHealth. 2016 ; 4(2):e34

R1R2R3R4R5R6R7R8R9



Van Berkel JJ, Lambooij MS, Hegger I. Empowerment of patients in online discussions about

medicine use. BMC Med Inform Decis Mak. 2015; 15:24

Submitted journal articles

Hegger I, Marks L, Bal RA, van Oers HAM. The complex relationship between research and

health policy: a comprehensive overview of theoretical approaches.

Moltó-Puigmartí C, Vonk RAA, van Ommeren GL, Hegger I. A logic model for product- and

patient-focused pharmaceutical care.

RIVMreports(2011-2017)

Hegger I, Moltó-Puigmartí C. Registratie op werkingsmechanisme: Een verkenning naar de

mogelijkheden om geneesmiddelen te registreren op werkingsmechanisme. RIVM report

2016-0214. Bilthoven: RIVM; 2016.

Van Berkel J, Lambooij M, Hegger I. Luisteren naar de digitale patiënt: verkennende analyse

van gesprekken op social media over medicatie en ziekte. RIVM report 2015-0160. Bilthoven:

RIVM; 2015.

Woutersen M, Tiesjema G, Jeurissen S, de Bruijn A, Herremans J, Hegger I. Producten op

de grensvlakken Warenwet-Wet op de medische hulpmiddelen-Biocideverordening. RIVM

report 2015-0184. Bilthoven: RIVM; 2015.

Van Kerkhof L, van de Laar K, Schooneveldt B, Hegger I. e-Medication met behulp van apps.

RIVM report 2015-015. Bilthoven: RIVM; 2015.

Vonk RAA, van de Laar, CWE, Hegger I, Ezendam J, Janssen SWJ, Hoebert JM. Legal barriers

for the use of alternatives to animal testing: do current EU regulations and guidelines for

regulatoryacceptanceofmedicinalproductsposelegalbarriers?RIVMreport2015-0084.

Bilthoven: RIVM; 2015.

Hoebert JM, Vonk RAA, van de Laar CWE, Hegger I, Weda M, Janssen SWJ. Minds Open

Sustainability of the Euroepan regulatory system for medicinal products. RIVM report 2014-

0033. Bilthoven: RIVM; 2014.


List of Publications (2011-2017) | 209

L

Hegger I, Janssen SWJ. Betrouwbare geneesmiddeleninformatie beter toegankelijk voor de

gebruiker! Een gezamenlijke inspanning van CBG, Lareb, RIVM en Zorginstituut Nederland.

RIVM letterreport 132054001. Bilthoven: RIVM; 2014.

Hegger I, Akkermans A, Notenboom K, de Vries B. Efficiency en uitkomst van onaangekondigde

GMP-inspecties bij farmaceutische bedrijven. RIVM report 360006002. Bilthoven: RIVM;

2012.

Notenboom K, Molema-Buursma AR, Hegger I. Contract Research Organisaties in Nederland:

inventarisatie en kwaliteitsniveau. RIVM report 370005001. Bilthoven: RIVM; 2011.

Conferencecontributions(2011-2017)

Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFEM, van Oers HAM. Research for Policy

(R4P): a reflection tool for researchers based on the evaluation of knowledge contributions

to policy-making. Presentation at EES Conference 29 September 2016, Maastricht, the

Netherlands.

Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFEM, van Oers HAM. Research for

Policy (R4P): a reflection tool for researchers to enhance contributions to policymaking.

Presentation at FUSE conference 27 April 2016, NewCastle, UK.

HeggerI,VonkR,vandeLaarK.Thecommunitypharmacist:unknown,unloved,unwanted?

Research project on System Assessment Development for pharmaceutical patient care and

the pharmacist’s role in integrated primary care. Presentation at policy debate session,

European Forum for Primary Care Conference, 30 August -1 September, Amsterdam, the

Netherlands.

Van Berkel JJ, Lambooij MS, Hegger I. Empowerment of patients in online discussions

about medicine use. Presentation at International Conference on Health Informatics and

Technology, 27-29 July 2015, Valencia, Spain.

Van Kerkhof L, de Jong C, van de Laar K, Hegger I. Characterization of apps and other e-tools

for medication use. Presentation at International Conference on Health Informatics and

Technology, 27-29 July 2015, Valencia, Spain.

R1R2R3R4R5R6R7R8R9



Van Kerkhof L, de Jong C, van de Laar K, Weda M, Hegger I. eMedication met behulp van

apps – onderzoek naar kansen en risico’s. Presentation at Prisma Symposium, 19 mei 2015,

Amersfoort, the Netherlands.

Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFEM, van Oers HAM. Kennisversneller:

beter laten doorwerken van kennis in beleid. Presentation at Nederlands Congres

Volksgezondheid 2014, 11 april 2014, Rotterdam, the Netherlands.

Hegger I, Kok MO, Janssen SWJ, Keijsers JFEM, Schuit AJ, van Oers HAM. Verbetering van

het gebruik van kennis voor beleid en toezicht. Poster-presentation at Nederlands Congres

Volksgezondheid 2012, 11-12 April 2012.

Research for Policy A

study on improving the contribution of scientific know

ledge to evidence-informed health policy

Ingrid Hegger

Research for PolicyA study on improving the contribution of

scientific knowledge to evidence-informed health policy

Ingrid Hegger

Evidence-informed health policy-making is generally considered as an important approach

for safeguarding public health: governments should take into account the best available

research evidence in health policy-making. Most countries have established a National

Public Health Institute to support their government in essential public health operations by

activities such as health protection, population health assessment and research to produce

evidence for policy-making. However, researchers experience that achieving contributions

to health policy-making appears to be more difficult than one would expect in view of the

institute’s mission. In the body of scientific literature on knowledge utilization, alignment

between researchers and policy-makers is recognized as an important key for enhancing

contributions of scientific knowledge to policy-making.

This thesis describes a study investigating how alignment is achieved and can be improved

in research projects conducted by a National Public Health Institute in commission of

governmental organizations.

Uitnodiging

voor het bijwonen van de openbare verdediging van mijn proefschrift

Research for PolicyA study on improving

the contribution of scientific knowledge to evidence-informed

health policy

op woensdag 6 september 2017 om 16:00 precies in de aula van de

Universiteit van Tilburg,Warandelaan 2 te Tilburg

Aansluitend bent u

van harte welkom opde receptie ter plaatse

Ingrid HeggerSoestdijkseweg Zuid 19

3732 HC De [email protected]

06-38464916

Paranimfen:Thomas Bakker

[email protected]

Esther [email protected]

De aula bevindt zich in het Cobbenhagengebouw,

te bereiken via het Koopmansgebouw. Bij de universiteit is voldoende

parkeergelegenheid. Station Tilburg Universiteit is op

10 minuten loopafstand.https://www.tilburguniversity.edu/nl/

contact/campus-map/

Tilburg University Research for policy Hegger, Ingrid · van mijn proefschrift Research for Policy...

Documents

Transcript of Tilburg University Research for policy Hegger, Ingrid · van mijn proefschrift Research for Policy...