The Nigeria Experience - WHO
Transcript of The Nigeria Experience - WHO
Operational Manual for Strengthening Institutional Capacity to Employ Evidence in Health Policymaking for Developing Countries The Nigeria Experience
Knowledge
Translation
Platform
Health Policy & Systems Research Project
Ebonyi State University Abakaliki Nigeria
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 2
Operational Manual for Strengthening Institutional Capacity to Employ Evidence in
Health Policymaking for Developing Countries
The Nigeria Experience
By
Chigozie Jesse Uneke
Abel Abeh Ezeoha
Chinwendu Daniel Ndukwe
Patrick Gold Oyibo
Friday Onwe
Esther Baragha Igbinedion
Peter Nnamdi Chukwu
September 2010
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 3
Acknowledgements
The authors are grateful to the World Health Organization for the provision of financial support for
this project through the Alliance for Health Policy and Systems Research (research grant no.
2009/25025-0; PO-No. 2 00072059); for Supporting National Processes for Evidence-informed Policy
in the Health Sector of Developing Countries, 2008.
The Project Team is grateful to The Vice-Chancellor Ebonyi State University (EBSU) Abakaliki
Nigeria, Prof. Francis Idike, for his support, and to the Deputy Vice-Chancellor, Prof Egwu U. Egwu
for his encouragement. The Team wishes to thank all our Mentors from EBSU Abakaliki for their
participation in this phase of the project; these include: Prof. Ogbonnaya Ogbu, Prof Emmanuel Agha,
Dr. Festus Iyare, Dr. Bethrand Ngwu, Dr. Linus Idam, and Dr. Ben Ozor. The Team appreciates all the
policymakers, researchers, and other stakeholders in the Nigeria health sector and their various
organizations and institutions for participating in this interventional phase. The Project Team wishes to
thank Prof. John Lavis, of, McMaster University Canada, for providing valuable information and tools
used for the analysis of the data collected from target population.
Disclaimer
All views expressed in this document are those of the authors and do not necessarily reflects the views
of WHO.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 4
Table of Content
Abbreviations and acronyms……………………………………………………………………..6
Chapter 1. Introduction and general consideration.....................................................................7
1.1 Introduction and background information...................................................................................7
1.2 Project Goals and Objectives.......................................................................................................9
1.3 Purpose and scope of the Manual...............................................................................................12
1.4 Constituting a project team.........................................................................................................12
1.5 Appointment of project mentors.................................................................................................12
1.6 Target Users................................................................................................................................12
1.7 Pilot Test.....................................................................................................................................13
1.8 Organization of the Manual........................................................................................................13
Chapter 2. Overview of the methodological approach…………………………………………14
2.1 Preparatory Phase........................................................................................................................14
2.2 Base line data collection Phase...................................................................................................14
2.3 Intervention/implementation Phase.............................................................................................15
2.4 Monitoring and Evaluation Phase...............................................................................................15
2.5 Theoretical foundation underlying the methodological approach...............................................16
Chapter 3. Procedures for execution of activities of preparatory phase………………………18
3.1 Situational analysis......................................................................................................................18
3.2 Mapping of target populations.....................................................................................................18
3.3 Definition of participant category................................................................................................18
3.4 Researchers/Policy makers‟ interactive forum............................................................................19
3.5 Project monitoring and evaluation design………………………………………………………19
3.6 Data from social research.............................................................................................................21
Chapter 4. Procedures for execution of activities of baseline data collection phase..................24
4.1 Development and production of data collection instruments.......................................................24
4.2 Strategies for the administration of data collection instruments..................................................25
4.3 Analysis of data collected.............................................................................................................25
4.4 Interventional strategies to address the capacity constraints as informed by the
Preparatory and Baseline Data Collection Phases........................................................................25
Chapter 5. Overview of procedures for the intervention/implementation phase.......................36
5.1 Basis for interventional strategies.................................................................................................36
5.2 Training workshop pattern and methods for analysis of data collected........................................37
5.3 Development and production of policy briefs...............................................................................38
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 5
Chapter 6. Training workshop on capacity enhancement in information and communication
technology for policymaking...........................................................................................................40
6.1 Methods for Information and Communication Training Workshop............................................40
Chapter 7. Training workshop on capacity enhancement for acquisition, assessment, adaptation
& application of research evidence for policy making and partnership links............................45
7.1 Methods for the Training Workshop............................................................................................45
7.2 Participation of project mentors...................................................................................................47
Chapter 8. Training workshop on capacity enhancement to conceptualize, formalize, implement
and evaluate the research needed for better understanding and improved performance
of the health systems.........................................................................................................................54
8.1 Methods for the Training Workshop............................................................................................54
8.2 Policy dialogue on health systems strengthening.........................................................................55
Chapter 9. Training workshop on enhancing capacity to own and drive the agenda for
strengthening health systems/health sector reforms.....................................................................69
9.1 Methods for Training Workshop.................................................................................................69
9.2 Outcomes evaluation....................................................................................................................71
Chapter 10. Training workshop for capacity enhancement on leadership, governance and
management for health systems strengthening………………………………………………….81
10.1 Methods for Training Workshop...............................................................................................81
10.2 Staff and organizational appraisal/performance assessment.....................................................82
Chapter 11. Project evaluation design..........................................................................................89
11.1 Conceptual design of the evaluation…………………………………………………….…….89
11.2 Data from monitoring/operational research...............................................................................90
11.3 Administering the evaluation questionnaire……………………………………………….….91
Chapter 12. Conclusion and recommendations............................................................................95
References.........................................................................................................................................97
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 6
Abbreviations and acronyms
AHPSR Alliance for Health Policy and Systems Research
CHSRF Canadian health services research foundation
EBM Evidence-Based Medicine
EBP Evidence-Based Policymaking
EIP Evidence-Informed Policy making
FGD Focus Group Discussion
GHAIN Global HIV/AIDS Initiatives Network
HPSR Health Policy and Systems Research
HRM Human Resource Management
ICT Information and Communication Technology
KTP Knowledge Translation Platforms
M&E Monitoring and Evaluation
MDGs Millennium Development Goals
MDR Median rating
MNR Mean Rating
MOH Ministry of health
UNDP United Nations Development Programme
WHA World Health Assembly
WHO World Health Organization
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 7
CHAPTER 1
INTRODUCTION AND GENERAL CONSIDERATION
1.1 Introduction and background information
In most developing countries of the world, health outcomes have been described as unacceptably low
and at the centre of this human crisis is a failure of health systems (WHO 2007). According to the
World Health Organization (WHO), the health systems comprise of all the organizations, institutions
and resources that are devoted to producing health actions, and have a primary purpose of improving
health (WHO 2000a). In many low income and middle income countries, weak health systems are
impeding the success of the various health intervention programmes being initiated and implemented.
Bowen and Zwi (2005) noted that a key challenge to public health in most developing countries is to
better contextualize evidence for more effective policymaking and practice.
There is therefore increasing recognition that strong and effective health systems that are evidence-
based in their operations are necessary to achieve continued improvement in health outcomes in an
efficient and equitable manner (WHO 2008a; Travis et al. 2004). A number of previous reports have
provided convincing information to prove that evidence from research can enhance the health systems
operations and health policy process and development by identifying new issues for the policy agenda,
informing decisions about policy content and direction, or by evaluating the impact of policy
(Campbell et al. 2009; Dobrow et al. 2004; Hanney et al. 2003; Innvær et al. 2002).
There is an increasing recognition worldwide of the importance and the necessity of the use of more
rigorous processes to ensure that health care recommendations are informed by the best available
research evidence (Gonzalez-Block 2004). There is also increasing awareness of the need for the
production of new knowledge to improve how societies organize themselves to achieve health goals- a
phenomenon described as health policy and systems research HPSR (AHPSR 2007). HPSR aims to
produce reliable and rigorous evidence which helps to inform the many and varied critical decisions
that must be made by ministers of health, senior policy-makers and health service managers about how
to organize the health system and effect changes. Therefore if there must be an improvement in the
health care sector in its entirety particularly in developing countries, HPSR becomes imperative.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 8
Unfortunately, HPSR is a somewhat new phenomenon in the health sector in many developing
countries and most health researchers, health policy makers, health service managers and other major
stakeholders at government and non-governmental levels are completely ignorant of its value in
policy-making and practice (evidence-informed health service management, and evidence-informed
policy making). The major constraint to the use of evidence in policy and practice in developing
countries is the grossly deficient capacity development at the individual and organizational levels,
particularly the lack of formally trained human resources among public health policy makers and
health service managers (Uneke et al. 2009). Health systems research is further burdened with a
number of other such challenges including under-investment, lack of human capacity, lack of public
demand, inadequate utilization, and poor dissemination of results (Uneke et al. 2010).
Health policy and systems research (HPSR) has been shown to have the potential of producing reliable
and rigorous evidence which can help to inform policy development and the policymaking process
(AHPSR 2007). In most developing countries there is a lack of recognition of the importance of HPSR
and there is also little interest in transfer and uptake of research into policy and practice in these
countries (Uneke et al. 2009). A few instances where policymaking involved the use of research
evidence occur mainly on clinical decision-making (evidence-based medicine) and only in a number
of tertiary health institutions such as teaching hospitals. A major factor contributing to this problem is
the lack of both national and sub-national effort to create awareness on the value HPSR in health
policymaking process; this is in addition to the dearth of any active on-going research on HPSR in the
country.
Furthermore, there are generally no financial incentives for doing studies related to HPSR, except for a
number of foreign-funded projects and consultancies, which are usually awarded to a relatively small
and select group of health professionals who may not involve policy makers and other stakeholders.
Moreover little efforts are usually made to widely and effectively disseminate findings from such
studies. There is also insufficient coordination between donors, research community, NGOs and
government bodies as well as weak links between programs, monitoring systems and use of
information.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 9
The World Health Organization like many other international agencies is currently vigorously
supporting the process of contextualizing evidence and translating it to policy through the utility of
health policy and systems research in many developing countries (WHO 2003; AHPSR 2007). This is
in line with the resolution of the World Health Assembly (WHA) held in Geneva in May 2005, which
laid emphasis on how to harness health research more effectively in order to achieve the United
Nations‟ Millennium Development Goals in low- and middle- income countries (WHA 2005).
Capacity constraints at the individual and organizational levels are perceived to be major impediments
in HPSR evidence use in the health policymaking process in most developing countries including
Nigeria (Gonzalez-Block and Mills 2004; Uneke et al. 2009).
According to the World Health Organization, one of the greatest challenges facing the member states
is how to ensure access to safe and effective health services for those population groups most in need;
and strengthening individual and organizational capacities for a more functional health systems is a
core part of this challenge (WHO, 2007). The United Nations Development Programme (UNDP)
defined capacity as the ability of individuals, institutions and societies to perform functions, solve
problems, and set and achieve objectives in a sustainable manner (UNDP, 2006). The capacity to
direct the focus of HPSR is pivotal to shaping evidence-informed national health policies and systems
(AHPSR, 2007). Green and Bennett (2007) had noted that evidence is needed about how capacity
constraints in various countries inhibit evidence-informed health policy, and which strategies are
effective in addressing these constraints.
1.2 Project Goals and Objectives
There is a dearth of information on the status of HPSR evidence use in policymaking at individual and
institutional levels in most developing countries. The scarcity of such baseline information hampers
the following: (1). development effective of strategies to promote the application of HPSR in
policymaking; and (2). development of effective health system strengthening interventions. There was
therefore a dire need to identify the challenges associated with HPSR evidence use in policymaking in
these countries and the potential strategies to address them. There was also the need to assess the
capacity status of health policymaking actors and their organizations, in terms of individual staff skill
and institutional incentives in HPSR evidence use in policymaking. This is with the view to providing
scientific information that can be used to direct policy development and intervention program
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 10
implementation on health system strengthening and also provide baseline measurement for monitoring
and evaluating interventions.
This strategies contained in this manual were designed to assist in building sustainable and effective
health systems in developing countries and include the discovery and innovation for health systems,
i.e. in stewardship, governance, financing, resource management and informatics, as well as service
delivery. Thus the strategies are mainly focused on capacity enhancement in health policy and systems
research evidence use in policy making in any low income setting. It is well established that the
capacity to direct the focus of HPSR is pivotal to shaping evidence-informed national health policies
and systems. Consequently, this manual has been designed to enable the research community develop
a platform to work with national and sub-national health policy-makers and capable local health
system managers to promote HPSR. This is with the view to bridging the wide gap that exists between
researchers and policy makers in developing countries. The improvement of the skills of policy
makers in evidence-informed policy making and the establishment of enabling environments and
capacity for HPSR for the Health ministry and the research community are the overriding goals of this
project.
The intervention goal for evidence informed policy that form the basis of the strategies described in
this manual is within the following categories:
(a). Enhancing pull factors: The intervention to be considered includes;
(i) Establishment of rapid response mechanisms or policy advisory units within Ministries of Health
and/or other policy-making bodies or strengthening existing capacity with additional training;
(ii) Technical support and or training to help policy makers to facilitate commissioning of appropriate
research or syntheses; and
(iii) Training policy makers in identifying and applying evidence to decision making.
(b). Supporting exchange efforts: The intervention to be considered includes;
(i) Supporting forums, meetings, workshops, or conferences to facilitate exchange between researchers
and research users;
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 11
(ii) Encouraging the establishment of mechanisms which allow for ongoing exchange between
researchers and research users, such as research priority setting committees;
(iii) Supporting secondment of trusted researchers, who already have established links with policy
makers into policy making units.
The specific objectives of this program of work are therefore grouped as follows:
(1). Individual capacity strengthening: Because specific capacities, such as an individual‟s skills,
experience, values and beliefs, leadership, knowledge, resources, organizational support, partnership
links, and participation in networks, have been shown to influence the adoption and adaptation of
evidence in practice, this project is aimed at the enhancement these capacities which will undoubtedly
strengthen evidence use in policy making.
(2). Strengthening skill in developing research initiatives: Another important capacity this project is
designed to enhance is the skill in developing research initiatives on health systems. The capacity to
conceptualize, formalize, implement and evaluate the research needed for better understanding and
improved performance of the country‟s health system would be enhanced in this project. Also
included are technical support and training to help policy makers to facilitate commissioning of
appropriate research or syntheses, establishing stronger relationships with researchers, and also in
identifying, interpreting and applying evidence to decision making and tools to assess the legitimacy
of the filtering and amplification function of diverse organizations.
(3). To build enabling environments and capacity for HPSR: This is specifically targeted at the
health ministry and would include building the capacity to own and drive national/sub-national agenda
for strengthening health systems. Emphasis would be placed on how to enable Ministries of health to
take a strong lead in capacity building for all functions of the health system, including stewardship
(leadership, governance and communications), financing, resource management, informatics, service
delivery and research. The project would introduce strategies that would facilitate the development of
capacity for the generation, dissemination and use of health policy and systems research knowledge
among researchers, policy-makers and other stakeholders.
(4). The institution of a stronger incentive for evidence use: An important capacity that is required
and which is more relevant at organizational level is the institution of a stronger incentive for evidence
use. This would be achieved through the creation of stronger incentives for evidence use.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 12
1.3 Purpose and scope of the Manual
The challenges associated with translating evidence to health policy in developing countries are
enormous and complex. These challenges are compounded by the lack of capacity for evidence
informed policy making at institutional level as well as on the part of policymakers. The purpose of
this manual is to provide a description of the strategies that were used to enhance institutional capacity
for evidence informed policy making in a low income setting; with focus on strengthening the health
systems in the following areas: health service delivery, information and evidence, medical products
and technology, health financing, health workforce, leadership and governance. The strategies are also
designed to strengthen evidence-to-policy link and improve the capacity of policymakers and other
stakeholders in the health policy making process in low and middle income countries. This manual
presents detailed and general descriptions of the procedures that can be used to identify and address
the capacity constraints often encountered at individual and institutional levels in the policymaking
process of developing countries.
1.4 Constituting a project team
To conduct a project of the sort described in this manual, a team must be constituted. This team can be
made up of either policymakers or researchers or preferably a combination of both. Depending on the
scope of coverage (national, regional, sub-regional, district etc), the team cane consist of seven to as
many as fifteen members. The team should be essentially multidisciplinary in composition, making up
of individuals with background in health sciences, social sciences, anthropology/sociology,
management sciences, and information technology, etc.
1.5 Appointment of project mentors
It is important to appoint project mentors who should be essentially senior academics (Professors,
Associate Professors or Senior Lecturers) from university or other tertiary institutions. These
individuals would provide technical support and consultancies to the project and will participate in
some of the workshops as resource persons or to chair/moderate focus group discussions.
1.6 Target Users
This manual is principally designed for use by Ministries of health at federal (national), state (district)
and local (municipal) government levels. The major target end users include the following: health
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 13
professionals in charge of the health systems; national, regional, state and local officers of the health
ministry; health professionals working with specific programs in the health ministry who wish to use
HPSR to improve the impact of their strategies; staff and consultants involved in public health issues
within the health ministry; political/legal advisers on health related matters; program/project managers
under the health ministry; researchers and teachers in public health training institutions; leaders and
staff of health-based civil society groups including non-governmental, bilateral or multilateral
organizations; directors and managers of health services in the uniform services.
1.7 Pilot Test
The approach described in this manual has been pilot-tested in Ebonyi State south-eastern Nigeria.
The duration of the project was 32months and the manual also incorporates the feedback from the pilot
test with outlines of recommendation.
1.8 Organization of the Manual
This manual is organized in a way that makes easy to follow the procedures. The manual is divided
into twelve chapters. References and further reading are provided at the end of each chapter. Chapter 1
provides the Introduction and general considerations. Chapter 2 provides an overview of the
methodological approach developed and used in this project. An outline of the procedures for
execution of activities of preparatory phase is shown in Chapter 3. Chapters 4 and 5 provide the
procedures for execution of activities of baseline data collection and for the
intervention/implementation phases respectively. The descriptions of the various training workshops
are provided in Chapters 6-10. Chapter 11 describes the project evaluation design in greater details
while the conclusion and recommendations are outlined in chapter 12. Assessment tools are contained
in relevant chapters where these tools are described.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 14
CHAPTER 2
OVERVIEW OF THE METHODOLOGICAL APPROACH
The methodological approach in this manual was systematically designed to strengthen evidence-to-
policy link. The program of work to strengthen evidence-to-policy link is categorized into four phases.
The first phase is the preparatory phase; the second is the baseline data collection phase; the third
phase is the intervention/implementation phase; while the fourth is the monitoring/evaluation phase.
2.1 Preparatory Phase
This phase of the project focuses on refining the design of the intervention in order to obtain a clearer
picture of the project objectives and interventions. The preparatory phase consists of the following
activities;
(i). Advocacy/consultation visits,
(ii). Situational analysis for evidence use in policy making,
(iii). The design and execution of mechanism that would enhance proper project evaluation at the end
of the intervention, such as the use of process evaluation.
(iv). Identification of relevant training materials and tools for the implementation phase.
2.2 Base line data collection Phase
The baseline data collection is accomplished using three steps as follows:
(i). Data collection using structured pre-tested questionnaire which consists of questions on:
geographical focus of the health ministry, level of health systems research that the health ministry is
engaged in, formal/official collaboration, informal/unofficial collaboration, key activities/objectives of
the health ministry in terms of health policy and systems research and use of evidence in policy
making, key areas of work including key strengths, weaknesses, opportunities, and threats of the
health ministry in HPSR and use of evidence in policy making, level of staff awareness of HPSR and
use of evidence in policy making, computer literacy and skills in the use of information technology
pertaining to health policy and evidence use.
(ii). Key informant interviews of a selected number of individuals in the target group, conducted
face-to-face or by telephone using an interview guide.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 15
(iii). Group discussions with a focus group of 9-12 persons among the target group conducted with a
moderator or two co-moderators using a discussion guide that centred on HPSR and use of evidence
in policy making.
2.3 Intervention/implementation Phase
Two major areas of intervention include:
(i) To strengthen individual staff skills to enhance capacity to use evidence in policy-making: Through
the provision of training and mentoring in use of acquisition, assessment, adaptation and application
of research evidence, e.g. training in the commissioning of research studies and the interpretation of
systematic reviews.
(ii). To strengthen institutional incentives to employ evidence: Through the creation of stronger
incentives for evidence use (e.g. through integrating this dimension into recruitment policies,
performance assessments, and staff appraisals); integration of modules on use of evidence into
leadership training courses for policy makers; and recognition awards - providing financial or in-kind
prizes to policy makers that have shown outstanding performance in applying evidence to policy.
2.4 Monitoring and Evaluation Phase
Process evaluation is useful because it informs learning about the outcomes, and is crucial to the
success of the impact evaluation. It helps to address questions such as: What effect (positive and
negative) is the intervention perceived to have on target participants (and non-participants)? What
effect does the intervention appear to have more widely, for example in terms of cultural change?
What factors appear to underpin differing impacts and outcomes? Are there particular sub-groups
within the target group which do better or worse from the intervention and why? Do particular models
of implementation appear to produce better outcomes? If so, how and why? Are there groups for
which the intervention appears to create more sustainable outcomes?.
Two approaches of the process evaluation include:
(a) Monitoring/operational research
This which involves the use and analysis of data collected other than for the evaluation such as:
(i) The analysis of administrative data about the eligible population
(ii) The analysis of data from Management Information Systems e.g. programme database
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 16
(iii) The collection and analysis of performance measurement data about resources (staff and
financial), etc.
(iv) Special monitoring exercises e.g. pro formas attached to specific cases to identify stages and
timing
(v) In-depth area studies
(b) Social research
Designed specifically to collect data for the evaluation and this involves;
(i) large scale quantitative studies using structured questionnaire,
(ii) qualitative studies via individual and paired depth interviews, group discussions, observational
work and documentary analysis,
(iii) case studies which considers individuals from the multiple perspectives of key actors.
2.5 Theoretical foundation underlying the methodological approach
The target participants are health service/policy providers because a supply-driven outcome that would
address capacity constraints in HPSR evidence use in policymaking and policy implementation by
these individuals was anticipated. According to the AHPSR the supply-driven model has been used
extensively to design capacity strengthening initiatives in developing countries and this was based on
the assumption that if the skills of the main actors (researchers and policymakers) are enhanced via
training and enough institutional capacity is built, research outputs will be put to good use (AHPSR
2004). Although argument has been intensified in HPSR circles to focus it on the demand side, taking
the supply-driven approach has a stronger tendency to accomplish a high level of ownership of
policies. This is because health policies are implemented better when those saddled with this
responsibility are made to identify the capacity challenges and the solutions required to address these
challenges.
Capacity constraints at the individual level are perceived to be major impediments in HPSR evidence
use in the health policymaking process in most developing countries (Gonzalez-Block and Mills 2004;
Uneke et al. 2009). Green and Bennett (2007) had noted that evidence is needed about how capacity
constraints particularly among policymakers in various countries inhibit evidence-informed health
policy, and which strategies are effective in addressing these constraints. No other category of
individuals is in the best position to identify the capacity challenges of service/policy providers in
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 17
HPSR evidence use in policy than the service/policy providers themselves. This informed the
advocacy of this methodological approach for evidence-to-policy link for low income countries. It is
pertinent to state however, that the intention is not to restrict the evidence-to-policy investigation to
the supply-driven model. The goal is to first generate information using the supply-driven model and
subsequently use it to stimulate the demand driven aspect which is also very vital to achieving
evidence-based policymaking and practice. A number of earlier reports have provided evidence which
proves that supply-side capacity building strategies that do nothing to stimulate the demand for
research are unlikely to achieve expectations, and may actually further distort allocations (Bhagavan
1992; Acemoglu 1997). The essence of this approach is to address the uncoordinated “pushing” of
research results by scientists and “pulling” of research results by market oriented users (AHPSR 2004)
Focus Group Discussion (FGD) is a very useful approach because this it draws upon respondents‟
attitudes, feelings, beliefs, experiences and reactions with respect to capacity constraints in HPSR
evidence use in policymaking. This is the most feasible method of accomplishing this as other
methods such as observation, one-to-one interviewing, and questionnaire surveys do not enhance the
social gathering and the interaction which being in a focus group entails. The approach taken to elicit
information from key informants in the focus groups was described by Kitzinger (1995). The
theoretical foundations underlying this approach are based on the report of Thomas et al. (1995) who
described focus group as „a technique involving the use of in-depth group interviews in which
participants are selected because they are a purposive, although not necessarily representative,
sampling of a specific population, this group being „focused‟ on a given topic‟. Richardson and Rabiee
(2001), noted that individuals participating in a focus group are usually selected based on the fact that
they have a working knowledge of issues addressed, are within the age-range, have similar socio-
demographic characteristics and would be comfortable talking to the interviewer and each other.
According to Burrows & Kendall (see Rabiee 2004), „this approach to selection relates to the concept
of „Applicability‟, in which subjects are selected because of their knowledge of the study area.‟
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 18
CHAPTER 3
PROCEDURES FOR EXECUTION OF ACTIVITIES OF PREPARATORY PHASE
3.1 Situational analysis
A comprehensive situational analysis can be conducted as follows;
(i). A preliminary mapping of the potential stakeholders/target group that would be involved in the
intervention phase; this is with the view to identifying eligible individuals (policy makers and/or those
in position to influence the health policy making process).
(ii). A document search can be done at the offices and websites of the various organizations and some
individuals of the target group can be asked to indicate documents and research projects possibly
associated with health policy. The documents should include reports, policy briefs, bulletins, media
communications, etc., which have been produced to inform about research projects and use of
evidence in policy making. Emphasis should be placed on more recent documents/publications. These
documents should be evaluated regarding their congruence with the objectives of the intervention.
(iii). A review of available literature and databases available at the State office of the World Health
Organization (WHO), and at WHO website, and supplemented by Medline and Google search of
journal articles and other literature pertinent to evidence use in HPSR as it relates to the health sector.
3.2 Mapping of target populations
During consultation/advocacy visits, the project team should endeavour to convince heads of the
various organizations to make commitments to approve the participation of policymakers in their
organization in the project. The criteria for inclusion of participants should include;
(i). Participant must be a full-time senior staff or officer in the organization.
(ii). Participant must be a policy maker or in position to influence the policy making process in the
health sector.
3.3 Definition of participant category
Participants should be categorized into two broad groups based on how their job specifications and
nature of influence on the policymaking process as follows:
(1). Participants who have direct influence of policymaking process: These refer to individuals who
receive processed information, data, reports and submissions on health-related issues and
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 19
synthesize/translate them into items required for policy drafting. They rarely take part in data
generation/collection activities. They participate in forums/meetings where various types of policy
documents are produced including policy briefs, policy drafts, and principally the main/final policy
documents.
(2). Participants who have indirect influence of policymaking process: These refer to individuals who
are mainly involved in the generation, collection and assembling of relevant information, and
processing of data and reports on health-related issues from the different sectors of the health system
and prepare them into forms that can be submitted for the drafting of policy documents. They may
make inputs during the production of policy briefs and policy drafts but do not usually participate in
the writing of the main/final policy documents.
3.4 Researchers/Policy makers’ interactive forum
As part of strategies to standardize the study protocol, identify priority areas (priority setting), prepare
for the data collection phase, and to make the intervention phase more result oriented, an interactive
forum should be held with target participants at the Ministry of Health. During each forum the
following activities should be performed;
(a). Introduction of project team and participants
(b). An explanatory speech by Project Director
(c). Administration of a simple questionnaire on policy making capacity assessment (Box 3.1)
(d). Questions/answers and interactive session
3.5 Project monitoring and evaluation design-How the preparatory phase helps to inform the
design
The overriding goals of a project of this sort should be to improve the skills of policy makers in health
policy and systems research evidence use in policy making and to assist in building sustainable and
effective health systems in a given low income setting. To achieve the goals of the project, series of
interventional activities should be developed to be implemented and these should be targeted at
strengthening individual staff skills to enhance capacity to use evidence in policy-making, and
enhancing institutional incentives to employ evidence in policy-making. It is intended that the
processes, sets of tools, descriptions, and analyses resulting from the project will provide useful
information to identify potential policy options to strengthen evidence-informed policymaking, which
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 20
will facilitate the use of scientific knowledge to improve actions within health systems, and ultimately
improve health and health equity.
One of the major areas of interest should be to determine whether the interventions planned are
delivered as intended to the targeted recipients at the end of the project. To be able to determine this,
the process evaluation design should be adopted which is a sort of implementation assessment
mechanism. The use of process evaluation design mechanism is necessitated by the following facts:
(i) Health policy and systems research evidence use in policy making is a relatively new
programme in most low income settings and further development of the programme appears
very needful before an assessment of the impact of the final model can be done;
(ii) The various health system constituents are under-performing and thus questions have arisen
about the organization, operations, delivery, quality, and success.
Information obtained from interaction with heads of the various organizations, some policymakers
nominated as participants, and assessed the data collection system of the organizations should inform
the evaluation design by helping to improve the quality content of the evaluation and the
implementation processes. Information obtained should aid setting priorities, appropriate evaluation
questions, baselines, quality criteria and thresholds to be determined, with respect to the project
specific objectives. In line with this, certain questions have been developed to assess the extent to
which the programme is being implemented as planned; assess participant and stakeholder experience
and satisfaction with the programme; assess programme reach/output; and improve programme
performance. The key questions are as follows: (a). Is the programme well organized? (b). What are
the main models of programme organization and delivery? (c). Are the activities being implemented as
planned? (d). Is the implementation consistent with quality assurance standards? (e). Are enough of
the policymakers participating? (f). What are participants' experiences of contact with the programme?
(g). Do participants understand the nature of the programme, its intention and its various elements?
(h). Are participants satisfied with the programme? (i). Are there particular groups within the target
population which do not receive the programme and why? and (j). Are there any unintended activities
and processes? These questions were expected to provide information on whether the programme
activities were accomplished; on quality of programme components; on how well programme
activities were implemented; on whether the target population was reached; and how external factors
influenced program implementation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 21
From the assessment of data collection system of the organizations, the team should gained an
understanding of the type of relevant data available, data being collected, how and where to find
them, how to collect them, and analyze them in a meaningful way. This information will enhance the
development of the monitoring/operational research as well as the social research mechanisms which
are the strategies designed to accomplish the process evaluation design in a project of this sort (see
chapter 11 for details).
The social research mechanism of the process evaluation is described below while the
monitoring/operational research mechanism is described in chapter 11
3.6 Data from social research
The Social research evaluation approaches include;
(a) large scale quantitative studies using structured questionnaire,
(b) qualitative studies via individual and paired depth interviews, group discussions, observational
work and documentary analysis,
(c) case studies which considers individuals from the multiple perspectives of key actors.
All the materials to be used for data collection at the evaluation phase including questionnaires,
interview/group discussions guides etc., should be developed at this preparatory phase and revised as
the project progresses. Information from some publications including the self-evaluation tool
developed by Canadian Health Services research Foundation, and evaluation toolkits/guidelines
developed by CDC Atlanta, WHO, researchers etc., are useful in the standardization of the evaluation
design. Consequently four categories of data to be collected in line with the specific objectives of this
project include;
(i). Data on individual capacity: These include data on specific capacities for evidence use in health
policymaking, such as an individual‟s skills, experience, values and beliefs, leadership, knowledge,
resources, organizational support, partnership links, and participation in networks
(ii). Data on skill in developing research initiatives: These include data on the skill in developing
research initiatives on health systems, capacity to conceptualize, formalize, implement and evaluate
the research needed for better understanding and improved performance of the country‟s health
system, capacity to facilitate commissioning of appropriate research or syntheses, establishing stronger
relationships with researchers, and also in identifying, interpreting and applying evidence to decision
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 22
making and tools to assess the legitimacy of the filtering and amplification function of diverse
organizations.
(iii). Data on organizational/institutional capacity for HPSR: These include data on the capacity to
own and drive national/sub-national agenda for strengthening health systems, capacity to take a strong
lead in capacity building for all functions of the health system, including stewardship (leadership,
governance and communications), financing, resource management, informatics, service delivery and
research.
(iv). Data on incentive for evidence use: These include data on organizational/institutional capacity for
institution of a stronger incentive for evidence use.
All data collection tools including structured questionnaires, interview/group discussions guides etc
should be designed to obtain the information. In this manual all the data collection tools described in
Chapters 3-10, are used for social research. The tool for monitoring/operational research is described
in chapter 11.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 23
Box 3.1 Questionnaire for interactive forum with policymakers
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT QUESTIONNAIRE
The following questions are designed to help us determine your capacity for evidence-based policy making to enable
us channel appropriate interventional strategies to enhance your capacity on the use of research evidence for policy
making in the health sector.
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose only.
1. Designation………………………………………………………………………………
2. Duration in present designation…………………………………………………………
5. Age Category: 25- -
8. Do you use the internet to source fo
9. Do you have an e-
specify)………………………………
14. Do you have any existing health policy working document in your organization? Yes
15. Have you been involved in drafting any health policy document for your organization or the
17. If Yes, what source(s)? Education
18. Have you been involved in any health related research since assumption of your present office?
alth laboratory-
specify)……………………………………….
20. Do you think it is needful to collaborate with researchers from educational institutions in policymaking process?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 24
CHAPTER 4
PROCEDURES FOR EXECUTION OF ACTIVITIES OF BASELINE DATA COLLECTION
PHASE
The baseline data collection phase involves the development and production of data collection
tools/instruments and the design of strategies to reach the target population for the administration of
the instruments.
4.1 Development and production of data collection instruments
The data collection instruments consist of a structured questionnaire, key informant interview guide
and focus group discussion guide. The questionnaire consists of questions which centred on the
following: geographical focus of the organization, level of health systems research that the
organization is engaged in, formal/official collaboration, informal/unofficial collaboration, key health-
related activities/objectives of the organization, in terms of health policy and systems research and use
of evidence in policy making, key areas of work including key strengths, weaknesses, opportunities,
and threats of the organization, in HPSR and use of evidence in policy making, level of staff
awareness of HPSR and use of evidence in policy making, computer literacy and skills in the use of
information technology pertaining to health policy and evidence use. The questionnaire is adapted
from the self-evaluation tool developed by Canadian Health Services Research Foundation [10], but
with some modifications to accommodate the objectives of the project and peculiarities of the low
income health systems. The measurement strategies include the use of the likert scale rating of four
options (1. grossly inadequate, 2. inadequate, 3. fairly adequate, 4. very adequate); yes or no; and
closed end answers. The questionnaire is outlined Box 4.1.
The interview and focus group discussion guide comprise of questions which centred on:
Identification of specific capacity constrains and challenges which impede the development of HPSR
evidence use; identification of potential strategies and solutions that would address capacity constrains
to HPSR evidence use in policy making; identification of critical gaps in HPSR evidence use in policy
making, with a focus on improving public health and; identification of the barriers to, and solutions
for, translating research into policy and practice via evidence use. The guide is outlined in Box 4.2.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 25
4.2 Strategies for the administration of data collection instruments
In a low income setting, it is a lot easier to obtain information from large number of staff of the
ministry by inviting them together to a meeting/workshop. During this meeting the focus group
discussion (FGD) can be held. Focus groups can be constituted usually between 7-12 persons per
group and the FGD can last up to 45 minutes. Copies of the questionnaire can also be sent to other
participants and organizations who cannot be invited to a meeting. Effort should be made for a follow-
up to retrieve completed questionnaires. Selected individuals (particularly heads of health
organizations) should be interviewed (between 35-60 minutes) using a tape recorder and an interview
guide (Box 4.2).
4.3 Analysis of data collected
The data collected via the questionnaire can be analysed using the methods developed at McMaster
University Canada by Johnson and Lavis (2009). The recorded key informant interviews were
transcribed. All the responses from the focus group discussion should be noted. The transcribed
interviews and responses from the focus group discussions can be analyzed based on Giorgi's
phenomenological approach (Giorgi 1985), which is further described by Albert et al. (2007). The
analysis follows the following steps: (i) going over all the textual data to gain an overall
impression,(ii) identifying all comments that appeared significant to the research, extracting these
meaning units, (iii) independent abstracting of the meaning units, followed by discussion and
consensus; (iv) independent categorization and summarization of abstractions into challenges of
Health Policy and System Research evidence use in policymaking and the solutions as perceived by
policy-makers, followed by discussion and consensus; and finally (v) returning to the extracted text to
ensure a good fit with the final list of challenges and solutions.
4.4 Interventional strategies to address the capacity constraints as informed by the Preparatory
and Baseline Data Collection Phases
The Project Preparatory and Baseline Data Collection Phases involve an in-depth assessment of
institutional and individual staff capacity for evidence-based policy making. The findings generated
from the preparatory (first) phase and the base line data collection (second) phase should enable the
Project Team to identify the specific areas within the initial proposed interventions to focus on in
order to achieve the following: (i). Individual capacity strengthening; (ii). Strengthening skills in
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 26
developing research initiatives; (iii). Building enabling environments and capacity for HPSR; and (iv).
The institutionalization of a stronger incentive for the use of evidence. Table 4.1 shows an example of
proposed intervention and activities based on the outcome of the study in Nigeria.
Box 4.1 Questionnaire for baseline data collection
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT QUESTIONNAIRE
The following questions are designed to help us determine your capacity for evidence-based policy making to enable
us channel appropriate interventional strategies to enhance your capacity on the use of research evidence for policy
making in the health sector.
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose only.
1. BIODATA
- -
(i
2. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
(vi). Membership of health-
3. KNOWLEDGE & APPLICATION OF NFORMATION/COMMUNICATION
TECHNOLOGY
(ix). Do you have an E-
(x). If yes, please state……………………………………………………………………………
(xi). How often do you use your E-
4. GEOGRAPHICAL FOCUS
(ii). How would you assess your organization‟s capacity/competence to cover your geographical area of operation?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 27
(a). Manpower
(b). Logistics
(c). Funding
(d). Facilit
(iii). How would you describe the accessibility (patronage) of the services provided by your organization
5. HEALTH SYSTEM RESEARCH & EVIDENCE
(A). ACQUISITION OF RESEARCH EVIDENCE
(a). Individual research skill Hints Ratings
(i). How adequate is your present
knowledge about initiating/conduction
research?
Identification of research problems;
construction of research questions;
designing of research methodology;
writing of research
proposals/protocols; analysis &
interpretation of research results;
writing of research reports.
(ii). How effective is your ability to
access and use existing research
evidence?
Journals, internet & library assess;
non journal reports eg. newspapers,
textbooks, reports from national &
international agencies, databases,
websites; works from researchers &
peers.
(b). Institutional/organizational
incentive for research Hints Ratings
(i). How would you describe the
capacity of your organization to carry
out research?
Existence of research programmes,
departments, officers & facilities; any
reputation on specialized areas of
research; research outputs; regularity
of research activities.
(ii). How would you describe the
capacity of your organization to
source for research evidence?
Existence of databases; relationship
with research institutions;
collaboration with researchers &
experts; commissioning of research
projects.
(iii). What is the level of research
incentives available in your
organization?
Availability of library, internet
facilities; availability of research
grants; award of honours/promotions;
in-service training; stipends, bonuses
& per-diem for research works;
provision of research facilities;
sponsorship to conferences/workshops;
institutional subscription of research
materials (periodicals eg. journals),
databases, websites.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 28
(B). ASSESSING THE VALIDITY, QUALITY & APPLICABILITY OF RESEARCH EVIDENCE
(a). Individual research skill Hints Ratings
(i). How would you describe your
capacity to assess the authenticity,
validity, reliability & high quality of
research evidence?
1. The skill to evaluate & appropriate
the quality of research methodology.
2. The skill to evaluate the reliability
of specific research evidence and to
compare research methods and
results.
(ii). How would you describe your
capacity to assess the relevance and
applicability of research evidence?
1. The skill to identify relevant
similarities and differences between
research evidence.
2. The skill to evaluate the
differences in the research evidences
in the context of your organization.
(b). Institutional/organizational
incentive for research Hints Ratings
(i). Does your organization have
necessary incentives for assessment
of the validity, quality and
applicability of research evidence?
Use of consultants; application of
computer software, statistical package;
well-equipped laboratory; existence of
quality control units; promotion of
ethical standards/practices.
(ii). Does your organization have
incentives to encourage the
application of research evidence?
Availability of research evidence
implementation committee; availability
of administrative process for
accepting/implementing research
evidence.
(C). ADAPTING THE FORMAT OF THE RESEARCHER RESULTS TO PROVIDE INFORMATION USEFUL
TO DECISION MAKERS
(a). Individual research skill Hints Ratings
(i). How would you describe your
ability to summarize results in a user-
friendly way.
1. Present research results concisely in
accessible language
y adequate
2. Synthesize in one document
relevant research as well as
information and analysis from other
sources.
3. Link the research results to key
issues and provide recommendations.
(ii). How would you describe your Use of charts, tables, graphs,
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 29
ability to present results of research to
decision makers.
pictogram, bullet/power point
presentations, etc.
(b). Institutional/organizational
incentive for research Hints Ratings
(i). Does your organization have
incentives to encourage the provision
of research evidence to decision
makers?
Summarized and easy-to-use research
evidence is routinely brought to the
attention of relevant decision makers
(such as through regular meetings or
reports; or participation by
researchers/analysts in management
meetings to present/discuss evidence).
(D). APPLICATION OF EVIDENCE IN DECISION MAKING
(a). Institutional/organizational
incentive for research Hints Ratings
(i). Do we lead by
example and
show we value
research use?
1. Using research is a priority: our
organisation has committed
sufficient people, time, training and
budgets to access, appraise,
adapt and apply research in making
decisions.
2. Our organisation‟s job descriptions
and performance incentives include
enough focus on activities which
encourage using research.
equate
3. Both management and front-line
staff support and participate in
frequent forum where staff and
invitees present and discuss
research evidence related to the
organisation‟s main goals.
ly adequate
4. Management has clearly
communicated corporate strategy and
priority areas for improvement, so that
people creating or monitoring research
evidence know what is needed.
5. Our organisation has effective
communication channels so that
priorities, evidence and ideas are
exchanged across divisions, as
well as between management and front
lines.
adequate
6. Our corporate culture is to value
and reward flexibility, change, and
continuous quality improvement, and
we provide adequate resources at all
levels to support change.
(ii). Do our decision making
processes have a place for research?
1. When we make major decisions, we
usually allow enough time to identify
researchable questions and create/
obtain, analyse and consider research
results and other evidence.
quate
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 30
2. Our management team has enough
expertise to evaluate the
feasibility of each option, including
potential impact across the
organisation as well as on its clients,
partners and other
stakeholders.
3. When staff develop or identify high
quality and relevant research,
decision makers will usually give
formal consideration to any
resulting recommendations.
4. Staff and appropriate stakeholders
know when and how major decisions
will be made, how and when they can
contribute evidence and how that
information will be used
e
5. The staff who have provided
evidence and analysis usually
participate in the discussion before a
decision is made and, when possible,
so do relevant non-staff researchers
irly adequate
6. When a decision is made, feedback
to staff and appropriate
stakeholders includes a rationale for
the decision, and review of
how the available evidence influenced
the choices made
airly adequate
6. KEY AREAS OF ACTIVITIES
(A). HEALTH ADMINISTRATION
Specific activities Hints Ratings
(i). Planning. 1. What is the capacity of your
organization to use evidence when
planning?
2. What is the level of external input
in your evidence-based planning
process?
(ii). Performance measurement. 1. What is the capacity of your
organization to make consistent
evidence-based policies?
2. How effective is the performance of
your organization‟s evidence-based
policies?
(iii). Bench marking/best practice. 1. What is the capacity of your
organization to adopt/use
international/national health policy
standard/guidelines?
quate
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 31
2. What is the accessibility of
international/national health policy
standard/guidelines to your
organization
(B). SERVICE DELIVERY
Specific activities Hints Ratings
(i). Service organization and
management.
1. How functional is your
organization‟s management structure
in the use of evidence in service
delivery?
rossly inadequate
2. How does your service delivery
conform with other evidence-based
health practices?
(ii). Access to health services. 1. How adequate are your
organization‟s health
facilities/services in evidence-based
service delivery?
2. How accessible are the health
facilities/services rendered by your
organization?
(iii). Quality of health services/safety. 1. How efficient is your organization‟s
manpower in evidence-based health
service delivery?
2. To what extent do international
quality and safety guidelines influence
health service delivery by your
organization?
(C). INFORMATION/EVIDENCE
Specific activities Hints Ratings
Generation/Strategic use of
information.
1. How do you describe your
organization‟s Information
Technology capacity to source,
generate and manage health related
information?
2. To what extent does your
organization have access to key
sources of health-related information
(media, research findings, educational
materials)?
equate
(D). MEDICAL PRODUCTS AND TECHNOLOGIES
Specific activities Hints Ratings
Access to essential medical
products/technologies.
1. How do you describe the capacity
of your organization to acquire and
use essential medical products and
technologies ?
2. How scientifically sound,
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 32
efficacious and cost effective are
medical products and technologies
used by your organization?
(E). HEALTH WORKFORCE
Specific activities Hints Ratings
(i). Employment and disengagement
of workforce.
1. How do you assess the personal
management practice in your
organization?
2. How do you describe the
availability of qualified health
workers in your organization?
(ii). Training, remuneration and
performance.
1. How do you rate the effectiveness
of your organization‟s policy on
training, remuneration and
performance of health workers?
2. How available and accessible are
training programmes and external
incentives to staff?
(F). HEALTH FINANCING
Specific activities Hints Ratings
(i). Health system funding. 1. How do you describe the
affordability of the health services
rendered by your organization?
adequate
2. How adequate is your
organization‟s access to key sources
of health funding?
(ii). Resource management. 1. How do you describe the
budgetary, procurement, accounting
and auditing practice in your
organization?
2. How does external funding
influence your organization‟s policy
on resource allocation and utilization?
(G). LEADERSHIP AND GOVERNANCE
Specific activities Hints Ratings
(i). Effective oversight and control. 1. How does your organization adhere
to international and natural regulations
and control mechanisms for health
system management?
2. To what extent do the activities of
regulatory agenci s and legislators
influence leadership and governance
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 33
in your organization?
(ii). Attention to health design issues. 1. How do you assess your
organization‟s effort to initiate an
update health system framework?
2. How does global practice influence
the process of development and
adoption of health system framework
in your organization?
te
Box 4.2 Focus group discussion/key informant interview guide
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT KEY INFORMANT INTERVIEW/FOCUS DROUP DISUSSION GUIDE
PREAMBLE -Introduction of the subject matter by interviewer/moderator of discussion
-Collection of information on the biodata, designation attribute & participant‟s organization
FOCUS A:
Capacity constraints and challenges which impede the development of Health Policy and Systems Research
evidence use.
Questions:
1. What are your specific capacity constraints which impede the development of Health Policy and Systems
Research evidence use?
2. What are your organization‟s major challenges in the development of Health Policy and Systems Research
evidence use?
FOCUS B:
Identification of potential strategies and solutions that would address capacity constraints and challenges of Health
Policy and Systems Research evidence use.
Questions:
1. What do you think are the possible solutions to the identified capacity constraints and challenges on Health Policy
and Systems Research evidence use?
2. What strategies do you think your organization can adopt in the Health Policy and Systems Research evidence use
capacity constraints and challenges?
FOCUS C:
Identification of critical gaps in Health Policy and Systems Research evidence use with a focus on improving public
health.
Questions:
1. What do you think are the critical gaps in Health Policy and Systems Research evidence use?
2. How do you think these factors (gaps) affect evidence-based health care delivery?
FOCUS D:
Identification of barriers to and solution for translating research into policy and practice via evidence use.
Questions:
1. What do you think are the barriers to the use of research evidence in policy making process and practice?
2. What do you think can be done to facilitate the process of translating research evidence into policy and practice?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 34
Table 4.1 Proposed intervention and activities for strengthening capacity for evidence-based
health policymaking based on the outcome of the preparatory and baseline data collection
phases in the Nigeria study. Interventions proposed Specific areas of focus identified
from phases 1&2
Activities in line with proposed
intervention
Remarks
1. Individual capacity
strengthening
(a). Improving information and
communication technology
(computer) skill of participants.
Note: Most participants lacked
adequate ICT(computer) skill, a
very vital requirement for access
to information necessary for
policymaking.
Training on Computer appreciation;
Computer application (use and
application of Microsoft Office
Word; Microsoft Office
PowerPoint); Use of the Internet.
To be
accomplished
during the
first training
workshop.
(b). Improving capacity for
acquisition, assessment,
adaptation & application of
research evidence.
Note: Most participants lacked
adequate knowledge of what
evidence is, how to acquire it,
assess it, adapt it and apply it, in
the policymaking process.
Although most of the capacity
ratings ranged from inadequate to
fairly adequate, our goal is to
achieve a very adequate capacity
for participants.
Training on what evidence is; types
& sources of evidence; how to
identify most relevant evidence;
constructing search protocol;
developing search; gathering the
evidence; conducting searches and
documenting the process; selecting
relevant evidence; assessing the
quality of the evidence; extracting,
synthesizing and presenting
evidence; deriving evidence
statements; and assessing
applicability.
Training to, and also in identifying,
interpreting and applying evidence
to decision making and tools to
assess the legitimacy of the filtering
and amplification function of
diverse organizations.
To be
accomplished
during the
second
training
workshop.
(c). Enhancing partnership links,
and participation in networks.
Note: Most participants do not
know about partnerships and
networks and how these can be
useful to them in the
policymaking. Response from
focus group discussion and
informant interview confirmed
this.
Introducing health-related
partnership & networks; their
importance & benefits; how to
establish partnership links and
participate in networks. Establishing
good working relationship between
policy makers and researchers.
To be
accomplished
during the
second
training
workshop.
2. Strengthening skills in
developing research
initiatives
Enhancing the capacity to
conceptualize, formalize,
implement and evaluate the
research needed for better
understanding and improved
performance of the country‟s
health system.
Note: Apart from participants
with doctorate degree who had
very adequate capacity, most of
Training on research methods and
the usefulness of research findings
in the policy process, and how to
facilitate commissioning of
appropriate research or syntheses
and the interpretation of systematic
reviews.
Training to develop research
capacity of individuals in relation to
HSPR evidence use in policy
To be
accomplished
during the
third training
workshop.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 35
the other participants had capacity
ratings ranging from grossly
inadequate to fairly adequate, in
terms of the ability for research
initiative (conceptualize,
formalize, implement and
evaluate research).
Response from focus group
discussion and informant
interview confirmed this.
making; knowledge translation
programs and processes.
3. Building enabling
environments and
capacity for HPSR
Enhancing capacity to own and
drive the agenda for strengthening
health systems. Enhancing
organizational capacity to take
strong lead in capacity building
for all functions of the health
system.
Note: Results of the baseline data
collection indicate ratings for
key areas of health system
activities (especially health
financing, health workforce &
information/evidence) as
inadequate. Response from focus
group discussion and informant
interview confirmed this.
Training on strategies that would
enable participating organizations to
establish capacity development
mechanism for the health system
(including stewardship [leadership
&governance], communications,
financing, resource management,
informatics, service delivery and
research).
Training on strategies that would
facilitate the development of
organizational capacity for the
generation, dissemination and use of
health policy and systems research
knowledge among researchers,
policy-makers and other
stakeholders.
To be
accomplished
during the
fourth
training
workshops.
(4). The
institutionalization of a
stronger incentive for the
use of evidence.
Enhancing organizational
capacity for the institution of
stronger incentive for research
and use of evidence in
policymaking.
Note: Results of the baseline data
collection indicate ratings for
organizational capacity for
research incentive mostly ranging
from grossly inadequate to
inadequate. Response from focus
group discussion and informant
interview confirmed this.
A leadership training workshop to
be conducted. Exclusively for
policy makers at helm of affairs of
health ministry such as
commissioners, permanent
secretaries, political/legal advisers
on health related matters;
program/project managers under the
health ministry and top officers of
NGOs. Focus on strengthening
institutional incentives to employ
evidence in policymaking.
To be
accomplished
during the
fifth training
workshops.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 36
CHAPTER 5
OVERVIEW OF PROCEDURES FOR THE INTERVENTION/IMPLEMENTATION PHASE
5.1 Basis for interventional strategies
Findings from the preparatory and the baseline data collection phases inform the categorization of the
interventional strategies into two broad areas. Training workshops and special mentoring programme
consist the major components of the intervention phase. The description of the intervention activities
under the two broad areas is as follows:
5.1.1 Strategies to strengthen individual staff skills to enhance capacity to use evidence in policy-
making: This should be executed through the provision of training and mentoring in use of
acquisition, assessment, adaptation and application of research evidence, e.g. training in the
commissioning of research studies and the interpretation of systematic reviews. The training programs
should consist of the development of research capacity of individuals in relation to HSPR evidence use
in policy making; knowledge translation programs and processes; comparative HPSR studies with
emphasis on evidence use, especially in the areas of research utilization, social health insurance, health
systems research on emerging infectious diseases, health issues during crisis periods such as natural
disasters, and equity. Knowledge sharing and networking should be fostered through the internet, and
the feasibility of establishing an HPSR „observatory‟ for knowledge syntheses and evidence-informed
policy recommendations should be explored.
Emphasis should be placed on the development of key HPSR objectives including priority
setting, knowledge generation, advocacy/communications, knowledge translation and capacity
building. Training should also focus on the health systems operations with emphasis on the building
blocks of health systems including service delivery; information and evidence; medical products and
technologies; health financing; health workforce; and leadership/governance (stewardship). Policy
briefs should be produced on these health systems building blocks, followed by policy dialogues.
Other key research areas affecting the health systems such as health sector reforms, human resource
development, equity, and poverty reduction should also be explored. A fellowship/mentoring scheme
that would enable the participating policy makers to take leave to spend time in a research
environment should be developed to commence after all interventional workshops. Arrangements
should be made participating policy makers to be mentored by an expert under whose supervision they
would be required to undertake a research project on HPSR as it pertains to evidence use in policy
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 37
making. The mentor in this case should be a senior researcher from the university (A Professor, an
Associate Professor or a Senior Lecturer).
5.1.2 Strategies to strengthen institutional incentives to employ evidence: This can be achieved
through the initiation of the process that would enhance the creation of stronger incentives for
evidence use (e.g. through integrating this dimension into recruitment policies, performance
assessments, and staff appraisals) in the participating organizations; integration of modules on use of
evidence into leadership training courses for policy makers; and recognition awards - providing
financial or in-kind prizes to policy makers that have shown outstanding performance in applying
evidence to policy. The components that can be considered should include: identification and building
the right team; identification and initiating relevant partnerships and linkages; promoting commitment
from key stakeholders; setting priorities on content topics to be tackled; planning of realistic
milestones for evaluation; and identification of local financial and in-kind support.
5.2 Training workshop pattern and methods for analysis of data collected
The major aim of the workshops should be principally to train policy makers and other stakeholders to
enhance their capacity for evidence-based health policymaking. Formal letters of invitation should be
sent to target participants at least three weeks to the date of each workshop, followed by an SMS
reminder via mobile phones a day or two before each workshop. At the workshops each participant
should be registered and given a participant identification number. Practical approaches to
policymaking process should be emphasized in each of the training activity. A pre-workshop
assessment questionnaire should be administered prior to actual training to assess the level of
knowledge, attitude, practices and perception of the participants on the specific topics to be covered
within the theme of each workshop. Space should also be provided for participants general comments
in the questionnaire. After the administration of the pre-workshop questionnaire the training should
commence and should be facilitated by resource persons who could be senior researchers drawn from
the academic institutions and senior policymakers from the health ministry.
All teaching sessions should be done using power-point presentation and handouts on each
topic and should be produced and distributed to all participants. It should be made mandatory for all
lectures to be delivered in simplified, practical and easily comprehensible patterns, with little or no
emphasis on complex mathematical or scientific computations/models for the benefit of non-
specialists who usually constitute the majority of the participants. Key informant interviews, focus
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 38
group discussions, dialogues, question/answer sessions, role play, demonstration, simulations, group
work and presentations from participants are some methods that can be used during the workshops.
Room should be created for personal/private interactions among participants and with
Facilitators/Resource persons. At the end each workshop, a post-workshop assessment questionnaire
should be administered to the participants to evaluate the impact of the workshop and also to obtain
the impression of the participants about the training and the materials used for the training.
The data collected via the questionnaire can be analysed using the methods developed at
McMaster University Canada by Johnson and Lavis (2009). The analysis is based on mean rating
(MNR), median rating (MDR) and range (R). For instance the figures represent Likert rating scale of
1-4 points, where 1point=grossly inadequate; 2 points=inadequate; 3 points=fairly adequate; and 4
points=very adequate. In terms of analysis, values ranging from 1.00-2.49 points are considered low,
whereas values ranging from 2.50-4.00 points considered high.
All recorded interviews should be transcribed. Also all the responses from the focus group
discussions and group works should be noted. The transcribed interviews and responses from the focus
group discussions and group works should be analyzed based on Giorgi's phenomenological approach
(Giorgi 1985), which is further described by Albert et al. (2007). The analysis follows the following
steps: (i) going over all the textual data to gain an overall impression,(ii) identifying all comments that
appeared significant to the research, extracting these meaning units, (iii) independent abstracting of the
meaning units, followed by discussion and consensus; (iv) independent categorization and
summarization of abstractions into challenges of Health Policy and System Research evidence use in
policymaking and the solutions as perceived by policy-makers, followed by discussion and consensus;
and finally (v) returning to the extracted text to ensure a good fit with the final list of challenges and
solutions.
In the Nigeria study a total of six training workshops were organized. The following chapters
provide detailed descriptions of the methods that can be used in the training in the workshops.
5.3 Development and production of policy briefs
The policy briefs should be prepared using the techniques outlined by Lavis et al. (2009a); Jones and
Walsh (2008) and GHAIN (2008). The briefs should be subjected by both internal and external review
to ensure it meets up with the minimum acceptable standard. The policy briefs should focus on the
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 39
specific areas of the health systems where interventions are mostly required. In the Nigeria Study the
following are the titles of the policy briefs produced are as follows:
1. Strengthening the generation/strategic use of health information and evidence for health
systems operations and policymaking in Nigeria.
2. Ensuring availability and equitable access to essential medical products, equipment and
technologies in Nigeria.
3. Improving access, availability and efficiency in the use of health financial resources.
4. Strengthening the organization, access and quality of health service delivery in Nigeria.
5. Enhancing the distribution, efficiency and performance of health workforce in Nigeria.
6. Ensuring effective leadership and governance to enhance the performance of the health
systems in Nigeria.
Similar titles can be adopted depending on the area of the health systems where intervention is
required.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 40
CHAPTER 6
TRAINING WORKSHOP ON CAPACITY ENHANCEMENT IN INFORMATION AND
COMMUNICATION TECHNOLOGY FOR POLICYMAKING
6.1 Methods for Information and Communication Training Workshop
An Information and Communication Technology (ICT) training workshop can be organized for
participating policy makers at an ICT Centre with facilities to accommodate such training workshop.
This could be the first in the series of the training workshops for the interventional phase of the project
as it was the case in the Nigeria study. In Nigeria, the training was organized at the ICT Centre of
Ebonyi State University Abakaliki. The theme of the workshop in the Nigeria study was: “Enhancing
the capacity for evidence-based health policy making in Nigeria through information and
communication technology.” Similar theme can be adopted depending on the local needs and the
outcome of the baseline data collection. The workshop can commence with the administration of a
pre-workshop pretested structured questionnaire (see Box 6.1). Using the questionnaire, an assessment
can be conducted on participants‟ Knowledge and application of information/communication
technology particularly in the following areas: (i). Computer appreciation; (ii). Microsoft word; (iii).
Power point; and (iv). Internet use. After the questionnaire administration, the training should
commence and should be focused on the development of the capacity of the policymakers on
computer appreciation and application.
A training manual should be developed and made available to participants. The training
manual should cover topics in computer appreciation; use of Microsoft word; use of power point and
use of internet. Participants should also be taught the importance of ICT to the health sector
(improving the functioning of health care systems; improving the delivery of health care; and
improving communication about health), and how to gain access to health information from the
internet.
This aspect of the training should focus on the following:
(a). Benefits of engaging ICT in running the health sector;
(b). The search protocol for information;
(c). Identification of and search strategies of a wide range of electronic resources, in addition to
the traditional scientific and clinical databases;
(d). Information audit to accompany search strategies.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 41
At the end of the training, a post-workshop questionnaire should be administered (See Box 6.2) to
assess the impact of the workshop and participants impression about the workshop.
Box 6.1 Pre-ICT training workshop questionnaire
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
PRE-ICT TRAINING QUESTIONNAIRE
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose
only.
1. BIODATA
- -44 yea
2. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
3. KNOWLEDGE & APPLICATION OF INFORMATION/COMMUNICATION TECHNOLOGY
A. COMPUTER APPRECIATION
(i). Rate your ability to identify the different basic components of the computer.
(ii). What is your level of knowledge on the functions of basic computer components?
y adequate
(iii). What is your ability to differentiate between the input and output computer devices?
(iv). How would you describe your ability to set up a computer workplace?
(v). How would you describe your ability to boot and shut down a computer?
(vi). Rate your present ability to identify different computer storage devices.
(vii). What is your knowledge of basic care and safety procedures in computer handling/usage?
B. MICROSOFT(MS) WORD
(i). Rate your ability to link up to MS word page from a computer window
(ii). What is your level of knowledge on the identification and functions of the menu/standard tool
bars in an MS word page?
(iii). How would you describe your ability to create and save MS word files and folders?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 42
(iv). How would you describe your ability to access/assess existing files and folders in MS word?
(v). How would you describe your ability to type and format texts/paragraphs in MS word?
C. POWER POINT
(i). How would rate your knowledge of the following power point functions?
(a). Opening and inserting slides:
(b). Bullet presentation:
(c). Copying/pasting from MS word to power point:
Very adequate
(d).Use of graphics/charts/tables in power point:
(e). Use of animations in power point:
(ii). Rate your ability to use power point in conference/workshop presentations and meetings
quate
(iii). How would you describe your present ability to identify and apply basic devices for power point
D. INTERNET USE
(i). How would you describe your knowledge on the importance and benefits of the internet?
(ii). Do you now have an e-
(iii). How would you describe your ability to create and use e-mail address
(iv). How is your ability to locate information on the internet?
(v). How is your ability to locate and access websites of different organizations?
(vi). How is your level of knowledge on the types and use of major search engines?
(vii). How is your ability to locate and access relevant databases?
Comments:…………………………………………………………………………………………………………
……………………………………………………………………………………………………………………
…………………………………………………………………………………………………………
PARTICIPANT No…………………
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 43
Box 6.2 Post-ICT training workshop questionnaire
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
POST-ICT TRAINING QUESTIONNAIRE
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose
only.
1. BIODATA
(i)
- -
2. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
3. KNOWLEDGE & APPLICATION OF INFORMATION/COMMUNICATION TECHNOLOGY
A. GENERAL QUESTIONS ON ICT TRAINING
(iii). If yes when? (month/year)……………………
(iv). Is the present training t
(v). How adequate was this training in relation to your job description?
(vi). How would you rate the quality of the training?
(vii). How would you rate the impact of the training?
B. COMPUTER APPRECIATION
(i). Rate your present ability to identify the different basic components of the computer.
(ii). What is your present level of knowledge on the functions of basic computer components?
(iii). What is your present ability to differentiate between the input and output computer devices?
(iv). How would you describe your present ability to set up a computer workplace?
(v). How would you describe your present ability to boot and shut down a computer?
(vi). Rate your present ability to identify different computer storage devices.
(vii). What is your present knowledge of basic care and safety procedures in computer handling/usage?
C. MICROSOFT(MS) WORD
(i). Rate your present ability to link up to MS word page from a computer window
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 44
(ii). What is your present level of knowledge on the identification and functions of the menu/standard tool
bars in an MS word page?
(iii). How would you describe your present ability to create and save MS word files and folders?
(iv). How would you describe your present ability to access/assess existing files and folders in MS word?
(v). How would you describe your present ability to type and format texts/paragraphs in MS word?
D. POWER POINT
(i). How would rate your present knowledge of the following power point functions?
(a). Opening and inserting slides:
(b). Bullet presentation:
(c). Copying/pasting from MS word to power point:
(d).Use of graphics/charts/tables in power point:
(e). Use of animations in power point:
(ii). Rate your present ability to use power point in conference/workshop presentations and meetings
(iii). How would you describe your present ability to identify and apply basic devices for power point
E. INTERNET USE
(i). How would you describe your present knowledge on the importance and benefits of the internet?
(ii). Do you now have an e-
(iii). How would you describe your present ability to create and use e-mail address
(iv). How is your present ability to locate information on the internet?
(v). How is your present ability to locate and access websites of different organizations?
(vi). How is your present level of knowledge on the types and use of major search engines?
(vii). How is your present ability to locate and access relevant databases?
Comments:…………………………………………………………………………………………………………
……………………………………………………………………………………………………………………
…………………………………………………………………………………………………............................
PARTICIPANT No…………………
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 45
CHAPTER 7
TRAINING WORKSHOP ON CAPACITY ENHANCEMENT FOR ACQUISITION,
ASSESSMENT, ADAPTATION & APPLICATION OF RESEARCH EVIDENCE FOR
POLICY MAKING AND PARTNERSHIP LINKS
7.1 Methods for the Training Workshop
This training workshop should be the second in the series of training workshops to be organized for
capacity enhancement. In the Nigeria study the theme of the workshop was: “Improving capacity for
acquisition, assessment, adaptation & application of research evidence for policy making &
enhancing partnership links.” Participants comprised of health policy makers, stakeholders and
researchers and were drawn from the health ministry at the State and local government levels, non-
governmental organizations, and civil society groups such as medical and health workers associations
of physicians, nurses, pharmacists, laboratory scientists, and hospital administrators. Also represented
were heads of health divisions of the uniform services such as the police, army, prisons, and road
safety commission.
Similar theme and participant selection can be adopted depending on the local needs and the outcome
of the baseline data collection. The project mentors should be invited to participate in this workshop
because of the role they will play in building partnership link which is also a focus of this workshop.
The workshop should commence with the administration of a pre-workshop pretested structured
questionnaire (see Box 7.1). At the end of the training sessions a post-workshop pretested structured
questionnaire (see Box 7.2) should be administered. Prior to the administration of the post-workshop
questionnaire a focus group discussion (FGD) can be conducted on partnership links/bridging the gap
between policy makers and researchers (see Box 7.3 for FGD guide). The lecture sessions should
focus on the following topic areas:
7.1.1 Introduction to health policy and systems research/ building evidence-informed policy
environments
(i). Definition and scope of health policy and systems research (HPSR); HPSR aim and focus;
(ii). The building block health systems (health service delivery, information and evidence, medical
products and technology, health financing, health workforce, leadership and governance).
(iii). Building evidence-informed policy environments, understanding policy contexts, the role of
interests, ideology and values.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 46
(iv). Research–policy interface and the policy-making processes
(v). Stage of the policy process: Agenda setting; Policy formulation; Implementation and Evaluation
(vi). Policy making and use of evidence
(vii). Types of evidence: systematic reviews; single research studies; pilot studies and case studies;
information available on the internet; and experts‟ opinion.
7.1.2 Acquisition, assessment, adaptation & application of evidence in health policy making
(i). Definition of evidence?
(ii). Evidence in policymaking context: The core issues and Types of evidence
(iii) Scientific evidence (Context-free scientific evidence and Context-sensitive scientific evidence):
(iv). Colloquial evidence (Expert testimony and Consultation with stakeholders):
(v). Identifying evidence for policy making and issues to consider when searching for public health
literature
(vi). Search Procedure (Constructing the search protocol; Developing the search strategy; Gathering
the evidence, conducting searches and documenting the process):
(vii). Additional searches: „Snowballing‟ to find citations, A search of the grey literature,
Journal/publication hand-searches especially use of libraries and other information databases,
Making contact with experts and stakeholders.
(viii). Documenting the search process & Submission of evidence for policymaking
ongoing research; studies that have only been published as abstracts; data on adverse effects
relevant to the interventions being studied; economic models; studies of public or professional
views and experiences; confidentiality information.
7.1.3 Building effective linkage, partnership & exchange between health policy-makers and
researchers
(i). The policy environment
(ii). Actors in Health Policy Making
(iii). Need for Partnership in the Health Policy Making Process
(iv). The „Two Communities‟ Model of Researchers and Policy-Makers (University Researchers and
Government Officials)
(v). The Policy Making Process: The policy process and how networks may affect it
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 47
Stage of the policy process (Analysis of the Policy Making Process; Setting the Agenda;
Improving the characteristics of research results)
(v). Adopting the Communication Process and Products to Stakeholders‟ needs
(vi). Building Evidence into the Policy Making Process (Engineering view, Enlightenment view,
Elective Affinity view)
(vii). Models of Researchers versus Policy Makers Linkage
(viii). Difficulties in Charting Effective Researcher/Policy-Maker Linkage
7.1.4 Capacity Development and Enhancement for Evidence-Informed Health Policy-Making
(i). Definition of Capacity
(ii). How to Identify Lack of Capacity
(iii). Focus of Capacity Building
(iv). Steps in Building capacity for Evidence-Based Health Policy Research
(v). The capacity Pyramid
(vi). Elements of Capacity Building Capacity development
(vii). The link between the Need for capacity Development and Evidence Informed Health Policy
Making
(viii). Elements of Evidence-Informed National Policy
7.2 Participation of project mentors
During the workshop the project mentors should complete a Mentor‟s Research-Policy Link
questionnaire (Box 7.4), which is designed to assess the mentors‟ involvement into research/policy
process. Each project mentor should serve as chair of each focus group discussion. The discussion
lasted up to 40minutes and the senior policymaker from each group presented group‟s resolution to all
participants. The following were the components of the discussion: (i). The need for
partnership/collaboration between researchers and policymakers; (ii). Types of partnership; (iii).
Causes of existing gaps between researchers and policymakers in health policymaking; (iv).
Implications of the gaps between researchers and policymakers in health policymaking; and (v).
Strategies for effective partnership/collaboration between researchers and policymakers
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 48
Box7.1 Pre-Workshop Questionnaire: Improving Capacity for Acquisition, Assessment,
Adaptation & Application of Research Evidence for Policy Making HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
PRE WORKSHOP QUESTIONNAIRE
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose only.
A. BIODATA
(i). Gende
- -
B. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
C. POLICY MAKING PROCESS
1. To what extent are you involved in the policymaking process in your organization?
2. What is your level of knowledge of the meaning of policy?
3. How would you rate your understanding of policy context?
4. What is your level of understanding of policy environment?
5. How would you rate your understanding of different types of policies?
6. What is the level of your knowledge about stakeholders‟ and various actors‟ involvement in policy making?
7. Rate your level of understanding of policy making process.
8. How is your level of understanding of the meaning of priority setting/policy agenda in policy making?
9. What is your level of understanding of the meaning of a policy brief?
10. Rate your level of understanding of what a policy dialogue is.
11. What is your level of understanding of the meaning of a policy draft?
12. What is the level of your knowledge on the role of researchers in policy making?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 49
D. CAPACITY FOR USE OF EVIDENCE
1. What is your level of understanding on what evidence is in policy making context
2. Rate your knowledge on the types of evidence that can be used for policy making?
3. What is your level of knowledge on the sources of evidence used for policy making?
4. How would you rate the effectiveness of your ability to source for evidence used in policy making?
5. How would you rate your capacity to identify/select relevant evidence for policy making?
6. How would you rate your ability to adapt (extract, synthesize, & present) evidence used for policy making?
7. How would you rate your ability to transform evidence into policy useable form?
8. How is your knowledge of filtration and amplification function (optimal use of relevant evidence) in policy
making?
Comments:………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………
……………………………………………………………………………………………
PARTICIPANT No…………………
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 50
Box 7.2 Pre-Workshop Questionnaire: Improving Capacity for Acquisition, Assessment,
Adaptation & Application of Research Evidence for Policy Making HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
POST WORKSHOP QUESTIONNAIRE
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose only.
A. BIODATA
- -
B. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
C. GENERAL QUESTIONS ON POLICY & EVIDENCE
(i). Are capacity enhancement/strengthening necessary for evidence based policy making in your work?
(ii). Have you had any training programme/workshop on capacity enhancement/strengthening for evidence
based policy m
(iii). If yes when? (month/year)……………………
(iv). Is the present training the first formal training you have had on capacity enhancement/strengthening for
(v). How adequate was this training in relation to your job description?
(vi). How would you rate the quality of the training?
(vii). How would you rate the impact of the training?
D. POLICY MAKING PROCESS
1. What is now your level of knowledge of the meaning of policy?
2. How would you rate your present understanding of policy context?
equate
3. What is your present level of understanding of policy environment?
4. How would you rate your present understanding of different types of policies?
5. What is the level of your present knowledge about stakeholders‟ and various actors‟ involvement in policy
making?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 51
6. Rate your present level of understanding of policy making process.
7. How is your present level of understanding of the meaning of priority setting/policy agenda in policy
making?
8. What is your present level of understanding of the meaning of a policy brief?
9. Rate your present level of understanding of what a policy dialogue is.
10. What is your present level of understanding of the meaning of a policy draft?
11. What is the present level of your knowledge on the role of researchers in policy making?
E. CAPACITY FOR USE OF EVIDENCE
1. What is your present level of understanding on what evidence is in policy making context?
dequate
2. Rate your present knowledge on the types of evidence that can be used for policy making?
3. What is your present level of knowledge on the sources of evidence used for policy making?
4. How would you rate the effectiveness of your ability to source for evidence used in policy making?
5. How would you rate your present capacity to identify/select relevant evidence for policy making?
6. How would you rate your present ability to adapt (extract, synthesize, & present) evidence used for policy
making?
7. How would you rate your present ability to transform evidence into policy useable form?
8. How is your present knowledge of filtration and amplification function (optimal use of relevant evidence)
in policy making?
Comments:………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………
……………………………………………………………………………………………
PARTICIPANT No…………………
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 52
Box 7.3 Focus group discussion guide on partnership links/bridging the gap between policy
makers and researchers
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
FOCUS GROUP DISCUSSION GUIDE
Topic: BRIDGING THE GAP BETWEEN HEALTH POLICY MAKERS & RESEARCHERS
1. Preamble: Introduction of the subject matter by moderator.
2. Discussion on the need for partnership/collaboration between researchers and policymakers.
3. Types of partnership (eg. formal/informal, use of joint committees of researchers/policymakers, use of
consultants/knowledge brokers)
4. Causes of existing gaps between researchers and policymakers in health policymaking.
5. Implications of the gaps between researchers and policymakers in health policymaking.
6. Challenges associated with partnership between researchers and policymakers.
7. Strategies for effective partnership/collaboration between researchers and policymakers.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 53
Box 7.4 Project Mentors’ questionnaire on assessment of research- policy link HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
MENTOR QUESTIONNAIRE
A. INSTITUTIONAL AFFILIATION
(i). Name of your institution……………………………………………………………..
(ii). Faculty/department……………………………………………………………………..
B. RESEARCH/POLICY LINK
1. To what extent is research relevant to health policy making in Nigeria
2. How do you rate the level of partnership presently existing between researchers and
health policy makers in Nigeria.
3. How frequent have your previous research outcomes being made available to policy
makers?
4. How adequate are the following factors in researchers/health policy makers partnership
in Nigeria?
(i). Time factor (in terms of time taken to initiate, execute & make available research
evidence ie. Information/data):
(ii). Resources:
(iii). Incentives:
5. Rate the level of influence of the following factors in researchers/health policy makers
partnership in Nigeria?
(i). Research methodology:
(ii). Politics:
(iii). Ownership/Intellectual property:
6. Have you held any policy making position outside the academic environment in the
pas
7. Have you participated in any policy making process outside the academic
8. If yes, to what extent did you use research evidence in policy making process?
9. How often are you being consulted by policy makers for research evidence?
10. Rate your preparedness to partner with policymakers in the policymaking process.
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 54
CHAPTER 8
TRAINING WORKSHOP ON CAPACITY ENHANCEMENT TO CONCEPTUALIZE,
FORMALIZE, IMPLEMENT AND EVALUATE THE RESEARCH NEEDED FOR BETTER
UNDERSTANDING AND IMPROVED PERFORMANCE OF THE HEALTH SYSTEMS
8.1 Methods for the Training Workshop
This workshop should be the third in the series of interventional workshops. Participants should
comprise of health policy makers, stakeholders and researchers and were drawn from the health
ministry at the State and local government levels, non-governmental organizations, and civil society
groups such as medical and health workers associations of physicians, nurses, pharmacists, laboratory
scientists, and hospital administrators. Also to be represented are heads of health divisions of the
uniform services such as the police, army, and prisons. In the Nigeria study the theme of the workshop
was “Enhancing the capacity to conceptualize, formalize, implement and evaluate research for better
understanding and improved performance of health systems.” A similar theme can be adopted
depending on the local needs and the outcome of the baseline data collection. The project mentors
should be invited to participate in this workshop.
The workshop should have four main objectives as follows:
(i). To enhance the capacity for the development of effective health systems policy;
(ii). To develop research capacity of individuals in relation to HSPR evidence use in policy making;
knowledge translation programs and processes;
(iii). To facilitate commissioning of appropriate research or syntheses and the interpretation of
systematic reviews;
(iv). To enhance the knowledge of participants on the value of policy briefs and policy dialogues in the
policy making process.
A policy dialogue should be held during this workshop, the detail of the procedure is described in
section 8.2. The workshop should commence with the administration of a pre-workshop pre-tested
structured questionnaire which focuses on the assessment of collaborative initiatives, health policy
briefs and health policy dialogues (see Box 8.1). This should be followed by lectures in the following
topic area:
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 55
8.1.1 Framework for Development of Effective Health Systems Policy
(i). Building blocks‟ of health systems (Service delivery; Information and evidence; Medical products
and technologies; Health workforce; Health financing; Leadership and governance)
(ii). Evidence-based health policy yields strong and effective health systems
(iii). Improving evidence-based policy-making for health (Strengthening demand from policy-makers;
Creating formal processes to facilitate dialogue; Improving packaging of evidence; Enhancing
technical capacity; Implementing incentives for researchers; Recognizing the role of informal
relationships)
(iv). Practical steps to developing evidence-based health policy
(v). Establishing collaborative health policy initiative
(vi). Principles and framework for collaborative initiative (setting the vision for collaboration; setting
out the guiding principles for collaboration; defining and setting goals and objectives; understanding
and defining roles and responsibilities; implementation requirements; evaluation)
(vii). Policy Design
(viii). Agenda/Priority-setting
(ix) Policy Briefs (what a policy brief is; preparation of policy briefs; key ingredients of effective
policy briefs; structure/outline of a policy brief; value of policy briefs)
(x). Policy Dialogue (what is policy dialogue is; importance of policy dialogue; organising and using
policy dialogues to support evidence-informed policymaking; characteristics of a policy dialogue)
(xi) Preparation of policy draft and submission to relevant authorities/government
(xii) Engaging the public in evidence-informed policymaking
(xiii) Monitoring and evaluation of policies
8.2 Policy dialogue on health systems strengthening
Strengthening the health systems is a major focus of this project. Therefore the policy briefs produced
in the six health systems building block areas should be subjected to a policy dialogue during this
workshop. Table 8.1 shows the titles of the policy briefs produced in the Nigeria study:
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 56
Table 8.1 Policy briefs produced for the Nigeria study Policy
Brief
No.
Health Systems
Building Block
Title of Policy Brief
1. Information &
Evidence
Strengthening the Generation/Strategic Use of Health Information and Evidence
for Health Systems Operations and Policymaking in Nigeria.
2. Medical Products
& Technology
Ensuring Availability and Equitable Access to Essential Medical Products,
Equipment and Technologies in Nigeria.
3. Health Financing Improving Access, Availability and Efficiency in the Use of Health Financial
Resources in Nigeria.
4. Service Delivery Strengthening the Organization, Access and Quality of Health Service Delivery in
Nigeria.
5. Health
Workforce
Enhancing the Distribution, Efficiency and Performance of Health Workforce in
Nigeria.
6. Leadership &
Governance
Ensuring Effective Leadership and Governance to Enhance the Performance of
the Health Systems in Nigeria.
As a part of the preparation for the policy dialogue, a letter of invitation and a policy brief that is most
relevant to each invited participant‟s designation/job description should be sent to the policymakers
two weeks before the workshop. This is to give ample time for participants to study the policy briefs
and prepare their inputs; and also to enable the Project Team to obtain participants‟ assessment of the
policy briefs. Participants should be advised in the accompanying letter of invitation to prepare inputs
on policy options in relation to the policy briefs they have received. During the workshop, before the
commencement of the dialogue, a policy brief evaluation questionnaire developed by Johnson and
Lavis (2009) (see Box 8.2) should be administered. This policy brief questionnaire is designed to
provide an in-depth evaluation of the participant‟s views about how the policy brief was produced and
designed; overall assessment of the policy brief; and views about what can be done better or
differently. After this participants should be grouped into six with each group undertaking a policy
dialogue on each of the six policy briefs. Participants should be assigned to the groups depending on
each person‟s designation/job description; for instance those at management positions including the
Permanent Secretary of the health ministry should be placed in the group that will undertake the
dialogue on the policy brief that centred on leadership and governance. Those in service delivery-
related positions eg., Director of Hospital Management, Director of Medical Services, Director of
Nursing services, etc., should undertake the dialogue on the policy brief focusing on health service
delivery. Each of the groups should consist of 7-12 individuals and should be chaired by a senior
researcher (Professor/Associate Professor/Senior Lecturer) who is a project mentor from the university
and a senior policymaker should serve as a deputy chair. A policy dialogue guideline should be
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 57
provided for participants (see Box 8.3). The guideline includes a description on how to evaluate the
policy brief documents in terms of content quality and relevance, as well as the policy issues
presented, the magnitude of the problems to be addressed and how actionable the policy options
recommended are. The dialogue should last up to one hour and each group should document its
resolutions and submit same to the Project Team. Thereafter a policy dialogue evaluation
questionnaire developed by Johnson and Lavis (2009) (see Box 8.4) should be administered. The
purpose of the questionnaire is to evaluate the participant‟s views about how the policy dialogue was
designed; overall assessment of the policy dialogue; views about what can be done better or
differently; and views about using research evidence more generally.
Box 8.1 Pre-workshop questionnaire: Enhancing the capacity to conceptualize, formalize,
implement and evaluate the research needed for better understanding and improved
performance of health systems HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
EVIDENCE-BASED POLICY MAKING CAPACITY ENHANCEMENT
PRE WORKSHOP QUESTIONNAIRE
Please answer the questions as correctly as possible. The questionnaire is purely for research purpose
only.
A. BIODATA
- -
B. OFFICIAL DESIGNATION ATTRIBUTES
(i). Name of your organization…………………………………………………………………………
(ii). Designation…………………………………………………………………………………..
(iii). Duration in designation………………………………………………………………………
C. GENERAL QUESTIONS ON COLLABORATIVE INITIATIVE
(i). How would you rate your understanding of collaborative initiative in health policymaking?
(ii). Are collaborative initiatives necessary for evidence based health policy making in your work?
(iii). How would you rate the level of involvement of your organization in collaborative initiative in
policymaking?
(iv). Have you ever been involved in any collaborative initiative in health policymaking before this project?
(v). If yes how many times in the last two years? ……………………
(v). Is the present project your first formal involvement in collaborative initiative for evidence based policy
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 58
(vi). How would you rate the collaborative initiative in this project since your involvement as participant?
D. GENERAL QUESTIONS ON POLICY BREIFS
(i). How would you rate your understanding of policy briefs in health policymaking?
(ii). Are policy briefs necessary for evidence based policy making in your work?
(iii). How would you rate the level of involvement of your organization in the production of policy briefs in
(iv). Have you had any training programme/workshop on policy briefs for evidence based policy making
(v). If yes how many times in the last two years? ……………………
(vii). If yes how many times in the last two years? ……………………
(viii). Is the present training workshop your first formal training on policy briefs for evidence based policy
E. GENERAL QUESTIONS ON POLICY DIALOGUES
(i). How would you rate your understanding of policy dialogues in health policymaking?
(ii). Are policy dialogues necessary for evidence based policy making in your work?
(iii). How would you rate the level of involvement of your organization in policy dialogues in
(iv). Have you had any training programme/workshop on policy dialogues for evidence based policy making
(v). If yes how many times in the last two years? ……………………
(vi). Have you ever participated in a policy dial
(vii). If yes how many times in the last two years? ……………………
(viii). Is the present training workshop your first formal training on policy dialogues for
Comments:……………………………………………………………………………………………....
PARTICIPANT No…………………
Thank you very much for your participation.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 59
Box 8.2 Policy brief evaluation questionnaire
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
Evaluation – Policy Brief
Please circle the number that corresponds to your answer and (if you wish) offer any suggestions about how the policy brief
can be improved.
Several questions make reference to "stakeholders." The term "stakeholders" includes: staff or members of civil society groups;
staff or members of health professional associations or groups; staff of donor agencies (e.g., European Community, Swedish
International Development Agency) or international organizations (e.g., World Health Organization); and staff of
pharmaceutical or other biotechnology companies.
Section A – Views about how the policy brief was produced and designed
1. The policy brief described the context for the issue being addressed. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? _______________________________________
____________________________________________________________________________________
____________________________________________________________________________________
2. The policy brief described different features of the problem, including (where possible) how it affects particular
groups. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? _______________________________________
____________________________________________________________________________________
____________________________________________________________________________________
3. The policy brief described three options for addressing the problem. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
4. The policy brief described what is known, based on synthesized research evidence, about each of the three options and
where there are gaps in what is known. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
5. The policy brief described key implementation considerations. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 60
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
6. The policy brief employed systematic and transparent methods to identify, select, and assess synthesized research
evidence. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
7. The policy brief took quality considerations into account when discussing the research evidence. How useful did you
find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
8. The policy brief took local applicability considerations into account when discussing the research evidence. How
helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
9. The policy brief took equity considerations into account when discussing the research evidence. How helpful did you
find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
10. The policy brief did not conclude with particular recommendations. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
11. The policy brief employed a graded-entry format (e.g., a list of key messages and a full report). How helpful did you
find this approach?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 61
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
12. The policy brief included a reference list for those who wanted to read more about a particular systematic review or
research study. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
13. The policy brief was subjected to a review by at least one policymaker, at least one stakeholder, and at least one
researcher (called a “merit” review process to distinguish it from “peer” review, which would typically only involve
researchers in the review). How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Section B – Overall assessment of the policy brief
14. The purpose of the policy brief was to present the available research evidence on a high-priority policy issue in order
to inform a policy dialogue where research evidence would be just one input to the discussion. How well did the
policy brief achieve its purpose?
Failed Moderately
failed
Slightly
failed
Neutral Slightly
achieved
Moderately
achieved
Achieved
1 2 3 4 5 6 7
Comments: _________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Section C – Views about what can be done better or differently
15. Reflecting on your reading of the policy brief, please list at least one element of how the policy brief was produced
and designed that should be retained in future policy briefs.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
16. Reflecting on your reading of the policy brief, please list any element(s) of how the policy brief was produced and
designed that should be changed in future policy briefs.
_______________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 62
17. Reflecting on what you learned from reading the policy brief, please list at least one important action that
policymakers, stakeholders, and/or researchers can do better or differently to address the featured policy issue.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
18. Reflecting on what you learned from reading the policy brief, please list at least one important action that you
personally can do better or differently to address the featured policy issue.
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
Section D – Role and background
19. I am a (please tick (√ ) single most appropriate role category):
Broad
role category
Specific role category
Tick
(√)
single
most
appro-
priate
Policymaker Public policymaker (i.e., elected official, political staff, or civil servant) in the national
government
Public policymaker (i.e., elected official, political staff, or civil servant) in a sub-
national government (e.g., province/state or a district if the latter has independent
policymaking authority)
Manager in a district/region (if it does not have independent policymaking authority)
Manager in a healthcare institution (e.g., hospital)
Manager in a non-governmental organization (NGO)
Stakeholder Staff/member of a civil society group/community-based NGO
Staff/member of a health professional association or group
Staff of a donor agency (e.g., European Community, Swedish International
Development Agency) or international organization (e.g., World Health Organization)
Staff of a pharmaceutical or other biotechnology company
Representative of another stakeholder group
Researcher Researcher in a national research institution
Researcher in a university
Researcher in another institution
Other
20. I have been working in my current position for _____ years.
21. If you identified yourself as a policymaker, stakeholder, or "other," please indicate if you have training and/or
extensive experience as a researcher (circle one):
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 63
Yes / No
22. If you identified yourself as a researcher, stakeholder, or "other," please indicate if you have experience as a
policymaker (circle one):
Yes / No
Thank you!
ID #: __________ (Your responses will be kept confidential and data will not be reported in ways that could potentially
identify you or your organization.)
Box 8.3 Policy dialogue guideline
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT Policy Dialogue Guidelines
1. General assessment of the policy brief documents:
►1. Does the policy brief present research findings, policy options and recommendations
appropriately?
►2. Does Policy brief synthesise a large amount of complex information and present
findings and recommendations in a format that enables the reader to easily and quickly
understand an issue?
►3. Was the Policy brief written in clear, jargon-free language, and pitched towards educated
non-specialists in the topic?
►4. Are there other features lacking and needed to be included in the policy brief?
2. The policy issues:
►1. Does the policy brief address a high-priority issue and describe the relevant context of
the issue?
► 2. Does the policy brief provide adequate background information on the policy issue?
► 3. Does the background information reflect the true scenario in our local context?
►4. Are there other aspects of the background information lacking and needed to be
included in the policy brief?
3. Magnitude of the problem:
►1. Does the policy brief provide definition and a short overview of the root causes of the
problem such that its features can be understood; a clear statement on the policy
implications of the issue; shortcomings of the current approach?
► 2. Does the policy brief describe the problem, costs and consequences of past options to
address the problem, and the inadequacies or failures of past policies?
►3. Are there other aspects of the problem lacking and needed to be included in the policy
brief?
4. Policy options ►1. Does the policy brief provide policy recommendations that are actionable and clearly
connected to specific decision-making junctures in the policy-making process?
►2. Was the implementation considerations taken into account?
►3. Are there other policy recommendations lacking and needed to be included
in the policy brief?
5. General comments
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 64
Box 8.4 Policy dialogue evaluation questionnaire
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
Evaluation – Policy Brief
Please circle the number that corresponds to your answer and (if you wish) offer any suggestions about how the policy brief
can be improved.
Several questions make reference to "stakeholders." The term "stakeholders" includes: staff or members of civil society groups;
staff or members of health professional associations or groups; staff of donor agencies (e.g., European Community, Swedish
International Development Agency) or international organizations (e.g., World Health Organization); and staff of
pharmaceutical or other biotechnology companies.
Section A – Views about how the policy brief was produced and designed
1. The policy brief described the context for the issue being addressed. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? _______________________________________
____________________________________________________________________________________
____________________________________________________________________________________
2. The policy brief described different features of the problem, including (where possible) how it affects particular
groups. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? _______________________________________
____________________________________________________________________________________
____________________________________________________________________________________
3. The policy brief described three options for addressing the problem. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
4. The policy brief described what is known, based on synthesized research evidence, about each of the three options and
where there are gaps in what is known. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
5. The policy brief described key implementation considerations. How helpful did you find this approach?
Very Moderately Slightly Neutral Slightly Moderately Very helpful
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 65
unhelpful unhelpful unhelpful helpful helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
6. The policy brief employed systematic and transparent methods to identify, select, and assess synthesized research
evidence. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
7. The policy brief took quality considerations into account when discussing the research evidence. How useful did you
find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
8. The policy brief took local applicability considerations into account when discussing the research evidence. How
helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
9. The policy brief took equity considerations into account when discussing the research evidence. How helpful did you
find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
20. The policy brief did not conclude with particular recommendations. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
21. The policy brief employed a graded-entry format (e.g., a list of key messages and a full report). How helpful did you
find this approach?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 66
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
22. The policy brief included a reference list for those who wanted to read more about a particular systematic review or
research study. How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
23. The policy brief was subjected to a review by at least one policymaker, at least one stakeholder, and at least one
researcher (called a “merit” review process to distinguish it from “peer” review, which would typically only involve
researchers in the review). How helpful did you find this approach?
Very
unhelpful
Moderately
unhelpful
Slightly
unhelpful
Neutral Slightly
helpful
Moderately
helpful
Very helpful
1 2 3 4 5 6 7
How could the policy brief be improved in this regard? ______________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Section B – Overall assessment of the policy brief
24. The purpose of the policy brief was to present the available research evidence on a high-priority policy issue in order
to inform a policy dialogue where research evidence would be just one input to the discussion. How well did the
policy brief achieve its purpose?
Failed Moderately
failed
Slightly
failed
Neutral Slightly
achieved
Moderately
achieved
Achieved
1 2 3 4 5 6 7
Comments: _________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Section C – Views about what can be done better or differently
25. Reflecting on your reading of the policy brief, please list at least one element of how the policy brief was produced
and designed that should be retained in future policy briefs.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
26. Reflecting on your reading of the policy brief, please list any element(s) of how the policy brief was produced and
designed that should be changed in future policy briefs.
_______________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 67
27. Reflecting on what you learned from reading the policy brief, please list at least one important action that
policymakers, stakeholders, and/or researchers can do better or differently to address the featured policy issue.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
28. Reflecting on what you learned from reading the policy brief, please list at least one important action that you
personally can do better or differently to address the featured policy issue.
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
Section D – Role and background
29. I am a (please tick (√ ) single most appropriate role category):
Broad
role category
Specific role category
Tick
(√)
single
most
appro-
priate
Policymaker Public policymaker (i.e., elected official, political staff, or civil servant) in the national
government
Public policymaker (i.e., elected official, political staff, or civil servant) in a sub-
national government (e.g., province/state or a district if the latter has independent
policymaking authority)
Manager in a district/region (if it does not have independent policymaking authority)
Manager in a healthcare institution (e.g., hospital)
Manager in a non-governmental organization (NGO)
Stakeholder Staff/member of a civil society group/community-based NGO
Staff/member of a health professional association or group
Staff of a donor agency (e.g., European Community, Swedish International
Development Agency) or international organization (e.g., World Health Organization)
Staff of a pharmaceutical or other biotechnology company
Representative of another stakeholder group
Researcher Researcher in a national research institution
Researcher in a university
Researcher in another institution
Other
20. I have been working in my current position for _____ years.
23. If you identified yourself as a policymaker, stakeholder, or "other," please indicate if you have training and/or
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 68
extensive experience as a researcher (circle one):
Yes / No
24. If you identified yourself as a researcher, stakeholder, or "other," please indicate if you have experience as a
policymaker (circle one):
Yes / No
Thank you!
ID #: __________ (Your responses will be kept confidential and data will not be reported in ways that could potentially
identify you or your organization.)
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 69
CHAPTER 9
TRAINING WORKSHOP ON ENHANCING CAPACITY TO OWN AND DRIVE THE
AGENDA FOR STRENGTHENING HEALTH SYSTEMS/HEALTH SECTOR REFORMS
9.1 Methods for Training Workshop
This workshop should be the fourth of the series of interventional training workshops. Efforts should
be made to retain the same participants from the first training workshop till this fourth workshop.
The objectives of the workshop should be to enhance participants‟ capacity and knowledge on: (i).
health sector reforms, equity and evidence-informed policy recommendations; (ii). Knowledge
generation, advocacy, communications, and knowledge translation programs and processes; (iii).
Comparative HPSR studies in the areas of research utilization, social health insurance and poverty
reduction; (iv). Emergency preparedness and health systems research on emerging infectious diseases
and health issues during crisis periods such as natural disasters. A pre-workshop questionnaire on the
assessment of health systems performance (see Box 9.1) should be administered.
To achieve the objectives of the workshop lectures should focus on the following topic areas:
(1). Health sector policy reforms, equity and evidence-informed policy recommendations.
(2). Health policy and systems studies in the area of research utilization, social health insurance and
poverty reduction.
(3). Health systems research on prevalent and emerging infectious diseases.
Each lecture could take up to 35 minutes and should be accompanied by an interactive session which
can involve deliberations, comments, questions and answers. A breakdown of the contents of each
lecture is described below:
9.1.1 Health sector policy reforms, equity and evidence-informed policy recommendations
(i). Health Sector Reforms
(ii). Components of health sector reform programmes in developing countries (improving the
performance of the civil service, decentralization, improving the functioning of ministries, broadening
choices for health financing, introducing managed competition, working with the private sector)
(iii). Equity in Health
(iv). Equity in health and health care
(v). Gender equity
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 70
(vi). Conceptual framework to analyze health sector reform and equity (equity in access to health care
services, equity in the utilization of health care services, equity in resource allocation, equity in the
delivery of quality services, equity in the delivery of effective services)
(v). Engaging the public in evidence-informed policymaking.
(vi). Monitoring and evaluation of policies
9.1.2 Health policy and systems studies in the area of research utilization, social health insurance
and poverty reduction
(i). Building research infrastructure for Health Policy Formulation and implementation with respect to
developing a viable health insurance scheme and scaling up poverty reduction.
(ii). The compelling reasons to improve the design and implementation of efficient health policy data
gathering and empirical analysis management and development
(iii). Addressing the problems of availability of empirical health (social insurance and poverty
reduction inclusive)
(iv). Macro data for policy origination, implementation and monitoring of national health policies.
(V). Philosophy and objectives of national health policy and system
(vi). Health policies and health objectives
(vii). The challenges of public health policy and systems research
9.1.3 Health systems research on prevalent and emerging infectious diseases
(i). Types of health systems research needed to address prevalent and emerging infectious diseases
(efficacy and cost-benefit research, disease patterns research, emerging infectious disease research,
immunology and vaccine research, disease-specific research, health promotion and communications
research, safe food and water research, knowledge translation research)
(ii). Capacity-building for health systems research (attraction and retention, education and training,
collaboration and networks, funding)
(iii). Communication: dissemination of research findings and obtaining public health input
(iv). Surveillance, informatics and databases
(v). Agenda for public health systems research on emergency preparedness against infectious diseases
(vi). Elements of emergency preparedness (operations-ready workers and volunteers, countermeasures
and mitigation strategies, health risk assessment, psychosocial outcomes and community resilience,
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 71
mass health care, public information and communication, testing operational capabilities, interagency
communication and coordination).
9.2 Outcomes evaluation
The tool to use for the outcomes evaluation was developed at McMaster University Canada by
Johnson and Lavis (2009). According to Johnson and Lavis (2009), the outcomes evaluation assess
three areas in which knowledge translation (KT) platforms anticipate achieving particular outcomes as
follows:
(i). the availability of health research evidence about high-priority policy issues;
(ii). the strength of relationships among policymakers, researchers, and knowledge-translation
specialists (i.e., individuals, other than journalists, whose expertise and role relate to
supporting the use of research evidence outside the research community); and
(iii). the strength of policymakers‟ and researchers‟ capacity to support the use of health
research evidence in health systems policymaking.
Specifically, the outcomes evaluation surveys policymakers', stakeholders', and researchers' views
about each of the above – with the focus being on policymakers' access to evidence and perceptions of
the utility of available evidence, as well as their interactions with researchers, and their participation in
training courses related to acquiring, using, presenting, and promoting evidence. The outcomes
evaluation questionnaire developed by Johnson and Lavis (2009) is shown in Box 9.2. This should be
administered at the end of the training sessions during the fourth training workshop.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 72
Box 9.1 Users’ perception of health systems operations questionnaire HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
Questionnaire on public & user’s perception on health systems operations
Please answer the questions correctly. The questionnaire is purely for research purpose only.
A. SOCIO-DEMOGRAPHIC DATA
- - -
(iii)
B. ASSESSMENT OF HEALTH SYSTEMS OPERATIONS
1. Health Information & Evidence
(a). To what extent do you have access to health information (eg. about new drugs, technologies, disease
prevention & health programmes)?
(b). How can you rate your confidence (reliability) on available health information?
(c). What is the level of sustainability of sources of health information (eg. media, health institutions,
government & non-government agencies)?
(d). How adequate is the role of government in the provision of health information?
2. Medical Products and Technologies
(a). How would you rate the availability of and access to essential medical products and technologies (eg.,
drugs, vaccines, bednets, chemicals) to the public?
(b). How would you rate the quality of medical products and technologies available to the public?
(c). How consistent are the availability of essential medical products and technologies to the public?
(d). How would you rate the effort of the government in ensuring the availability of medical products and
technologies to the public?
3. Health Finance
(a). How adequate is your personal financial resources for meeting your arising health needs?
(b). How would you rate the level of public confidence on health financial management policies and
practice (budgeting, expenditure, accounting & auditing) in the health sector?
(c). How would you rate the sustainability of the private and government health financing mechanisms?
(d). How would you rate the adequacy of government health finance policies (eg., National health
insurance scheme NHIS) in meeting the health needs of the general public.
4. Health Service Delivery
(a). How would you rate the availability and affordability of quality health service delivery to the public?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 73
(b). How would you rate the level of public confidence on the process of effective health service delivery?
(c). How would you rate the sustainability of health service delivery mechanism available to the public?
(d). How would you rate the efforts of the government to improve efficiency of health service delivery to
the public?
5. Health Workforce
(a). How would you rate the availability of and distribution of qualified health personnel?
(b). How would you rate the level of public confidence on the available health personnel to deliver quality
health services?
(c). How would you rate the level of incentives (eg., salaries, allowances) available to health personnel?
(d). How would you rate the sustainability of supply of qualified health personnel?
(e). How would you rate the efforts of the government in training, development and motivation of the
health workforce?
6. Leadership and Governance
(a). How would you rate the level of public and user‟s involvement by the government in the management
of the health system?
(b). How would you rate the public confidence in the capacity of the government to formulate and
implement functional health policies?
(c). How would you rate the effort of the government to maintain stable political environment that
promotes functional and sustainable health system?
(d). How would you rate the role of the government to ensure efficiency in accountability, transparency
and regulatory frameworks in the health system?
Comments (optional) How do you think the health systems can be improved in Nigeria?.................................................................
...................................................................................................................................................................................
................................................................................................................................ ...................................................
...................................................................................................................................................................................
...................................................................................................................................................................................
...........................................................................................................................................
Thank you for your participation
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 74
Box 9.2 Evaluation- outcomes questionnaire
HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
Evaluation – Outcomes Questionnaire
Please circle the number that corresponds to your answer and (if you wish) offer specific comments on any issues raised in particular
questions by identifying the question by number and adding your comments in the space provided on the final page of the
questionnaire.
In this questionnaire we refer to "the KT platform's jurisdiction." In your case this is...........
A number of questions offer "Don't know" as a response option. "Don't know" should be selected only if you feel that you do not
have sufficient information or knowledge to form a view.
Part 1: Views about evidence availability, relationships, and capacity
Section 1A – Availability of health research evidence about high-priority policy issues
1) Please indicate how often, in your view, the following activities or events took place in the KT platform's jurisdiction
(in light of how often it was feasible for each activity or event to take place) over the last two years (or for a shorter period if
you became involved in policymaking more recently).
Never Very rarely Rarely Occasionally Frequently Very
frequently
Always
1 2 3 4 5 6 7
a. Researchers were apprised of high-priority policy issues so that they could orient
their research accordingly
1 2 3 4 5 6 7 Don't
know
b. Primary research was undertaken on high-priority policy issues 1 2 3 4 5 6 7 Don't
know
c.
Systematic reviews of the research literature (i.e., reviews of the research literature
that follow explicit rules to reduce bias in searching the literature, identifying
eligible articles, extracting data, etc.) were conducted or updated on high-priority
policy issues
1 2 3 4 5 6 7 Don't
know
d.
Policy briefs that described evidence about a problem, options for addressing the
problem, and key implementation considerations were produced on high-priority
policy issues
1 2 3 4 5 6 7 Don't
know
e. Brief summaries of articles or reports about primary research (i.e., research
involving original data collection) were produced on high-priority policy issues
1 2 3 4 5 6 7 Don't
know
f. Brief summaries of systematic reviews were produced on high-priority policy
issues
1 2 3 4 5 6 7 Don't
know
2) Please indicate how often, in your view, the following activities or events took place in the KT platform's jurisdiction
(in light of how often it was feasible for each activity or event to take place) over the last two years
(or for a shorter period if you became involved in policymaking more recently).
Never Very rarely Rarely Occasionally Frequently Very
frequently
Always
1 2 3 4 5 6 7
a. Copies of articles or reports about primary research on high-priority policy issues
were widely disseminated to policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
b. Systematic reviews of the research literature on high-priority policy issues were
widely disseminated to policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
c.
Policy briefs that described evidence about a problem, options for addressing the
problem, and key implementation considerations were widely disseminated to
policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
d. Brief summaries of articles or reports about primary research on high-priority
policy issues were widely disseminated to policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 75
e. Brief summaries of systematic reviews on high-priority policy issues were widely
disseminated to policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
f. Newsletters that highlight research evidence on high-priority policy issues were
widely disseminated to policymakers working on these issues
1 2 3 4 5 6 7 Don't
know
3) Please indicate how often, in your view, the following activities or events took place in the KT
platform's jurisdiction (in light of how often it was feasible for each activity or event to take place)
over the last two years (or for a shorter period if you became involved in policymaking more recently).
Never Very rarely Rarely Occasionally Frequently Very
frequently
Always
1 2 3 4 5 6 7
a. Policymakers had access to a personal computer with a functional internet
connection
1 2 3 4 5 6 7 Don't
know
b. Policymakers had access to research evidence on high-priority policy issues
through a website focused on these issues
1 2 3 4 5 6 7 Don't
know
c. Policymakers had access to research evidence on high-priority policy issues
through a searchable database focused on these issues
1 2 3 4 5 6 7 Don't
know
d. Policymakers had access to research evidence on high-priority policy issues
through a list-serve focused on these issues
1 2 3 4 5 6 7 Don't
know
e.
Policymakers had access to research evidence on high-priority policy issues
through a network of researchers or research institutions who could respond in an
ad hoc way to questions about these issues
1 2 3 4 5 6 7 Don't
know
f.
Policymakers had access to research evidence on high-priority policy issues
through a network of researchers or research institutions who maintained some
reserve capacity (i.e., financial and/or human resources that can be redirected when
required) to respond in a systematic way to questions about these issues
1 2 3 4 5 6 7 Don't
know
4) Please indicate the extent to which you agree or disagree with the following statements concerning the
research evidence that was available on high-priority policy issues in the KT platform's jurisdiction over the last two years
(or for a shorter period if you became involved in policymaking more recently).
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a. Policymakers consider that the available research evidence had little practical
policy application
1 2 3 4 5 6 7 Don't
know
b. Policymakers consider that the available quantitative empirical research evidence
was not helpful in policymaking about health systems
1 2 3 4 5 6 7 Don't
know
c. Policymakers consider that the available research evidence lacked credibility 1 2 3 4 5 6 7 Don't
know
d. Policymakers consider that the available qualitative empirical research was helpful
in policymaking about health systems
1 2 3 4 5 6 7 Don't
know
e. Policymakers consider that the available conceptual (i.e., non-empirical) research
evidence was not helpful in policymaking about health systems
1 2 3 4 5 6 7 Don't
know
Section 1B - Relationships among policymakers and researchers
5) Please indicate how often, in your view, the following activities or events took place in the KT platform's jurisdiction
(in light of how often it was feasible for each activity or event to take place) over the last two years
(or for a shorter period if you became involved in policymaking more recently).
Never Very rarely Rarely Occasionally Frequently Very
frequently
Always
1 2 3 4 5 6 7
a.
Policymakers interacted with researchers as part of a priority-setting process to
identify high-priority policy issues for which primary research and systematic
reviews are needed
1 2 3 4 5 6 7 Don't
know
b. Policymakers interacted with researchers as part of primary research or systematic 1 2 3 4 5 6 7 Don't
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 76
reviews about high-priority policy issues that they commissioned know
c.
Policymakers interacted with researchers as part of primary research or systematic
reviews about high-priority policy issues with which they were involved as a co-
investigator
1 2 3 4 5 6 7 Don't
know
d. Policymakers interacted with researchers to provide assistance with undertaking
primary research or systematic reviews about high-priority policy issues
1 2 3 4 5 6 7 Don't
know
e.
Policymakers interacted with researchers to provide assistance with interpreting the
findings from primary research or systematic reviews about high-priority policy
issues
1 2 3 4 5 6 7 Don't
know
f.
Policymakers interacted with researchers to provide assistance with designing and
executing strategies to support policymakers‟ use of the findings from primary
research and systematic reviews about high-priority policy issues
1 2 3 4 5 6 7 Don't
know
g. Policymakers interacted with researchers to obtain assistance with acquiring
existing research evidence about high-priority policy issues
1 2 3 4 5 6 7 Don't
know
h.
Policymakers interacted with researchers to obtain assistance with assessing the
quality and local applicability of existing research evidence about high-priority
policy issues
1 2 3 4 5 6 7 Don't
know
i.
Policymakers interacted with researchers to obtain assistance with presenting
existing research evidence about high-priority policy issues to other policymakers
in a useful way
1 2 3 4 5 6 7 Don't
know
j. Policymakers interacted with researchers through legislative committee testimonies
and government-sponsored expert committees or public hearings
1 2 3 4 5 6 7 Don't
know
k. Policymakers interacted with researchers through a rapid-response service designed
to support policymaking processes
1 2 3 4 5 6 7 Don't
know
l.
Policymakers interacted with researchers through policy dialogues designed to
discuss high-priority policy issues and how research evidence can inform how to
address these issues
1 2 3 4 5 6 7 Don't
know
m. Policymakers interacted with researchers through research conferences 1 2 3 4 5 6 7 Don't
know
n. Policymakers interacted with researchers through informal conversations with
personal contacts
1 2 3 4 5 6 7 Don't
know
o. Policymakers interacted with researchers through long-term partnerships (e.g.,
through an advisory board)
1 2 3 4 5 6 7 Don't
know
Section 1C – Policymakers’ capacity to
support the use of health research evidence in health systems policymaking
6) Please indicate how often, in your view, the following activities or events took place in the KT platform's
jurisdiction (in light of how often it was feasible for each activity or event to take place) over the last two years
(or for a shorter period if you became involved in policymaking more recently).
Never Very rarely Rarely Occasionally Frequently Very
frequently
Always
1 2 3 4 5 6 7
a.
Policymakers participated in training courses, workshops, seminars, or briefings to
develop their capacity to acquire existing research evidence about high-priority
policy issues
1 2 3 4 5 6 7 Don't
know
b.
Policymakers participated in training courses, workshops, seminars, or briefings to
develop their capacity to assess the quality and local applicability of existing
research evidence about high-priority policy issues
1 2 3 4 5 6 7 Don't
know
c.
Policymakers participated in training courses, workshops, seminars, or briefings to
develop their capacity to present existing research evidence about high-priority
policy issues
1 2 3 4 5 6 7 Don't
know
d.
Policymakers participated in training courses, workshops, seminars, or briefings to
develop their capacity to promote a climate that supports research use and ensure
policymaking processes have a place for research evidence
1 2 3 4 5 6 7 Don't
know
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 77
Section 1D – KT platform’s contributions
7) Please indicate the extent to which you agree or disagree with these statements about the KT platform‟s contributions
over the last two years (or for a shorter period if you became involved in policymaking more recently).
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly
agree
1 2 3 4 5 6 7
a. The KT platform has contributed to the availability of research evidence on high
priority issues
1 2 3 4 5 6 7 Don't
know
b.
The KT platform has contributed to strengthening relationships among
policymakers, researchers, and knowledge-translation specialists (i.e., individuals,
other than journalists, whose expertise and role relate to supporting the use of
research evidence outside the research community)
1 2 3 4 5 6 7 Don't
know
c.
The KT platform has contributed to strengthening policymakers‟ and researchers‟
capacity to support the use of health research evidence in health systems
policymaking.
1 2 3 4 5 6 7 Don't
know
Part 2: Assessment of your unit’s or department’s capacity to acquire, assess, adapt, and apply health research
evidence on high-priority policy issues
Section 2A - Acquiring health research evidence
8) Can your unit/department find and obtain research evidence on high-priority policy issues?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a. We have skilled staff to find and obtain research evidence on high-priority policy
issues.
1 2 3 4 5 6 7 Don't
know
b. Our staff has enough time to find and obtain research evidence on high-priority
policy issues.
1 2 3 4 5 6 7 Don't
know
c. Our staff has the incentive to find and obtain research evidence on high-priority
policy issues (e.g., it is used in the policymaking process).
1 2 3 4 5 6 7 Don't
know
d. Our staff has the resources to find and obtain research evidence on high-priority
policy issues.
1 2 3 4 5 6 7 Don't
know
e. We have arrangements with external experts who find and obtain research evidence
on high-priority policy issues.
1 2 3 4 5 6 7 Don't
know
9) Is your unit/department looking for research evidence in the right places?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a.
We routinely look for research in journals that we can access by subscription,
online, or through a library. An example is the Bulletin of the World Health
Organization.
1 2 3 4 5 6 7 Don't
know
b.
We routinely look for research in non-journal reports (i.e., grey literature) that we
can access online, through a library, or by direct mail. An example is a program
evaluation report that was prepared for a donor.
1 2 3 4 5 6 7 Don't
know
c.
We routinely look for research in databases that we can access by subscription, on
CD, or online. Examples are The Cochrane Library and the Reproductive Health
Library.
1 2 3 4 5 6 7 Don't
know
d. We routinely look for information on websites that collate and/or evaluate sources
of research evidence. An example is The Reproductive Health Library.
1 2 3 4 5 6 7 Don't
know
e. We routinely interact with researchers through formal and informal networking
meetings.
1 2 3 4 5 6 7 Don't
know
f. We routinely get involved with researchers as a host for their research activities,
funding sponsor, or co-investigator.
1 2 3 4 5 6 7 Don't
know
g. We routinely learn from our peers through formal and informal networks to
exchange ideas, experiences, and best practices.
1 2 3 4 5 6 7 Don't
know
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 78
Section 2B - Assessing health research evidence
10) Can your unit/department tell if the available research evidence warrants serious consideration?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a. Staff in our unit/department has skills and tools for evaluating the quality of
research evidence.
1 2 3 4 5 6 7 Don't
know
b. Staff in our unit/department has skills and tools for taking into account equity
considerations in evaluating the research evidence.
1 2 3 4 5 6 7 Don't
know
c. Staff in our unit/department has the skills and tools for evaluating the local
applicability of research evidence that was conducted in other settings.
1 2 3 4 5 6 7 Don't
know
d. Staff in our unit/department has the skills and tools for taking into account scaling
up considerations in evaluating the research evidence.
1 2 3 4 5 6 7 Don't
know
e.
Staff in our unit/department has the skills and tools to identify the relevant
similarities and differences between what we do and what the research evidence
says.
1 2 3 4 5 6 7 Don't
know
f.
Our unit/department has arrangements with external experts who use critical
appraisal skills and tools to help us to assess if the research evidence warrants
serious consideration.
1 2 3 4 5 6 7 Don't
know
Section 2C - Adapting health research evidence
11) Can your unit/department present research evidence to policymakers in a useful way?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a.
Our unit/department has enough skilled staff with time, incentive, and resources
who use research communications skills to present research evidence concisely and
in accessible language.
1 2 3 4 5 6 7 Don't
know
b.
Our unit/department has enough skilled staff with time, incentive, and resources
who use research communications skills to synthesize in one document all relevant
research evidence, along with information and analyses from other sources.
1 2 3 4 5 6 7 Don't
know
c.
Our unit/department has enough skilled staff with time, incentive, and resources
who use research communications skills to link research evidence to key policy
issues facing policymakers.
1 2 3 4 5 6 7 Don't
know
d.
Our unit/department has enough skilled staff with time, incentive, and resources
who use research communications skills to provide specific policy options to
policymakers.
1 2 3 4 5 6 7 Don't
know
e.
Our unit/department has arrangements with external experts who use research
communications skills to help us to present research evidence to policymakers in a
useful way.
1 2 3 4 5 6 7 Don't
know
Section 2D - Applying health research evidence
12) Does your unit/department lead by example and show how it values the use of research evidence?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a. Using research evidence is a priority in our unit/department. 1 2 3 4 5 6 7 Don't
know
b. Our unit/department has committed resources to ensure research evidence is
acquired, assessed, adapted, and applied in policymaking processes.
1 2 3 4 5 6 7 Don't
know
c. Our unit/department ensures staff is involved in discussions about how research
evidence relates to our main goals.
1 2 3 4 5 6 7 Don't
know
d.
The management of our unit/department has clearly communicated our strategy and
policy priorities so those conducting or monitoring research know what is needed
to support our goals.
1 2 3 4 5 6 7 Don't
know
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 79
e. We communicate internally in a way that ensures there is information exchanged
across the entire unit/department.
1 2 3 4 5 6 7 Don't
know
f. Our corporate culture values and rewards flexibility, change, and continuous
quality improvement with resources to support these values.
1 2 3 4 5 6 7 Don't
know
13) Do your unit's/department's policymaking processes have a place for research evidence?
Strongly
disagree
Disagree Somewhat
disagree
Neither agree
nor disagree
Somewhat
agree
Agree Strongly agree
1 2 3 4 5 6 7
a.
When we develop or change a policy, we usually allow enough time to identify
researchable questions and acquire, assess, and consider research evidence that
addresses these questions.
1 2 3 4 5 6 7 Don't
know
b.
Our management team has enough expertise to evaluate the feasibility of each
policy option, including potential impact across the unit/department and on
patients/clients, healthcare professionals, development partners, and other
stakeholders.
1 2 3 4 5 6 7 Don't
know
c.
Policymakers in our unit/department usually give formal consideration to any
perspectives on a policy issue brought forward by staff who have identified high
quality, locally applicable research evidence on the policy issue.
1 2 3 4 5 6 7 Don't
know
d. Staff and appropriate stakeholders know when and how major policies will be
developed or changed.
1 2 3 4 5 6 7 Don't
know
e. Staff and appropriate stakeholders know how and when they can contribute
research evidence and how that information will be used.
1 2 3 4 5 6 7 Don't
know
f. Staff who have provided research evidence usually participate in policymaking
discussions.
1 2 3 4 5 6 7 Don't
know
g. Relevant on-staff researchers are part of policymaking discussions. 1 2 3 4 5 6 7 Don't
know
h. Staff and appropriate stakeholders receive feedback on policymaking processes
with a rationale for the new or changed policy.
1 2 3 4 5 6 7 Don't
know
i. Staff and appropriate stakeholders are informed of how available research evidence
influenced the choices that were made in our unit/department.
1 2 3 4 5 6 7 Don't
know
Part 3: Role and background
14) I am a (please tick (√ ) single most appropriate role category):
Broad
role category
Specific role category
Tick
(√)
single most
appro-priate
Policymaker Public policymaker (i.e., elected official, political staff, or civil servant) in the national
government
Public policymaker (i.e., elected official, political staff, or civil servant) in a
su national government (e.g., province/state or a district if the latter has
independent policymaking authority)
Manager in a district/region (if it does not have independent policymaking authority)
Manager in a healthcare institution (e.g., hospital)
Manager in a non-governmental organization (NGO)
Stakeholder Staff/member of a civil society group/community-based NGO
Staff/member of a health professional association or group
Staff of a donor agency (e.g., European Community, Swedish International
Development Agency) or international organization (e.g., World Health Organization)
Staff of a pharmaceutical or other biotechnology company
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 80
Representative of another stakeholder group
Researcher Researcher in a national research institution
Researcher in a university
Researcher in another institution
Other
15) I have been working in my current position for _____ years.
16) If you identified yourself as a policymaker, stakeholder, or "other," please indicate if you have training
and/or extensive experience as a researcher (circle one):
Yes / No
17) If you identified yourself as a researcher, stakeholder, or "other," please indicate if you have experience as a policymaker
(circle one):
Yes / No
Thank you!
ID #: __________ (Your responses will be kept confidential and data will not be reported in ways that could potentially
identify you or your organization.)
Additional Thoughts (Optional)
Do you have any comments regarding issues raised in particular questions?
(If the space provided is insufficient to accommodate all your ideas, please feel free to attach additional pages.)
_______________________________________________________________________________________
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 81
CHAPTER 10
TRAINING WORKSHOP FOR CAPACITY ENHANCEMENT ON LEADERSHIP,
GOVERNANCE AND MANAGEMENT FOR HEALTH SYSTEMS STRENGTHENING
10.1 Methods for Training Workshop
This workshop should be the fifth and final of the series of interventional training workshops. As in
previous workshops efforts should be made to retain the same participants from the first training
workshop till this fifth workshop. This fifth workshop should focus on enhancing the capacity for
effective leadership, governance and the management for the strengthening of the health systems. In
the Nigeria study the theme of the workshop was: “Leadership, Governance and Management:
Critical Competencies for Health Systems strengthening”. The lecture should be centered on the
following topic area:
10.1.1 Leadership, Governance and Management: Critical Competencies for Health Systems
strengthening
(i). Leadership and governance factors that ensure functionality of the health systems (Policy
guidance; Intelligence and oversight; Collaboration and coalition building; Regulation; System design;
Accountability
(ii). Impact of weak leadership and governance
(iii). Policy options and implementation strategies for addressing leadership and governance problem
(iv). Accountability of the key actors in the health system to the beneficiaries; a policy process that
enables the interplay of the key competing interest groups to influence policymaking; sufficient state
capacity, power, and legitimacy to manage and regulate the policymaking process; effective
engagement of non-state actors in the policy arena; instituting performance measurement system;
establishing health system research mechanism
(v). Leading/Managing and Health Systems Strengthening
(vi). Leadership and management strategies to improved health outcomes (scanning; focusing;
aligning and mobilizing; inspiring; planning; organizing; implementing; monitoring and evaluation)
(vii). Promoting Good Governance in Public and Private Health Organizations
(viii). Dimensions of governance in the health sector (information and assessment capacity; policy
formulation and planning; social participation and system responsiveness; accountability,
transparency, and regulation).
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 82
(ix). Improved Health System Performance issues (Equity; Access; Quality; Efficiency;
Sustainability).
10.2 Staff and organizational appraisal/performance assessment
Staff and organizational appraisal and performance assessment are major ways to evaluate the
effectiveness of leadership and governance of a health ministry in strengthening the health systems.
People-centered health systems cannot be strengthened without good management and leadership.
Good leadership aims to manage and lead better so that teams, units, or organizations can fully use and
continuously develop their potential to transform human and financial resources and other inputs into
improved services and, ultimately, improved health outcomes. The questionnaire in Box 10.1 is
designed to evaluate human resource management in terms of of recruitment policies, performance
assessments, and staff appraisals. The questionnaire is also a useful tool for organizational self-
assessment on the use research to make informed decision. The questionnaire was based on the
following assessment tools:
(i). Huddart (2005) developed for the Jordan Human Resources Development Project Report No. 5.
(ii). Canadian health services research foundation (CHSRF) self-assessment tool and discussion guide
for Health services management and policy organizations.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 83
Table 10.1 Staff and organizational appraisal/performance assessment questionnaire HEALTH POLICY AND SYSTEMS RESEARCH PROJECT
STAFF AND ORGANIZATIONAL APPRAISAL/PERFORMANCE ASSESSEMENT QUESTIONNAIRE
HR Components Stages of Human Resource Management Development and Characteristics Comments
1 2 3 4
HR Management Capacity
HR Budget There is no
budget
allocated for
HR staff
or HR activity
within the
organization.
There is limited
money available to
fund an HR
position or to
conduct HR
activities (e.g.
training, systems
development,
performance
planning and
evaluation.
Budget is allocated
for HR staff and
related activities.
Allocation is
irregular and
cannot be relied
upon for any useful
long-range
planning or the
development of
HR systems.
Money for HR staff and
related activities is a
permanent budget item,
reviewed annually and
adjusted if possible.
HR Staff There are no
staff
specifically
charged
with
responsibility-
y for
HR functions.
There are HRM
staff in the
organization, but
they have limited
experience related
to this field
(personnel,
recruitment,
management) and
/or have other
functions in the
organization as well
as HRM.
There are trained
HRM staff in the
organization, but
only at a level to
maintain basic
procedures and
record-keeping
functions.
There are experienced
HRM staff in the
organization who
maintain HR functions.
They participate in long-
range planning for the
organization.
Human Resource Planning
Organizational
Mission and
Goals
No formal
mission
statement or
organization
goals
exist.
Mission/goals exist
but are not formally
linked to HR
planning (e.g.
staffing plan,
training, etc.)
Mission/goals
linked to annual
HRD planning and
also used for
forecasting
long-range staffing
and recruitment
needs.
Mission/goals linked to
annual HR planning and
also used for forecasting
long-range staffing and
recruitment needs.
HR Planning No annual HR
plan
exists.
Annual HR plan
exists, but is not
based on a formal
assessment of the
mission,
organizational
goals, staffing
needs, training
outputs or existing
employee data.
Annual HR plan
exists based on
organizational
goals, staffing
needs, training and
employee data, but
it is not further
evaluated for
effectiveness.
Annual HR plan based
on organizational goals
and training outputs
exists. It is implemented,
evaluated and used for
long-range strategic
planning.
Personnel Policy and Practice
Job
Classification
System
No formal
system
exists to
classify jobs
and the skills
There is some
attempt to classify
jobs, but it is
uneven and
incomplete.
A job classification
system exists, but
it is not used as a
basis for other
HRM functions
A job classification
system exists and is used
in a formal manner for
other HR planning and
staffing functions.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 84
and
qualifications
required for
each position.
(e.g. job
descriptions,
hiring, salary and
benefits).
Compensation
& Benefits
System
No formal
system exists
for
determining
the salary
scale and
benefits
provided for
each position
within the job
classification.
A formal system
exists, but it is not
used in a routine
manner.
A formal system
exists, is
understood by all
employees and
used in a consistent
manner.
A formal system exists
and is used consistently.
It is also used to
determine salary upgrades
and merit awards.
Recruitment,
Hiring,
Transfer and
Promotion
No formal
process
exists for
recruiting,
hiring,
transfer and
promotion
according
to job
descriptions.
There are systems
for hiring, etc. but
they are not
followed.
There are formal
systems, based on
established criteria,
but they are not
used consistently.
There are formal systems,
monitored and used in all
hiring, transfer and
promotion decisions.
Orientation
Program
There is no
formal
orientation
program for
new
employees.
There is a program,
but it is not
implemented on a
regular basis.
Orientation is
offered in a routine
manner, but does
not emphasize the
mission, goals and
performance
standards expected
by the
organization.
Orientation is offered to
all new employees,
emphasizes the mission,
goals and performance
standards expected, and
also makes people fees
welcomed and valued.
Policy Manual
(Org‟al chart,
work hours,
policy, discipline,
grievances,
benefits, travel,
etc.)
No policy
manual exists.
Policy manual does
exist, but it is out of
date and does not
include all of the
relevant
information.
A current policy
manual does exist
but it is not
available to all
employees and is
not always used as
a basis for
personnel
decisions.
An upgraded policy
manual does exist and is
available to all
employees. It serves
as a reference guide to all
questions about
employment in the
organization and is
reviewed and updated
regularly.
Termination
& Grievance
Procedures
No formal
procedures
exist.
Formal procedures
do exist, but they
are not clearly
related to
performance
standards.
Formal procedures
based on
performance
standards exist, but
they are not
followed in any
consistent manner.
Formal procedures based
on performance standards
are known to all
employees and used
consistently.
Relationship
with Unions
(if appropriate)
There is no
link between
HRM,
management
and the
Links exist between
HRM, management
and
union, but roles are
not clear.
Management
involves HR union
issues, but on an
irregular basis.
Management, HRM and
the union work together
to resolve issues and
prevent problems.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 85
union/s.
Labor Law
Compliance
There is no
review of
HR policies to
ensure
compliance
with local
and/or
national labor
law.
There is some effort
to review labor law,
but it
is not done on a
regular basis.
A review of the
labor law is done
regularly as a
formal part of the
HR function, but
policy is not
always adjusted to
ensure compliance.
HR policy and practice is
adjusted as needed to be
in compliance with the
local and/or national
labor law.
Human Resource Data
Employee Data
(No , Where
deployed, skill
level, gender,
cadre, year of
hire, etc.)
None of this
data is
collected on
any kind of
systematic
basis.
Most of this data is
collected, but not
maintained or kept
up to date.
All of this data is
available and up to
date, but data is not
formally used in
HR planning or
forecasting.
All of this data is
available and up to date.
Systems are in place.
Data is formally used in
HR planning and
forecasting.
Computerisation
of Data
There are no
computers or
data systems
available to
the
organization,
externally or
internally.
There are
computers in
place, but no
resources to
develop systems for
data management.
Computers and
data
management
systems are
available, but HR
information is not
produced regularly
for management
decision making.
Computers and data
management systems are
in place and data files are
up to date. HR
information is regularly
produced and used for
decision-making.
Personnel
Files
No individual
employee
records exist.
Limited employee
personnel files are
maintained but not
regularly updated.
Personnel files for
all employees are
maintained & kept
up to date but there
is no policy for
employee access or
use of this data.
Updated personnel files
for all employees exist &
also policies for
appropriate use
(e.g. confidentiality,
employee access).
Performance Management
Job Descriptions
No job
descriptions
are developed.
Some staff have job
descriptions, but
they are not always
up to date and/or
are very
general, lacking job
responsibilities and
supervision.
All staff have job
descriptions, but
they are not all
complete or
up to date with
specific duties and
lines of
supervision.
Complete job descriptions
exist for every employee
and are kept up to date
through a regular process
of review. Specific duties
are clearly stated.
Staff
Supervision
There is no
clear
system of
supervision.
Lines of authority
are unclear. Staff
are not recognized
for their
achievements.
There are
established lines of
authority, but the
supervisor‟s role &
function is not
understood and
little supervision
takes place.
Limited staff
recognition.
Supervisors understand
their roles & lines of
authority and meet
regularly with their
employees to develop
work plans, evaluate
performance and publicly
recognize staff for their
achievements.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 86
Work
Planning and
Performance
Review
There is no
work
planning and
performance
review system
in place.
A work planning
and performance
review system is in
place, but it
is informal and
does not include
work plans and
performance
objectives
developed jointly
with staff.
There is a formal
system and
supervisors are
required to develop
work plans and
performance
objectives with
each employee and
to review
performance in the
past, but this is not
done on a
consistent basis.
Supervisors and
employees develop work
plans jointly and
performance reviews are
con-ducted on a regular
basis. Orientation
sessions and a manual are
provided to all staff.
Reviews are used for
personnel decisions.
Training
Staff Training There is no
staff training
plan.
Training is offered
on an ad hoc basis,
but is not based on
a formal process of
assessing staff
needs nor is it
linked to the
organization‟s key
priorities and
changes in the
health sector and
health practices.
Training is a
formal
component of the
organization and
linked to staff &
organizational
needs, but it is not
available for all
staff, nor is it
evaluated for
results.
Training is a valued part
of the organization and
opportunities are
developed for staff based
on their needs and also on
those of the organization.
Management
& Leadership
Development
There is no
policy or
philosophy
regarding
the
importance of
developing
strong
management
capacity
and future
leaders for
the
organization.
There is an
emphasis on
developing
management
capacity but it is no
done on a regular
basis.
The organization
makes an effort to
develop managers
and future leaders
through training,
and also through
mentoring and
challenging job
assignments, but
participation is
selective.
A plan for management
and leadership develop-
ment is in place and there
is an opportunity for
everyone to participate
based on performance
and other established
criteria.
Links to
External Pre-
Service
Training
There is no
formal link
with the pre-
service
training
institutions
which train
employees for
the health
sector.
There is a loose
relation-ship
between the
organization and
pre-service training
institutions but it is
not used in any
formal way for
workforce training
and development.
The organization
and pre-service
training institutions
work together to
ensure that the
curriculum is based
on skills,
knowledge and
attitudes required
in the
workplace.
The organization and pre-
service training
institutions also offer
regular in-service
training for staff already
in the workplace to
upgrade their skills and
knowledge (e.g. mgmt.
training).
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 87
Organizational Self-assessment on the use research to make informed decisions
Based on the self-assessment, our organization should work on the following areas so that we can use research
better to make informed decisions that help meet our goals and objectives: Comments
1. Establish research as a
priority in our organization: I
feel research in our organization
should have: (Check one)
(a). Much
higher
priority
(b).
Somewhat
higher
priority
(c). The same
priority
(d). Somewhat
lower priority
(e). Much
lower priority
2. Integrate the use of
research into the work of
people in our organization: I
feel we need to: (Check one)
(a).
Integrate
research
much
more
often
(b).
Integrate
research
slightly
more often
(c). Maintain
our current
level of
integrating
research
(d). Integrate
research
slightly less
often
(e). Integrate
research
much less
often
3. Encourage the use of
research by our decision
makers: I feel our decision
makers: (Check one)
(a). Do
not use
research
at all
(b). Use
research
poorly
(c). Use
research
inconsistently
(d). Use
research with
some
consistency
(e). Use
research
well/enough
4. Increase our capacity for
research:
(Check all that apply. If you
have more than one answer,
please rate your needs from 1 to
5, with 1 being the highest
priority.) We need: (Check one)
(a).
Skilled
staff
(b).
Resources
(c). Time
(d). Incentives
(e).
Arrangements
with external
experts
5. Acquisition of research: We
need better access to: (Check all
that apply. If you have more
than one answer, please rate
your needs from 1 to 6, with 1
being the highest priority.)
(a).
Journals
& Non-
journal
reports
(grey
literature)
(b).
Databases
(c). Web sites
(d).
Opportunities
to work with
researchers
(e). Learning
from peers
6. Assessment of research: We
need to:
(Check the one that is most
appropriate or best describes
your situation.)
(a). Begin
to assess
and adapt
research
(b). Assess
and adapt
research
more often
(c). Maintain
our current
ability to
assess and
adapt
research
(d). Adapt and
assess research
a bit less often
(e). Adapt and
assess
research
much less
often
7. Linking of research results
to key issues facing our
decision makers: Our decision
makers need to: (Check the one
that is most appropriate or best
describes your situation.)
(a). Begin
to
consider
research
in making
decisions
(b).
Consider
research
more often
in making
decisions
(c). Maintain
our current
frequency of
considering
research
(d). Consider
research a bit
less often
(e). Consider
research
much less
often
PARTICIPANT BIODATA
(i). Gender: Male
(ii). Age category: 25- -
OFFICIAL DESIGNATION ATTRIBUTES
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 88
(i). Name of your organization:…………………………………………………………………………..............................
(ii). Designation:………………………………………………………………………………….........................................
(iii). Duration in designation:………………………………………………………………………....................................
(iv). Influence on policymaking process: Direct
(vi). Membership of health-
Participant Workshop No……………………
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 89
CHAPTER 11
PROJECT EVALUATION DESIGN
11.1 Conceptual design of the evaluation
The conceptual design of the planned process evaluation takes into consideration the various
constituents of the health system. The constituents of the health systems have been summarized into eight
priority areas in line with the WHO‟s Framework for Action on health systems WHO (2007). These
priority areas include health administration, service delivery, information and evidence, medical
products and technologies, health work force, health financing, population and health interventions,
leadership and governance. These priority areas, which have also been described, by WHO (2007) as
health system building blocks, have been used to categorize the administrative data/data from
management information system in this manual (Table 11.1). Under each category the main data
components/elements, composite indicators and the specific indicators to be measured have been
identified. Evaluating the performance of the major constituents of the health system using a
combination of monitoring/operational research and social research techniques will provide useful
information on the effects of the intervention to be implemented. While the monitoring/operational
research involves the assessment of data collected by target organizations for purposes other than the
evaluation proper (eg. administrative data), the social research involves the collection of data specifically
for the evaluation (to be accomplished via qualitative and quantitative survey). The social research
evaluation instruments are designed in such a way that they would be consistent with administrative data
elements. Evidence from these two techniques is used to triangulate, verify, substantiate and qualify
findings.
This evaluation design is based on the well proven fact that a relationship exists between the health system
and health policymaking process (Gonzalez-Block 2004; Green and Benneth 2007). Therefore any change
in the health policymaking process is most likely to affect the health systems. The aim is to effect a
positive change in the health policymaking process through the promotion of HPSR evidence use, which is
expected to have an impact on the health system. Thus, the findings from the measurement of performance
of the various components of the health system will indicate the extent the interventions have satisfied the
project goals and purposes. This is the main concept of this evaluation design. The specific indicators of
the administrative data/data from management information system (MIS) to be measured are clearly shown
in Table 11.1. These specific indicators are adapted from a WHO previous report (Sadana et al. 2006), and
were modified to capture the requirements of this project. The methods of measurement include document
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 90
reviews, key informant interviews and re-analysis of existing data in line with an earlier report
(Sadana and Pang 2004). One or a combination of these methods is used for measurement depending
on the type of indicator being considered. The measurement strategy includes the use of the likert
scale rating of four options (eg. 1. grossly inadequate, 2. inadequate, 3. fairly adequate, 4. Adequate, 5.
very adequate); yes or no; proportions; percentages; etc. The administrative data/data from MIS is
obtained from the administrative records/data bases of target organizations, although the major target
organization would be the ministry of health. Some of the indicators may not be applicable to all the
target organizations.
From the assessment of data collection system of the organizations, the team should gained an
understanding of the type of relevant data available, data being collected, how and where to find
them, how to collect them, and analyze them in a meaningful way. This information will enhance the
development of the monitoring/operational research as well as the social research mechanisms which
are the strategies designed to accomplish the process evaluation design in a project of this sort.
11.2 Data from monitoring/operational research
Because monitoring/operational research involves the use and analysis of data collected other than for
the evaluation, efforts should be made to sensitize the heads of the organizations and their nominees
on the importance of such data so that proper handling, recording and documentation of the data are
ensured. The four categories of data to be collected which are relevant to the monitoring/operational
research include: (i). administrative data, (ii). data from Management Information Systems (MIS) e.g.
programme database, (iii). performance measurement data about resources (staff and financial), etc.,
(iv). Data from special monitoring exercises e.g. pro forma attached to specific cases to identify stages
and timing. These sources of evidence would be used to triangulate, verify, substantiate and qualify
findings. Emphasis was placed on the administrative data and data from Management Information
Systems because the other two categories can be classified under them.
The use of administrative data and data from MIS is vital to the evaluation phase because they are
data collected in the course of programmatic activities particularly in the Health Ministry for the
purposes of program operation, service provision, or decision-making-essentially; they are not
necessarily collected for research activities. Therefore in consonance with the objectives of this type of
project, the Project Team should concentrate on administrative data and data from MIS supplying the
following information (Table 11.1); (i). Health administration- planning, and administration,
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 91
performance measurement, benchmarking, and best practice promotion. (ii). Service delivery – service
organization and management, access, quality, safety, and continuity of care across health conditions,
across health facilities and over time. (iii) Information and evidence – the generation and strategic use
of information, evidence and research on health and health systems. (iv). Medical products and
technologies – access to essential medical products and technologies of assured quality, safety,
efficacy and cost-effectiveness, and their scientifically sound and cost-effective use. (v). Health
workforce – entry into and exits from the health workforce, distribution and performance of existing
health workers, staff training. (vi). Health financing – health system funding, utilization and resource
allocation. (vii). Population and health interventions – characteristics of people and communities,
health affecting interventions. (viii). Leadership and governance – strategic policy frameworks,
effective oversight and control, coalition-building, regulation, attention to health-system design issues,
promotion of accountability.
11.3 Administering the evaluation questionnaire
The process evaluation questionnaire should be administered to individuals who are directly in charge
of the operations of the health ministry e.g., the permanent secretary, or any other person who is in the
position to provide detailed accurate information on the measurement indicators outlined in Table
11.1. Information derived from completed questionnaire should provide the evaluation outcome.
Table 11.1 Questionnaire for project evaluation from Administrative data/ Data from
management information system.
Category of
Administrative
data/Data from
Management
Information
System
Main data
components/elements
Composite
indicators
Specific indicators/measurement strategy
1. Health
administration
(i). Planning Use of evidence in
planning, past record
use, capacity of
planners or
policymakers
(experience in health
policy, use of
evidence).
(a). existence of a policy on health research
involving all key stakeholders? (*yes/no). (b). are
stakeholders‟ views defined and integrated within a
policy on health research? (yes/no). (c). existence of
a forum or process to coordinate the setting of
health research priorities? (yes/no). (d). extent of
use of research done by others: (gi,in,fa,ad,va)*
(e). extent of use of research initiated/done by the
organization: (gi,in,fa,ad,va). (f). use of data
collected routinely or by survey: (gi,in,fa,ad,va).
(g). relevance of evidence used: (gir,ir,fr,re,vr)*.
(h). proportion of policymakers with following
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 92
qualifications: BSc/BA( ); MSc/MA ( ); PhD ( ).
(i). number of policy documents made by
policymakers in the last 5years ( ).
(ii). Performance
measurement
Number of health
policies made,
functionality of
health programmes,
coverage, public
acceptability, project
outcome
(a). number of health policies made or updated in
the last 5 years ( ). (b). functionality of health
programmes (eg. immunization): % coverage ( );
number of beneficiaries over number of relevant
population ( ); number/types of relevant facilities or
equipment acquired ( ); number of people
trained/employed ( ).
(iii). Bench
marking/best practice
Adherence to
international/national
health policy
standards/guidelines.
(a). availability of ethical unit (yes/no).
(b). availability of document on health research
ethics (yes/no). (c). availability of document on
bench marking/best practice (yes/no). (d). degree of
adherence to guidelines on ethics/ bench
marking/best practice (gi,in,fa,ad,va).
2. Service
delivery
(i). Service
organization and
management
Organizational/mana-
gement structure,
relationship/linkages
to health agencies,
conflicts, service
overlap.
(a). availability of organizational/management
structure (yes/no). (b). any relationship/partnership
with other agencies (yes/no); any counterpart
funding involvement (yes/no); number of MoU ( );
number of letters of collaboration/co-operation ( ).
(c). conflicts-number of reported cases of clash of
duties, tasks, responsibilities per year in the last 5
years ( ). (d). overlap-number of known
organization doing the same task, service/duties ( ).
(ii). Access to health
services
Distribution of health
facilities/services,
affordability,
acceptability,
attendance, hospital
bed occupancy rate.
(a). number of available health facilities-primary
health care centres ( ); secondary health facilities( );
tertiary health facilities ( ). (b). availability of
health intervention programmes (yes/no); if
available what types? (…).
(c). affordability/acceptance-rate of patronage of
available health facilities ( ). (d). hospital bed
occupancy rates in the secondary health facility ( );
in the tertiary health facility ( ).
(iii). Quality of health
services
Manpower
competence, type and
nature of service,
birth rate, death rate,
existence of
surveillance unit.
(a). proportion of doctors ( ), nurses ( ),pharmacists
( ), per health facility. (b). recorded birth rate. (c).
recorded death rate. (d). Availability of functional
disease surveillance unit (yes/no).
(iv). Safety Availability/quality
of safety manuals,
storage/sterilization
facilities.
(a). availability of safety guidelines/manual
(yes/no). (b). availability of storage/sterilization
facilities (yes/no).
(v). Continuity of
health care across
health condition and
across health facilities
Availability of drugs,
consistency of
disease control
programmes
(immunization,
malaria control, HIV
control etc),
availability of regular
field officers.
(a). regularity of drug supply: (gi,in,fa,ad,va). (b).
consistency of disease control programmes:
(gi,in,fa,ad,va). (c). availability and regularity of
health field officers: (gi,in,fa,ad,va).
3. Information (i). Generation and Availability and (a). proportion of policymakers with internet facility
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 93
and evidence strategic use of
information
types of information
technology facilities,
access to media,
educational
materials/research
reports, availability
of record keeping and
storage system, vital
statistics.
in their office ( ). (b). number of different
periodicals subscribed by the organization ( ).
(c). availability of organizational information data
base (yes/no). (d). number of health research output
published per year ( ) (e). availability of quality &
peer review mechanisms (yes/no). (e). existence of
vital statistics (yes/no).
(ii). Evidence and
research on health
systems
Availability and
access to research
findings, number of
research projects
initiated/executed.
(a). number of research projects initiated/executed
in the last 5 years ( ). (b). number of active health
researchers in the organization ( ). (c). number of
different health- related academic/professional
journals subscribed by the organization ( ). (d).
availability of mechanisms to review primary
research outputs (yes/no). (e). number of systematic
reviews produced per year ( ).
4. Medical
products and
technologies
(i). Access to essential
medical products and
technologies
Availability of
essential drugs/basic
medical equipment,
availability/use of
modern technologies.
(a). degree of availability of essential drugs:
(gi,in,fa,ad,va). (b). degree of availability of basic
medical equipment: (gi,in,fa,ad,va).
(ii). Scientifically
sound, safety, efficacy,
and cost effectiveness
Availability of
quality assurance
/control mechanism,
nature of
procurement system.
(a). availability of quality assurance/control
mechanism (yes/no). (b). degree of adherence to
quality assurance/control mechanism:
(gi,in,fa,ad,va). (c). nature of procurement system
(direct/indirect)
5. Health
workforce
(i). Entry into and exit
from health workforce
Employment/retireme
nt/disciplinary
policies.
(a). availability of policy document on
employment/retirement/discipline (yes/no).
(ii). Distribution and
performance of health
worker
Staff posting, staff
appraisal, tenure,
promotion system.
(a). availability of policy document on staff posting/
staff appraisal/ tenure/ promotion (yes/no).
(iii). Staff training Availability/regularit
y of training,
sponsorship
programmes, training
fields/specialties.
(a). availability of training/sponsorship programme
(yes/no). (b). degree of regularity of
training/sponsorship programme: (gi,in,fa,ad,va).
(c). relevance of training fields/specialties:
(gir,ir,fr,re,vr)
(iv). Remuneration
and compensation
Incentives/wages
(salaries, allowances,
benefits), payment
process.
(a). adequacy of incentives/wages: (gi,in,fa,ad,va).
(b). efficiency of payment process: (gi,in,fa,ad,va).
6. Health
financing
(i). Health system
funding
Types/sources/availa
bility of funding, user
fees,
subsidies/waivers,
health insurance
scheme.
(a). sources/proportion of funding: external grants
( ); government subventions ( ); loans ( ); income
( ). (b). amount of resources allocated in
accordance with nationally stated priorities? ( )
(c). availability of health insurance scheme
(yes/no). (d). availability of subsidies/waivers for
service users (yes/no). (e). implementation of user
fees (yes/no)
(ii). Resource
allocation, utilization,
and accountability
Budgetary system,
accounting/audit
system.
(a). existence of budgetary guidelines (yes/no). (b).
extent of adherence to budgetary guidelines:
(gi,in,fa,ad,va). (c). existence of audit unit
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 94
(yes/no). (d). ability of audit unit to identify frauds,
errors and lapses in accounting process:
(gi,in,fa,ad,va).
7. Population
and health
interventions
(i). Characteristics of
people and
communities
Environmental/socio-
demographic/cultural
/economic factors.
(a). availability of records/data on: environmental
factors-environmental temperature (yes/no); rainfall
(yes/no); natural disaster (flooding) (yes/no).
(b). availability of records/data on socio-economic
factors-population distribution (yes/no);
demography (age, gender, occupation, marital
status, religion, literacy level) (yes/no); income
levels (low, middle, high) (yes/no).
(ii). Health affecting
interventions
Epidemiological data,
essential drugs,
public health
services, transport
and other logistics,
infrastructure,
community services.
(a). availability of records /data on-epidemiological
information (yes/no); essential drugs (yes/no);
public health services (yes/no); transport (yes/no);
infrastructure (yes/no); community service (yes/no).
8. Leadership
and governance
(i). Strategic policy
framework
Organogram, conflict
resolution, research
and development.
(a). effectiveness of organizational structure:
(gi,in,fa,ad,va). (b). existence of mechanism for
conflict resolution (yes/no). (c). existence of
programmes/policy on research and development
(yes/no).
(ii). Effective
oversight and control
Use of committees,
regulatory
mechanisms, role of
legislators
(a). existence of boards, ethical committees, and
consultancy for oversight function (yes/no).
(b). extent of adherence to regulatory
provisions/guidelines (yes/no): (gi,in,fa,ad,va).
(c). existence of monitoring and evaluation
activities clearly linked with strengthening health
system (yes/no).
(ii). Coalition-building Internationally/extern
ally initiated
collaborations and
partnerships.
(a). level of collaboration/partnership with external
(international) agencies: (gi,in,fa,ad,va).
(iii). Attention to
health system design
issues
Initiation/updates/rev
ision of health system
framework.
(a). regularity of formulation/revision of health
system framework/policies: (gi,in,fa,ad,va).
(gi,in,fa,ad,va)* gi= grossly inadequate; in= inadequate; fa= fairly adequate; ad= adequate; va= very adequate
(gir,ir,fr,re,vr)* gir= grossly irrelevant; ir= irrelevant; fr= fairly relevant; vr= very relevant
(*yes)= if yes what date did it come into effect?
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 95
CHAPTER 12
CONCLUSION AND RECOMMENDATIONS
Although health policy and systems research HPSR as well as evidence-based policymaking EBP will
appear as new concepts to many policymakers in developing countries, experience in Nigeria has
shown that there was a high level of acceptance of the concepts by all actors in the policymaking
process. A similar trend can be anticipated in any developing country where HPSR and EBP will be
systematically introduced following the procedures outlined in this manual. Throughout the
developing world, particularly under democratic settings, government officials and health
policymakers are under intense pressure to make the health sector functional and deliver democracy
dividends to the public or risk losing popularity. Such pressures have left policy actors with no choice
but to be “open-minded” to any possible solution that can make the health sector more result oriented
and meet the expectation of the populace. Gonzalez-Block and Mills (2003) noted earlier that social
and political pressure towards health system equity and efficiency has stimulated increase in the
prominence of HPSR in low and middle income countries. This is why effective capacity building in
the health sectors of developing countries is essential (Potter and Brough, 2004). There is continuing
trend towards evidence-based policy formation and this has increased the demand for research outputs
that can provide clear, concise policy-relevant findings (WHO 2000b). Thus strengthening research
capacity in developing countries can enhance the generation and translation of knowledge into policy
decisions to improve health systems and increase equitable access to health services in the population
(Pang et al. 2003).
In this manual, as part of the interventional strategies towards improving the ability of the
policymakers and other stakeholders to to acquire, assess, adapt and apply research evidence
effectively, training workshops are recommended. Training workshops have many strategic benefits.
HIFA (2008) noted that workshops (when used as in-service training) are effective in doing what they
are supposed to do: presenting new information to groups of people, practicing new skills and
allowing health workers to share experiences and insights. Furthermore teaching delivered to groups is
often thought to be economical and another advantage is that it is thought that adults learn best by
sharing of experience, discussion and doing. Gates (2002) had earlier indicated that using training
workshop a significant impact on skill difference was gained on providers‟ skills in assessing and
managing HIV/AIDS patients. Methods used in the workshops such as the administration of pre-
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 96
workshop questionnaire and post-workshop questionnaire; focus group discussions; group works and
short presentations have been shown to be very effective in a previous study by Poulos et al.(2007).
Therefore the procedures described in this manual are capable of producing the following outcomes
1. The documentation of the actual situational analysis of HPSR evidence use in policy making (i.e.,
knowledge, attitudes and practices).
2. Increased awareness of the importance and value of HPSR evidence use in health policy making
and practice in a developing economy.
3. Identification of specific capacity constrains and challenges which impede the development of
HPSR evidence use in policymaking.
4. Identification of potential strategies and solutions that would address capacity constrains to HPSR
evidence use in policy making.
5. Identification of critical gaps in HPSR evidence use in policy making, with a focus on improving
public health.
6. Identification of the barriers to, and solutions for, translating research into policy and practice via
evidence use, and the formulation of general recommendations for HPSR collaboration and
coordination.
7. Improved staff skills in the acquisition, assessment, adaptation and application of research
evidence, e.g. the commissioning of research studies or the interpretation of systematic reviews.
8. Acquisition of skills in data processing and computer/information technology.
9. Creation of stronger incentives for evidence use in the health ministry (e.g. through integrating
this dimension into recruitment policies, performance assessments, and staff appraisals).
10. Development and integration of modules on use of evidence into leadership training courses for
target group.
11. Development of special training materials.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 97
References
Acemoglu D (1997). Training and Innovation in an Imperfect Labor Market. Review of Economic
Studies 64:445-464.
Albert MA, Fretheim A, Maïga D (2007). Factors influencing the utilization of research findings by
health policy-makers in a developing country: the selection of Mali's essential medicines. Health
Research and Policy System 5: 2.
Alliance for Health Policy and Systems Research (AHPSR) (2004). Strengthening health systems:
the role and promise of policy and systems research. 130p. Available at:
http://www.who.int/alliance-hpsr/resources/Strengthening_complet.pdf
Alliance for Health Policy and Systems Research (AHPSR) (2007). Sound choices: enhancing
capacity for evidence-informed health policy. World Health Organization.
Bowen S, Zwi AB (2005). Pathways to „evidence-informed‟ policy and practice: a framework for
action. PLoS Medicine, 2(7): e166.
Bhagavan M. (1992). The SAREC model: institutional cooperation and the strengthening of national
research capacity in developing countries. Stockholm SAREC, 1992.
Canadian Health Services Research Foundation (CHRF) (2010). Self-assessment tool (Is research
working for you? A self-assessment tool and discussion guide for health services management
and policy organizations). Available online at:
http://www.chsrf.ca/other_documents/working_e.php.
Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L (2009). Increasing the use of
evidence in health policy: practice and views of policy makers and researchers. Australia and New
Zealand Health Policy 6:21.
Dobrow MJ, Goel V, Upshur REG (2004). Evidence-based health policy: context and utilisation‟,
Social Science & Medicine, 58:207-217.
Gates JD (2002).Comparison of three types of training on providers skills in assessing and managing
HIV/AIDS patients. Int Conf AIDS. 2002 Jul 7-12; 14: abstract no. MoPeB3160.
Giorgi A (1985). Sketch of a psychological phenomenological method. In A. Giorgi, ed.
Phenomenology and psychological research: essays. Pittsburgh, Pa.: Duquesne University Press.
Global HIV/AIDS Initiatives Network (GHAIN) (2008). Guidelines for writing policy briefs
http://communication-resources.wikispaces.com/file/view/Guidelines+for+writing+policybriefs.pdf
Gonzalez Block, MA, Mills A (2003). Assessing capacity for health policy and systems research in
low and middle income countries. Health Research Policy and Systems 1:1.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 98
Gonzalez-Block MA. (2004). Health policy and systems research agendas in developing countries.
Health Research Policy System 5;2(1):6.
Green A, Bennett S (editors) (2007). Sound choices: enhancing capacity for evidence-
informed health policy. World Health Organization, Geneva. p172.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M (2003). The utilization of health research in
policy-making: concepts, examples and methods of assessment. Health Research and Policy System
1:2-29.
Healthcare for all (HIFA) (2008). CHILD2015 Summary: Are workshops effective? Available at: http://www.hifa2015.org/wp-content/uploads/2008/09/training_workshops_are_they_effective.pdf
Huddart J (2005). Assessment Of Ministry Of Health Human Resource Management Policies
And Practices Jordan Human Resources Development Project Report No. 5. Initiatives Inc.
Innvær S, Vist G, Trommald M, Oxman A (2002). Health policy-makers' perceptions of their use of
evidence: a systematic review. Journal of Health Services and Research Policy 7:239-44.
Johnson NA, Lavis JN (2009). Procedures Manual for the "Evaluating Knowledge-Translation
Platforms in Low- and Middle-Income Countries" Study. Hamilton, Canada: McMaster University
Program in Policy Decision-Making, 17 June 2009.
Jones N, Walsh C (2008) Policy briefs as a communication tool for development research
Background Note, Overseas Development Institute: London. Available at:
http://www.odi.org.uk/publications/background-notes/2008/policy-briefs-communication-tool-
development-research.pdf
Kitzinger J (1995). Qualitative Research: Introducing Focus Groups. British Medical Journal 311:
299–302. Retrieved June 23, 2010. <http://www.bmj.com/cgi/content/extract/311/7000/299>.
Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A (2009a). SUPPORT Tools for evidence-
informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-
informed Policymaking. Health Research Policy and Systems, 7(Suppl 1):S13.
Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A (2009b). SUPPORT Tools for evidence-
informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-
informed policymaking. Health Research Policy and System, 7(Suppl 1):S14.
Pang T, Sadana R, Hanney S, Bhutta ZA, Hyder AA, Simon J (2003). Knowledge for 1. better health –
a conceptual framework and foundation for health research systems. Bulletin of World Health
Organization 81:815-820.
Poulos RG, Zwi AB, Lord SR (2007). Towards enhancing national capacity for evidence informed
policy and practice in falls management: a role for a "Translation Task Group"?
Australian and New Zealand Health Policy 31; 4:6.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 99
Potter C & Brough R (2004). Systemic capacity building: a hierarchy of needs. Health Policy
and Planning, 19:336–345.
Rabiee F. 2004. “Focus-group interview and data analysis”. Proceedings of the Nutrition Society 63:
655–660
Richardson CA, Rabiee F (2001). A Question of Access – an exploration of the factors influencing the
health of young males aged 15–19 living in Corby and their use of health care
Services. Health Education Journal 60: 3–6.
Sadana R, Lee-Martin S, Lee J, HRSA Network (2006). Health Research System Analysis
(HRSA) Initiative: Methods for Collecting Benchmarks and Systems Analysis Toolkit. Tool
#1. A brief overview of WHO Health Research System Analysis initiative and an overview of
core indicators and descriptive variables. WHO/EIP/HRSA/06.1. Geneva, World Health
Organization.
Sadana R, Pang T (2004). Current approaches to national health research systems analysis: a brief
overview of the WHO health research system analysis initiative . Ciência & Saúde Coletiva,
9(2):351-362, 2004
Thomas L, MacMillan J, McColl E, Hale C, Bond S (1995). Comparison of focus group and
individual interview methodology in examining patient satisfaction with nursing care”. Social Sciences
in Health 1: 206–219.
Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA, Pielemeier NR, Mills A, Evans T (2004).
Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet
364(9437):900-6.
Uneke CJ, Ogbonna A, Ezeoha A, Oyibo PG, Onwe F, Ngwu BAF & Innovative Health
Research Group (2009). Health System Research and Policy Development in Nigeria: the
challenges and way forward.. The Internet Journal of World Health and Societal Politics 6:2.
Uneke CJ, Ngwu BAF, Ogbonna A, Ezeoha A, Oyibo PG, Onwe F (2010). Strategies to
enhance the capacity for evidence-informed health policymaking in Nigeria. The Internet Journal of
Healthcare Administration. 2010 Volume 7 Number 1
United Nations Development Programme (UNDP). (2006). Capacity development practice note. New
York, United Nations Development Programme. Available at: http://content.undp.org/go/cms-
service/download/asset/?asset_id=1654154
World Health Organization (WHO) (2000a). The World health report 2000: health systems: improving
performance. Geneva: World Health Organization.
World Health Organization (WHO) WHO (2000b): Promoting Evidence-based Reproductive Health.
Progress In Reproductive Health Research, No. 54, Part 1.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 100
World Health Organization (WHO). 2003. Global Programme on Evidence for Health Policy.
Guidelines for WHO Guidelines. EIP/GPE/EQC/2003.1. Geneva: World Health Organization.
World Health Assembly (WHA). 2005. World Health Assembly concludes: adopts key resolutions
affecting global public health. Fifty-eighth Session. Available at: http://www.who.int/mediacentre/news/releases/2005/pr_wha06/en/print.html
World Health Organization (WHO) (2007) Everybody‟s business: strengthening health systems to
improve health outcomes. WHO‟s Framework for Action. Geneva, World Health Organization.
World Health Organization (WHO). (2008a). Report on meeting on health systems strengthening and
primary health care. Report Series No.: RS/2008/GE/35(PHL). Regional Office for the Western
Pacific Manila, Philippines: World Health Organization.
Health Policy & Systems Research Project
Operational Manual for Strengthening Capacity to Employ Evidence in Policymaking Page 101
Knowledge
Translation
Platform
Health Policy & Systems Research Project
Ebonyi State University PMB 053 Abakaliki Nigeria
Copyright 2010