Practical Steps to Successful Operational and Implementation Research Jim Foreit The Population...

Practical Steps to Successful Operational and

Implementation Research

Jim ForeitThe Population Council

[email protected]

George SchmidWorld Health Organization

[email protected]

Doing Operations and Implementation Research

Format • Informal• Practical—if you want to hear about modeling (a legitimate,

important part of OR), run out the door now• The session is for you, and us• You will forget 90% of what you hear today. So, we've given you

materials on four CD-ROMs:– Designing HIV/AIDS Intervention Studies (and other materials) (Pop Council);– Training course materials in operations research (Pop Council)– Framework for Operations and Implementation Research in Health and

Disease Control Programmes (and other materials) (Global Fund and WHO)– Other materials (note the FHI guide to qualitative research)

Goals• We hope you will remember:

– OR is not a mystery– OR need not be difficult– You can—and should—do OR in every programme

when you encounter a problem you cannot solve by experience or common sense

What is OR? What is Implementation Research? What is (Public Health)

Evaluation? What is Translation Research?

OR

PHE

IR

TR

What is OR? What is Implementation Research?

Programme

Not implemented(or just starting)

Implemented

Effective? Efficient?

IR OR


• "…focuses on how to promote the uptake and successful implementation of evidence-based interventions…that have been identified through systematic reviews."1

1Sanders D, Haines A. PLoS Med 2006;3:e186


• "…focuses on how to promote the uptake and successful implementation of evidence-based interventions…that have been identified through systematic reviews."1

1Sanders D, Haines A. PLoS Med 2006;3:e186

Who does this?

What is OR? What is Implementation Research?

Programme

Not implemented(or just started)

Implemented

Effective? Efficient?

IR OR

What is the International Climate Towards OR?

• The Sydney Declaration (IAS—July 2007)– 10% of HIV programme funds for research

• The Global Fund suggests 5-10% of funds for M&E, including OR

Favorable!

Global Fund Grants and "OR and Information Systems"

Rounds 1-6

0

50

100

150

200

250

300

350

400

Rounds1-5

Round 6

Grants

Grants with ORcomponent

Grants with ORbudget

If OR budget, %19%

52%

2.6%

4.4%

Korenromp E et al 5th European Congress on Tropical Medicine and International Health

34% HIV

56% TB

74% Malaria

What is the International Climate Towards OR?

• The Sydney Declaration (IAS—July 2007)– 10% of HIV programme funds for research

• The Global Fund suggests 5-10% of funds for M&E, including OR

• PEPFAR February 2008 • WHO/World Bank/IAS/GF March 2008

meeting and 4 August Consensus Statement– Extends Sydney Declaration into capacity building

and implementation capability• Here: IAS session, ANRS session

Favorable!

What Skills Do You Need to Do OR?

• An enquiring mind• Effective (epidemiology, statistics, public

health, relevant science expertise to the project, e.g., behavioural science, medicine….)

• Efficient (economics, health services management….)

• Courses in OR– CDC– Population Council…and….

Format of Session

• This introduction

• What is OR?

• Some practical steps in OR

• Is it formative or operational research?

• Your thoughts/discussion

Questions!

What Is HIV/AIDS Operations Research?

2008 International AIDS Conference

Operations Research for Managers of Reproductive Health Programs© 2008 The Population Council, Inc.


Operations Research (OR) is the study of factors under the control of program managers.

OR uses research techniques to help choose among alternative uses of resources to meet program objectives.


Research Focus

Independent variables:

manipulated by managers

Dependent variables:

program outcomes

Independent variables:

manipulated by managers

Dependent variables:

program outcomes

Factors controlled by managers Outcomes desired by managers and clients


Managerial Control

Factors Under Control

Program systems: training, information, counseling …

Factors Not Under Control

Cultural beliefs, location, economic status, religion, education…


OR Supports Evidence-Based Programs

• Allows programmes to make evidence-based decisions

• Identifies service delivery problems

• Tests service delivery innovations (effectiveness)

• Tests service delivery costs (efficiency)


Costs are a Key Program FactorStudy Them!

• Cost-effectiveness—what is the least-expensive way to get the desired outcome– e.g., can nurses do the same job as a

doctor?

• Affordability


Research Techniques

• Systematic data collection

• Qualitative techniques

• Quantitative techniques

• Surveys, experiments, focus groups….


The OR Process

• Problem identification

• Solution generation

• Solution testing

• Results dissemination

• Results utilization


OR Requires Collaboration

• Managers: responsible for decision parameters and desired outcome

• Researchers: responsible for recommending and implementing research techniques

Taking My Problem Into Operational Research

Q1.What Proportion of Individuals Potentially Helped by Bed Nets are

Using Them?

1. 2%

2. 7%

3. 14%

4. 24%

Morel CM et al. BMJ 2005

Taking My Problem Into Operational Research

First, You Must Recognize a Problem

• Wanting to know

• Wanting to act

• Having the ability to act

Q2. Who Typically Identifies Problems for OR? (more than one answer

possible)

1. The patient

2. The programme staff

3. The programme manager

4. The District manager

5. National staff

Zambia Example

• Problem: TB patients are not being tested for HIV

• Observation: "TB Corners" where TB patients are seen are in open hallways

• Intervention: Provide counselling with privacy

As Jim said, some issues simply need some common sense, and initiative

Identifying Problems

• The role of anyone who works in the programme!

• But, primarily, the role of the manager

• Identifying problems requires a team approach– Managers must be present in their programmes– Managers must have good relationships with

their staff

You've Identified a Potential Problem

• Patients may not be taking their ART• People may not be using condoms• Patients may not be returning for follow-up visits• Your health care workers seem to have high

rates of absenteeism• ….

The OR Process

• Problem identification

• Solution generation

• Solution testing

• Results dissemination

• Results utilization

There are Lots of Ways to Approach OR

• Some are simple, others more formal

• Flexibility needed and desired, to meet the objectives of what you need to do

• But, let's talk about more formal approaches now….

The Menu of the WHO/GF OR Book

1. Research team

2. Determine Issues

3. Develop proposal

4. Ethical clearance

5. Funding

6. Budget

7. Capacity building

8. Monitor project9. Pre-test10. Quality control11. Stakeholder discussions12. Dissemination plan13. Disseminate results14. Document changes15. Monitor changes16. Consider further

improvement

WHO (TDR)/GF (Jane Kengeya-Kayondo, George Shakarishvili, Serge Xueref) and Bill Brieger and Amy Ellis of Johns Hopkins School of Public Health

Three Phases to OR

1. Planning

2. Implementation

3. Follow Through

Planning1. Organize the Research Group

• What disciplines do you need?– Epidemiology, behavioural science…?– Statistics?– Economics (It is difficult to retroactively do the economics

part of a study)?• Structure• Assign tasks, with deadlines• Meet regularly• Give credit!!!!!!Give credit!!!!!!

– Who had the research idea?– Builds teamwork among departments/persons and avoids

hard feelings– Pays off in the future

Planning2. Determine the research questions (and objectives)• Be clear, be specific• Research question—What do you want to know?

– Observation• "Our health care workers seem to have high rates of

absenteeism"– Research question

• "What is the rate of absenteeism among our nurses in the Kiev city hospital, and the reasons for it?"

• Objective– Not: To study adherence – But: To determine rates of adherence among attendees at the

XX hospital and reasons for adherence and nonadherence

Planning3. Develop a research proposal/protocol• A document that details:

1. What the research is about and why it is important

2. How the study will be conducted

3. How the results will be used

Planning3. Develop a research proposal/protocol—"standard parts"• Research objectives and questions• Background

– Literature review– Local context

• Research team• Methods (study design, study population,

sampling details, how data will be collected, data management, quality assurance)

• Data collection instruments• Plans for use of data• Budget

Planning3. Develop a research protocol

• Does every "formal" OR project need a research protocol?– Yes– Makes you clearly state what you will do

(and helps to improve your thinking)– Gives you the plan that you will follow, so

everyone knows what will be done, and should be done

Q3. Does Every OR Proposal need to go through Ethical Review?

1. Yes

2. No

Planning4. Obtain ethical clearance

Does Every Study Need to Go Through IRB (Ethics Board)?

• No…is the project "research," and, does it involve human subjects?

• What is research?– "A systematic investigation (i.e. gathering and analysis of information) designed to

develop or contribute to generalizable knowledge"1

– "…a class of activities designed to develop or contribute to generalizable knowledge."2

• What are human subjects?– "An individual about whom an investigator… conducting research obtains

• data through intervention or interaction with the individual or• identifiable private information"1

1U.S. Office of Research Integrity 2International Ethical Guidelines Biomedical Research. Council International Organizations Medical Sciences. www.cioms.ch

Is It Research Involving Human Subjects?

Yes NoStudying Task Shifting for IMAI Reviewing clinic waiting

times records

Who makes the determination of whether you need to go to the ethics board?

Can use expedited review!

Role of the ethics board

Q4. What is the most common thing wrong with consent forms?

1. Do not contain the contact information for the local ethics board

2. Cannot be understood by potential participants

3. Do not mention the word "research"

4. All risks are not revealed

5. Give the patient too much information

Implementation8. Pre-test all research materials• Questionnaires

– Is the wording clear? Do people understand what you want? Do the questions ask what they are supposed to?

– Test in 5-20 persons• Focus groups

– Are the guides clear?– Hold one or two focus groups.

• Consent forms– Do people understand them? Language level? (have

your child read it) (sometimes, to check understanding during study, develop a series of 5 or so questions—patient must answer 4 or 5 to enter study)

Q5. Who Need Not Be Part of the Pre-Test Evaluation of Materials?

1. Potential participants2. Staff who use the materials, e.g.,

interviewers, focus group leaders3. Data management staff4. The members of the research team

developing the materials5. All must be

Implementation9. Quality Assurance

• What?

• During implementation

• Throughout the study

• A quality assurance plan should be part of the protocol

If you do not perform the study exactly as you said you would, you cannot trust the results

Dissemination13. Disseminate the results and recommendations

• If you don't do this, you might as well not have done the study

• Identify the audiences

• Create the right documents/strategy/medium for the right audience

Q6. Which Format of Information Sharing Do Decision-makers Like

Least (Industrialized Country)

1. Internet/e-mail

2. Meetings/conferences

3. Colleagues

4. Journal articles

5. Short summaries

Dobbins M et al. Implementation Sci 2007

Q7. Who Typically Acts on Identified Problems?

1. The patient




5. The National staff

6. Too few people act

Four Things Most Commonly Wrong with Proposals

1. The study cannot achieve its objectives

2. Lack of clarity and detail

3. Inconsistency

4. Poor consent forms– Do not contain all needed information– Too complicated and not understandable

Q1.What Proportion of Individuals Potentially Helped by Bed Nets are

Using Them?

1. 2%

2. 7%

3. 14%

4. 24%

Morel CM et al. BMJ 2005

Q2. Who Typically Identifies Problems for OR? (more than one answer

possible)

1. The patient




5. National staff

Q3. Does Every OR Proposal need to go through Ethical Review?

1. Yes

2. No

Q4. What is the most common thing wrong with consent forms?

1. Do not contain the contact information for the local ethics board

2. Cannot be understood by potential participants

3. Do not mention the word "research"

4. All risks are not revealed

5. Give the patient too much information

Q5. Who Need Not Be Part of the Pre-Test Evaluation of Materials?

1. Potential participants2. Staff who use the materials, e.g.,

interviewers, focus group leaders3. Data management staff4. The members of the research team

developing the materials5. All must be

Q6. Which Format of Information Sharing Do Decision-makers Like

Least (Industrialized Country)

1. Internet/e-mail

2. Meetings/conferences

3. Colleagues

4. Journal articles

5. Short summaries

Dobbins M et al. Implementation Sci 2007

Q7. Who Typically Acts on Identified Problems?

1. The patient




5. The National staff

6. Too few people act

Discover if you are a “natural” at doing Operational Research

and win a valuable prize!

International AIDS Conference 2008

Operational Research

• Operational Research improves programs by testing new service delivery approaches to improve program outcomes

Service delivery approach = factor under control of manager Program outcomes = important success indicators

OR research question: Will having CHWs visit the homes of newly delivered mothers with HIV+ infants result in more Neviripine treatment within 72 hours?

Formative Research•Formative Research identifies health or program problems that need correction. Unlike operations research it does not test solutions to the problem

Health problem = The HIV prevalence rate is 5%Program problem = HIV prevalence is three times as high in province X than in province Y, but we have 90% of our program resources in province Y!

For all you health workers: Formative research diagnosis program problems, operations research treats program problems

Operational? Formative? Neither?

Examples

Title: “ Diversification of HIV-1 groups M and O”

Before answering: Is this a program problem? Can we use the study to identify a program problem Can we use the study to solve a program problem?


• Title 1: “ A comparison of the effectiveness of two approaches in recruiting men for adult circumcision” (Question 8)

• Title 2: “Differences in the quality of treatment provided MSM and other AIDS patients in an Indian hospital” (Question 9)


• Title 3: “HIV prevalence rates among the elderly in rural Illinois” (Question 10)

• Title 4: “The impact of three types of supervision on frequency of client counseling and testing by clinic workers” (Question 11)


• Title 5: “Survey of acceptability of male circumcision among mothers in Mysore India” (Question 12)

• Title 6: “Efficacy of an HIV prevention curriculum in South Africa: Results of a pilot study” (Question 13)

Practical Steps to Successful Operational and Implementation Research Jim Foreit The Population...

Documents

Transcript of Practical Steps to Successful Operational and Implementation Research Jim Foreit The Population...