Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear American Evaluation...

Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear

American Evaluation Association ConferenceNovember 5, 2011Gretchen B. Jordan

Sandia National Laboratories [email protected]

Parts of work presented here was completed for the U.S. DOE by Sandia National Laboratories, Albuquerque, New Mexico. Opinions expressed are solely those of the author.

G. Jordan AEA 2011 2

This talk will demonstrate

• A complex, one page logic model for the return on investment for health sciences research

• With several features to frame assessments that are similar enough to feed evaluation synthesis

• Built on agreed upon theory and definitions to specify aspects of both the healthcare system and the environment that affects health and individual behaviors;

• Categorized institutions and actors that are the pathways through which advances get to impact; and

• Individual studies look at on pathways to impact, noting other parts of the logic as context.


Background and Charge

• This logic model was part of a Major Assessment Panel led by the Canadian Academy of Health Sciences (CAHS) to develop a “Preferred Framework and Indicators to Measure Returns on Investment in Health Research.”

• Twenty-three different organizations sponsored• Charge to the Expert Panel: Develop a framework or model at

the outset of the assessment and demonstrate that the model fulfills the required criteria: – be useful to a range of funders/research types, – be compatible with what is already in place in Canada,– be transferable to international comparisons, and – be able to identify the full spectrum of potential impacts.


Model Design• For Canada’s four “pillars of health research” and cross cuts• Impacts at various levels — individual, institutional, provincial,

national or international• Track health-research impacts in five main categories:

– advancing knowledge – building capacity, – informing decision-making, – health impacts, and – broad socio-economic impacts.

• Subcategories identify and partition evaluation methods• A menu of indicators in each sub category• Intermediate outcomes and (eventually) quantify the value(s) of

impacts to end-users as a function of dollars invested


Model relies on evidence AND is potentially hypothesis generating

• The model built on academic and practical studies of– the determinants of health: World Health Organization paper (2006);

Evans and Stoddart (1990) and Dahlgren and Whitehead (1991) – diffusion of innovation theories: Everett Rogers (1995), Reed and

Jordan (2007) for the five groups influenced by R&D

• Synthesis across matched studies can test hypotheses and build theory– ease of synthesis enhanced if studies use similar terminology, good

research design, and make clear the full context– a single study cannot be large enough in scope; differences in context

are informative; points to features of an intervention that matter most– synthesis also shows where there are gaps in knowledge


•Topic Identification•Selection•Inputs•Process D

isse

min

ation

Secondary Outputs

Adoption Final Outcomes

PAYB

ACK

FR

AMEW

ORK

Prim

ary

Out

puts

Economic and Social

Prosperity

Improvements in Health and Well-

being

Initiation and Diffusion of Health Research Impacts

Impacts feed back into inputs for future research

Health Industry

Public Information,

Groups

OtherIndustries

Government

Research Agenda

Determinants of Health

Health care Appropriateness,

Access, etc.

Prevention and Treatment

Canadian Health Research•Biomedical•Clinical•Health Services•Population and Public Health•Cross-Pillar Research

Know

ledg

e Po

olHealth Status, Function, Well-being, Economic Conditions

Global Research

Research Capacity

Inte

racti

ons/

Feed

back

Re

sea

rch

R

esu

lts

CAHS Logic for Return on Investment to Health Research


Health Industry-Products/drugs-Services, databases-Practitioners’ behaviour-Clinical/manager’s guidelines-Institutional policies -Social care practices

Government (multiple levels) -Resource allocation-Regulation-Policy -Intervention programs-Taxes and subsidies

Determinants of health-Personal behaviour-Social/cultural determinants-Environmental determinants-Living and working conditions

Public Information, Groups - Advocacy groups- Media coverage- General knowledge - Confidence in data

Health care -Appropriateness-Acceptability-Accessibility-Competence -Continuity-Effectiveness-Safety

Occur through prevention and treatmentFor disease, illness, injury, or progressive condition-Prevention-Diagnosis/prognosis-Treatment/palliation-Post-treatment

Research Capacity-Increased understanding-Methodological advances-Larger, more comprehensive data sets-Human capital (absorptive capacity)-Student and faculty career paths-Reputation-Research revenues-Cross-fertilization of ideas/research -Education curriculum

Research Decision Making-R&D agendas/investment (industry/gov’t/foundations) -Identify issues, gaps-Evidence problems are being addressed-Tackle harder problems

OtherIndustries

-Products/services-Built infrastructure-Work environment

Economic and social prosperity

Improvements in health and well-being (disease prevalence and

burden)

External Influences:Interests, Traditions Technical limitations,Political dynamics

Canadian Health Research•Biomedical•Clinical•Health services•Population and public health•Cross-pillar research

Know

ledg

e Po

ol

Health status and function, well-being, economic conditions

Global Research

Cons

ulta

tion/

Col

labo

ratio

ns

Rese

arch

Res

ults

That influence decision making in… That affect healthcare, health risk factors, and other health determinants

Research activity That produces results

That contribute to changing health, well -being and economic and social prosperity

CAHS LOGIC in more detail


Applying the model to different research pillars

• By cross-mapping impact categories for each pillar, we can create a specialized framework to show where the majority of impacts are likely to fall within that pillar.

• Intent is to show where most impacts are likely to be found and thus where indicators or metrics for those impacts should be concentrated.

• However, the areas of impact for any one research pillar are not exclusive of impacts in other domains.


Biomedical Research

Canadian Health Research•Biomedical•Clinical•Health Services•Population and Public Health•Cross-pillar Research

Health Industry

Economic and Social

ProsperityDeterminants of Health

Public Information,

Groups

Know

ledg

e Po

ol


being

Healthcare Appropriateness,

Access, etc.


Health Status, Function, Well-being, Economic Conditions


Global Research

Research Capacity


Government

Research Agenda

Inte

racti

ons/

Feed

back

Re

sea

rch

R

esu

lts

OtherIndustries

Advancing Knowledge

Capacity Building

Informing Decision Making

Health Benefits

Economic Benefits

A

Population and Public Health Research

Canadian Health Research•Biomedical•Clinical•Health Services•Population and Public Health•Cross-pillar Research

Health Industry

Economic and Social

ProsperityDeterminants of Health

Public Information,

Groups

Know

ledg

e Po

ol


being

Healthcare Appropriateness,

Access, etc.


Health Status, Function, Well-being, Economic Conditions


Global Research

Research Capacity


Government

Research Agenda

Inte

racti

ons/

Feed

back

Re

sea

rch

R

esu

lts

OtherIndustries

Advancing Knowledge

Capacity Building

Informing Decision Making

Health Benefits

Economic Benefits

D


Summary and Conclusion• The logic model was a definite asset for categorizing indicators

and ensuring that important indicators were not overlooked. • The framework has been used for impact assessments by at least

three organizations,– Alberta Innovates Health Solutions– Catalan Agency of Health Information, Assessment and Quality in Spain– Canadian Cancer Society Research Institute (CCS RI)

• Some organizations continue to use the Payback Model• It remains to be seen if it is possible to copy this logic and menu of

indicators in other areas of research– The groups to influence are similar– But the common language and indicators for the intermediate outcomes

(“magic in the middle”) are less well defined outside of health area.


Thank you for your attention.

• For a full copy of the assessment report see www.afmc.ca/pdf/ROI_FullReport.pdf

• My contact information after December 15, 2011 [email protected]

http://www.afmc.ca/pdf/ROI_FullReport.pdf





Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear American Evaluation...

Documents

Transcript of Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear American Evaluation...