Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC...
-
Upload
peregrine-allan-greene -
Category
Documents
-
view
220 -
download
2
Transcript of Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC...
![Page 1: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/1.jpg)
Process Evaluation: Considerations and
Strategies
CHSC 433CHSC 433
Module 4/Chapter 8Module 4/Chapter 8L. Michele Issel
UIC School of Public Health
![Page 2: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/2.jpg)
Making Widgets
Count not only the widgets,
but
who gets the widgets and
what goes into making the widgets.
![Page 3: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/3.jpg)
Definitions
Systematic examination of coverage
and delivery
Measuring inputs into the program
Finding out if the program has all its
parts, if the parts are functional and
operational
![Page 4: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/4.jpg)
Program Components
Discrete interventions, or groups of
interventions, of the overall program
that are designed to independently
or synergistically effect recipients.
Objectives per component
![Page 5: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/5.jpg)
Types of Implementation Evaluations
Effort: quantity of input Monitoring: use of MIS information Process: internal dynamics, strengths
and weaknesses Component: assess distinct program
parts Treatment: what was supposed to have
an effect
![Page 6: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/6.jpg)
Purpose of Process Evaluation
Assurance and accountability
Understanding of outcomes
Mid-course corrections
Program replicability
![Page 7: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/7.jpg)
From Perspective of:
Evaluator
Funders (Accountability)
Management for the program
![Page 8: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/8.jpg)
Stakeholders and Expectations
Focus on the explicit• Objectives• Program descriptions
Uncover the implicit• Review program theory• Review objectives• Role play possible outcomes
![Page 9: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/9.jpg)
Program Theory Components
Program Theory
Impact TheoryProcess Theory
Service Utilization PlanOrganizational Plan
![Page 10: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/10.jpg)
Organizational Plan
How to garner, configure, and
deploy resources, organize
program activities so that the
intended service is developed and
maintained
![Page 11: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/11.jpg)
Service Utilization Plan
How the intended target population
receives the intended amount of
the intended intervention through
interaction with the program’s
service delivery system
![Page 12: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/12.jpg)
Rigor in Process Evaluation
Appropriateness of the method
Sampling strategy
Validity and Reliability
Timing of data collection
![Page 13: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/13.jpg)
Key Decision: 1
How much effort to expend ~
what data are needed to accurately describe the program.
Choose based on:• Expected across site variation
• Making report credible
• Literature about the intervention
![Page 14: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/14.jpg)
Key Decision: 2
What to look for ~ what program features are most critical,
valuable to describe. Choose based on:• What is most often cited in program
proposal• The budget• What may be related to program failure
![Page 15: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/15.jpg)
Go Back to Objectives
Process objectives per program component:• How much• Of what• Will be done• By who• By when
![Page 16: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/16.jpg)
Components and Objectives
Goal
OutcomeObjective
1
Component Awith ProcessObjectives
OutcomeObjective
2
OutcomeObjective
3
Outcome I
Outcome II
Component Bwith ProcessObjectives
Component Cwith ProcessObjectives
![Page 17: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/17.jpg)
Possible Foci of Process Evaluation
Place: Site, Program People
• Practitioner/provider• Recipient/participant
Processes• Activities• Processes• Structure
Policy
![Page 18: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/18.jpg)
Levels of Analysis
Individuals• Program participants• Program providers
Programs• As a whole
Geographic locations• Regions and state
![Page 19: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/19.jpg)
Types of Questions ?
What was done and by whom? How well was it done and how much
was done? What contributed to success/failure? How much of what resources were
used? Is there program drift?
![Page 20: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/20.jpg)
Sources of Program Variability
Staff preferences and interest Materials availability and
appropriateness Participants expectations, receptivity,
etc Site physical environment and
organizational support
![Page 21: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/21.jpg)
Roots of Program Failure
ProgramInterventions
ProgramInterventions
ProgramInterventions
CausalProcess
Desired Effect
CausalProcess
CausalProcess
Desired Effect
Desired Effectdid not set intomotion
did not leadto
set into
motion
which led to
which would
have led to
SuccessfulProgram
Theory Failure
ProcessFailure
set into
motion
![Page 22: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/22.jpg)
Causes of Program Failure
Non-Program• No participants• No program done
Wrong intervention• Not appropriate for the problem
Unstandardized intervention• Across site, within program variations
![Page 23: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/23.jpg)
Program Failure cont
Mis-management of program operations
Wrong recipients
Barriers to the program
Program components unevenly delivered, monitored
![Page 24: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/24.jpg)
Data Sources from Program
Resources used Participant provided data
• Quality• Match with process evaluation
Existing records• Sampling of records• Validity and reliability issues
![Page 25: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/25.jpg)
Data Sources from Evaluator
Surveys and Interview of
Participants
Observation of Interactions
Survey and Interview staff
![Page 26: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/26.jpg)
Evaluating Structural Inputs
Organizational structure• Supervision of staff• Place in organizational hierarchy
Facilities, equipment Human resources
• Leadership• Training
![Page 27: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/27.jpg)
Measures of Delivery
Measures of program delivery
Measures of coverage
Measures of effectiveness
![Page 28: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/28.jpg)
Measures of Implementation
Measures of Volume (Outputs):• Number of services provided
Measures of Workflow:• Client time• Staff work time
![Page 29: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/29.jpg)
Targets, Recipients, and Coverage
TARGETAUDIENCE
RECIPIENTS
NON-RECIPIENTS
IN NEED
NOT INNEED
NOT INNEED
IN NEED
![Page 30: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/30.jpg)
Measures of Coverage
Undercoverage= # recipients in need /# in need
Overcoverage=# recipients not in need /# recipients
Coverage efficiency=(under - over) x 100
![Page 31: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/31.jpg)
Measures of Effectiveness
Effectiveness Index =% reached per program standard per
program component
Program Effectiveness Index =Sum of Effectiveness Indexes/#
program components
![Page 32: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/32.jpg)
Bias in Participation
due to self-selection results in under or overcoverage may be related to recruitment can be identified with good data
collection (monitoring)
![Page 33: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/33.jpg)
Measures of Efficiency
ratio of input per output productivity per staff, per cost, per
hour cost per participant, per
intervention etc...
![Page 34: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/34.jpg)
Evaluating Costs
Payments by agency Payments by secondary funders Payments by participants
versus charges!
![Page 35: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/35.jpg)
Monitoring and CQI
Similar types of data presentation• Control charts• Fishbone diagrams• Flow charts• Gantt charts• etc.
Overlapping purposes
![Page 36: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/36.jpg)
Reaching Conclusions
Compare data to objectives
Compare data to needs assessment data
Compare data to other sites or other programs
![Page 37: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/37.jpg)
Worksheet Exercise
For each program objective:• What is the focus and level of the
process evaluation• What data sources needed• Who collects data
![Page 38: Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health.](https://reader038.fdocuments.us/reader038/viewer/2022110320/56649cd85503460f949a1210/html5/thumbnails/38.jpg)
References
Rossi, Freeman & Lipsey (1999). Evaluation: A systematic approach. Sage Publications
Patton (1997). Utilization focused evaluation. Sage Publications.
King, Morris, Fitz-Gibbon (1987). How to assess program implementation. Sage Publications.
Weiss (1972). Evaluation Research. Prentice Hall