Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the...
Transcript of Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the...
![Page 1: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/1.jpg)
Measuring and Communicating
the Impact of your
Coalition and Programs
Brian Bumbarger
Coalition Leadership Institute
June 2010
![Page 2: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/2.jpg)
If you don’t know where you’re going,
any road will get you there!
The Cheshire Cat, Alice in Wonderland
![Page 3: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/3.jpg)
It is not enough to be busy. So are the ants. The question is: What are we busy about?
Henry David Thoreau
![Page 4: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/4.jpg)
Don’t mistake movement for progress.
They might feel the same, but just shaking your butt won’t get you to
Disney World
Brian Bumbarger
![Page 5: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/5.jpg)
• Create community-level outcomes.
• Investigate potential programs, policies and practices.
• Select programs, policies and practices.
• Create program-level outcomes.
• Plan to evaluate.
• Create a written Community Action Plan.
![Page 6: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/6.jpg)
Evaluate (v.): to determine the significance,
worth, or condition of usually by careful
appraisal and study
Assess (v.) : to determine the importance,
size, or value of
![Page 7: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/7.jpg)
Pre-test Post-testPreventionProgram
We can compare the difference between pre-test and post-testBut…
• What if something else caused the change?
• If the post-test isn’t better, does that mean the program didn’t work?
• What would have happened in the absence of the program?
• Are these kids representative of all kids?
![Page 8: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/8.jpg)
Pre-test Post-testPreventionProgram
![Page 9: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/9.jpg)
Post-testPre-testPreventionProgram
Randomization
Pre-test PreventionProgram Post-test
Random
Se l ect i on
Fo l l owup
Data
![Page 10: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/10.jpg)
• Create community-level outcomes.
• Investigate potential programs, policies and practices.
• Select programs, policies and practices.
• Create program-level outcomes.
• Plan to evaluate.
• Create a written Community Action Plan.
![Page 11: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/11.jpg)
• Gives clear direction toward achieving community vision
• Provides built-in evaluation measures and accountability
• Required by many grant makers
![Page 12: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/12.jpg)
![Page 13: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/13.jpg)
(10 to 15 years)
• Increase in protective factors
• Decrease in risk factors
• Decrease in problem behaviors
• Changes in participant knowledge, attitudes, skills or behavior
• Program implementation fidelity
(3 to 10 years) (6 months to 2 years)(1 to 4 years)
![Page 14: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/14.jpg)
![Page 15: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/15.jpg)
![Page 16: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/16.jpg)
The Community Board will learn how to develop outcome-focused prevention plans.
![Page 17: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/17.jpg)
Participants will be able to:
1. Define outcome-focused planning.
2. Write desired outcomes:
– Behavior outcomes
– Risk-factor outcomes
– Protective-factor outcomes
3. Prepare for community evaluation.
![Page 18: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/18.jpg)
• Measuring changes
• Measuring progress
• Knowing what actions will take place
![Page 19: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/19.jpg)
1. Change in what
2. As measured by
3. The baseline or starting point for comparison
4. By how much, by when
![Page 20: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/20.jpg)
Community Plan Implementation Training 1-<#>
1. Priority risk-factor outcome to change
2. Indicator used to measure the outcome
3. Baseline data
4. How much change, by when
![Page 21: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/21.jpg)
Community Plan Implementation Training 1-<#>
• Identify targets for prevention-planning efforts.
• Evaluate the impact of prevention efforts over time.
• Engage Key Leaders and community members in supporting prevention.
![Page 22: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/22.jpg)
• Know the data to be collected.
• Know who is responsible for data collection.
• Know how outcomes will be reported.
• Determine what resources will be needed for future assessments.
![Page 23: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/23.jpg)
Community Plan Implementation Training 1-<#>
![Page 24: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/24.jpg)
Participants will learn how to develop participant and implementation outcomes in preparation for implementing and evaluatingtheir selected programs,policies and practices.
![Page 25: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/25.jpg)
Participants will be able to:
1. Develop participant outcomes.
2. Develop implementation outcomes.
3. Identify the elements of implementation.
![Page 26: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/26.jpg)
• Participant outcomes measure the changes a program produces.
• Implementation outcomes measure the process by which a program produces desired changes.
![Page 27: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/27.jpg)
• Knowledge
• Attitudes
• Skills
• Behavior.
![Page 28: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/28.jpg)
• Who the program will be delivered by
• When the program will be delivered, including how often and how long
• Where the program will be delivered
• How the program will be delivered
• Number of people to be affected by the program
• Who your target audience will be.
![Page 29: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/29.jpg)
Community Plan Implementation Training 1-<#>
![Page 30: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/30.jpg)
Participants will be able to:
1. Identify the uses of evaluation.
2. Explain basic evaluation concepts.
3. Assess the need for an expert evaluator.
![Page 31: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/31.jpg)
Evaluation helps you:
• monitor implementation fidelity
• monitor progress toward outcomes
• identify problems with program design or selection
• demonstrate achievements
• determine cost-effectiveness.
![Page 32: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/32.jpg)
• What do you want to evaluate?
• What is the purpose of your evaluation?
![Page 33: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/33.jpg)
• Community Board members
• Key Leaders
• Program implementers and site administrators
• Media
• Local interest groups
• Other community members
• Sources of funding
![Page 34: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/34.jpg)
• Monitor implementation fidelity.
• Identify and correct implementation problems before they result in program failure.
![Page 35: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/35.jpg)
• Records
• Activity logs
• Observation
• Questionnaires
![Page 36: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/36.jpg)
• Measure the program’s impact on participant knowledge, skills, attitudes or behavior.
• Monitor progress toward the community’s vision.
![Page 37: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/37.jpg)
• Observation
• Interviews
• Questionnaires
• Records
![Page 38: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/38.jpg)
An expert evaluator can help:
• select appropriate evaluation design
• develop evaluation measures
• analyze statistical information
• interpret results.
![Page 39: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/39.jpg)
• The program requires significant technical assistance
• Evaluation tools will need to be developed
• Data collection will be expensive and complex
• Complex data analysis will be required
• The Board lacks the expertise to complete the evaluation.
![Page 40: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/40.jpg)
• Local universities
• Research and consulting firms
![Page 41: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/41.jpg)
• The nature of the evaluation
• Who will be doing the evaluation
• Resources needed for the evaluation.
![Page 42: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/42.jpg)
Community Plan Implementation Training 3-42
![Page 43: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/43.jpg)
To help ensure that programs, policies and practices in the Community Action Plan are implemented with fidelity.
Community Plan Implementation Training 3-43
![Page 44: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/44.jpg)
Participants will be able to:
1. Explain the elements of successful, high-fidelity implementation.
2. Develop implementation plans.
3. Identify methods of monitoring implementation for fidelity.
Community Plan Implementation Training 3-44
![Page 45: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/45.jpg)
High-fidelity implementation means that:
• the target audience is reached
• all components of the program are delivered according to the original design
• the program is delivered at the intended dosage (duration, frequency).
Community Plan Implementation Training 3-45
![Page 46: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/46.jpg)
Studies have found that:
• the effectiveness of any program is significantly increased when the program is implemented with fidelity
• some programs are only effective when implemented with a high degree of fidelity.
(Blueprints, 2001)
Community Plan Implementation Training 3-46
![Page 47: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/47.jpg)
• Support of site administrators
• Qualified implementers who support the program
• Planning meetings and trainings
• A detailed implementation plan(who, what, where, when and how)
• Ongoing monitoring and technical assistance
Community Plan Implementation Training 3-47
![Page 48: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/48.jpg)
Select implementers who:
• have the credentials prescribed by the program
• support the program approach
• are able to reach/communicate with target audience.
Community Plan Implementation Training 3-48
![Page 49: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/49.jpg)
• Involve implementers in the planning process.
• Provide implementers with clear information about the program.
• Train implementers to deliver the program.
• Consider a written participation agreement.
Community Plan Implementation Training 3-49
![Page 50: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/50.jpg)
• To educate stakeholders and build support for the program
• To negotiate implementation logistics
• To develop detailed implementation plans
• To obtain signed participation agreements
Community Plan Implementation Training 3-50
![Page 51: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/51.jpg)
Implementers should leave trainings with:
• a clear understanding of the program goals, rationale, desired outcomes, components and methods
• confidence that they have the skills and expertise to deliver the program effectively.
Community Plan Implementation Training 3-51
![Page 52: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/52.jpg)
Include:
• target audience and expected number of participants
• plans for recruiting participants
• who the implementers are
• schedule for implementing all components.
Community Plan Implementation Training 3-52
![Page 53: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/53.jpg)
Community Plan Implementation Training 1-<#>Community Plan Implementation Training 3-53
![Page 54: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/54.jpg)
• To ensure that the program is implemented with fidelity to the original design.
• To identify and correct implementation problems.
• To fulfill requirements of funders.
• To provide “lessons learned” for future implementation efforts.
• To identify and celebrate early successes.
Community Plan Implementation Training 3-54
![Page 55: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/55.jpg)
• The target audience is not participating in great enough numbers.
• The program components are not being delivered at the intended dosage.
• Program delivery methods are not being used.
Community Plan Implementation Training 3-55
![Page 56: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/56.jpg)
Monitor:
• planning meetings before implementation
• training of implementers
• program delivery.
Community Plan Implementation Training 3-56
![Page 57: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/57.jpg)
• Who attended the trainings?
• What was covered at the trainings?
• Did implementers leave trainings with the skills, confidence and motivation to deliver the program effectively?
Community Plan Implementation Training 3-57
![Page 58: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/58.jpg)
• How were participants recruited?
• How many members of the target audience participated?
• Was the program delivered at the correct dosage?
• Did implementers use methods prescribed in the program design?
• Did implementers communicate effectively with the target audience?
Community Plan Implementation Training 3-58
![Page 59: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/59.jpg)
Community Plan Implementation Training 1-<#>Community Plan Implementation Training 4-5
![Page 60: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/60.jpg)
1. Engage stakeholders.
2. Define what will be evaluated.
3. Select an evaluation design.
4. Decide how you will collect and measure data.
5. Gather, analyze and interpret data.
6. Report your findings and use the results.
Community Plan Implementation Training 4-7
![Page 61: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/61.jpg)
• Use the instrument designed for the program.
• If no instrument is available, seek help from an expert evaluator.
Community Plan Implementation Training 4-15
![Page 62: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/62.jpg)
Consider:
• commitment to the project and to the evaluation
• objectivity
• ability to collect complete and accurate data.
Community Plan Implementation Training 4-18
![Page 63: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/63.jpg)
Determine:
• who will oversee the evaluation
• a central storage place for data from each program
• evaluation deadlines
• decision-making processes.
Community Plan Implementation Training 4-19
![Page 64: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/64.jpg)
• Choose an appropriate format.
• Write for your most important audience.
• Discuss results in terms of the overall program goals.
• Be honest.
Community Plan Implementation Training 4-20
![Page 65: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/65.jpg)
Identify individuals with:
• skills in designing evaluation instruments
• skills in designing and conducting evaluation studies
• skills in data analysis and presentation
• the ability to objectively assess outcome results and recommend possible solutions
• the ability to credibly communicate results to key stakeholders.
Community Plan Implementation Training 4-24
![Page 66: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/66.jpg)
• Celebrate success.
• Write a report detailing the results.
• Fulfill accountability requirements.
• Identify causes of unmet expectations.
• Revise and update the Community Action Plan.
Community Plan Implementation Training 4-25
![Page 67: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/67.jpg)
• Weak connection between the program and outcomes
• Unrealistic outcomes
• Failure to implement a program with fidelity
• Participant-related reasons
• External factors
• Measurement problems
• Decreases in sample size (attrition)
Community Plan Implementation Training 4-26
![Page 68: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/68.jpg)
• Celebrate success.
• Identify changing priorities.
• Select new tested, effective prevention strategiesto address changing priorities.
• Assist resource allocation decisions.
• Evaluate policies and practices.
Community Plan Implementation Training 4-30
![Page 69: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/69.jpg)
If you want to build a ship, don't drum up the
men to gather wood, divide the work, and
give orders. Instead, teach them to yearn for
the vast and endless sea.
-- Antoine de Saint-Exupéry
![Page 70: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/70.jpg)
Thank You!Brian K. Bumbarger
Evidence-based Prevention and Intervention Support Center
Prevention Research Center, Penn State University
206 Towers Bldg.
University Park, PA 16802
(814) 865-2618
www.prevention.psu.edu
![Page 71: Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute](https://reader033.fdocuments.us/reader033/viewer/2022042221/5ec764bc7821140b1667ac8e/html5/thumbnails/71.jpg)
Programs
(6 mos. - 2 years)
What kind of
future do you
envision for
your
community’s
children ?
What changes in
ATOD use,
delinquency,
violence, teen
pregnancy,
dropout will be
necessary to
achieve the
vision ?
What changes in
protective
factors are
needed to
reduce problem
behaviors ?
What changes in
risk factors are
needed to
reduce problem
behaviors ?
How will your
programs &
services change
risk &
protective
factors ?
Outcome-Based Planning Logic Model
Reduction in
problem
behavior
(3 - 10 years)
Increase in
Protective
Factors
(1-4 years)
Decrease in
priority risk
factors
(1-4 years)
Vision
(10 – 15 years)
Vision
Statement
Problem
Behaviors
Protective
Factor
Prevalence
Risk Factor
Prevalence
Program
Outcomes