Program Evaluation Needs Assessment Survey of Local Coalitions.
-
Upload
rodger-shields -
Category
Documents
-
view
214 -
download
1
Transcript of Program Evaluation Needs Assessment Survey of Local Coalitions.
![Page 1: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/1.jpg)
Program EvaluationNeeds Assessment Survey of Local
Coalitions
![Page 2: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/2.jpg)
How many evaluators does it take to change a light bulb? One to do a needs assessment One to do a feasibility study One to do a qualitative study to find out what bulb to change One to empower the bulb to change One to tender a contract for further study One to write performance indicators for success One to do a cost benefit analysis of the best bulb to buy One to do a meta-evaluation showing that all previous
studies have left everyone in the dark
![Page 3: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/3.jpg)
SO.......how many evaluators does it take?
None, actually. Evaluators don't change bulbs, that's an implementation problem!
-Program Evaluation gets a lot of flack!
![Page 4: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/4.jpg)
But it can save your assets!
![Page 5: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/5.jpg)
Objectives
Understand the purpose of program evaluation
Review local coalition survey results
![Page 6: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/6.jpg)
Most commonly asked PE question
How do I know I’m making a difference? Answer : Institute best practices… With fidelity, innovation, and critical
interpretation. Unfortunately, outcome evaluation is
expensive and rarely feasible with community intervention.
![Page 7: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/7.jpg)
Most commonly asked PE question
How do I know I’m making a difference? Answer : Institute best practices… With fidelity, innovation, and critical
interpretation. Unfortunately, outcome evaluation is
expensive and rarely feasible with community intervention.
![Page 8: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/8.jpg)
Process evaluation helps implement best practices with fidelity. Is my programming “evidence-based”? Nicotine patches are great. Stick one over each eye and
you can't find your cigarettes.
Is the “best practice” appropriate for my population?
What does the tobacco science say about adapting best practices to my population?
Am I appropriately following the science?
![Page 9: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/9.jpg)
Process vs. Outcome Evaluation Process evaluation: Determines whether best-practices
were adopted properly.
Outcome evaluation: Determines whether program was successful.
![Page 10: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/10.jpg)
Process Evaluation Assesses the delivery or implementation of a program. What happened and why? How is this different from
what was planned? To be evaluated:
• Activities• Materials• Delivery
• Numbers of staff • Audience• Logistics
![Page 11: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/11.jpg)
Process Evaluation TATU Example: How many youth participated in TATU
trainings? What were the youths’ perceptions of
each training? What were the implementation problems?
![Page 12: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/12.jpg)
Outcome Evaluation Measures what progress the program has
made towards its goals.
To be evaluated:
• Behaviors• Attitudes
• Knowledge• Skills
![Page 13: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/13.jpg)
Outcome Evaluation Example goals to evaluate for TATU: Short term – youth who participate in TATU
gain outreach skills. Intermediate – TATU youth reach fellow
students. Long term – Youth who TATU participants
reach actually use skills to abstain when tobacco is available.
![Page 14: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/14.jpg)
UW Tobacco Surveillance & Evaluation Program
“It is now proved beyond doubt that smoking is one of the leading causes of
statistics.”
http://www.medsch.wisc.edu/mep/
![Page 15: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/15.jpg)
UW Tobacco Surveillance & Evaluation Program Indirectly provides outcome evaluation of
Wisconsin’s coordinated tobacco control activities.
Fields surveys. Analyzes data to monitor state trends. Identifies progress and emerging challenges. Future – more geographically specific
information; better disparities data.http://www.medsch.wisc.edu/mep/
![Page 16: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/16.jpg)
Conclusion : Uses of program evaluation Integral part of every successful organization. Clarifies and re-focuses program goals and
strategies. Verifies that current strategies are still
appropriate. Refreshes staff expertise in best practices and
evidence-based programming. Documentation for supporters, funders, and
stakeholders.
![Page 17: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/17.jpg)
Survey Results
Results reported here are for fully-funded coalitions ($20,000 or more).
95% of fully-funded coalitions responded (40 of 42 coalitions that were contacted).
76% of coalitions responded in total (55 of 72). 94% of coalitions funded at $10,000 or more
responded (44 of 47).
![Page 18: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/18.jpg)
Survey Results: Is P.E. built into your regular program functioning?
85%
13%
0%
20%
40%
60%
80%
100%
Some currentevaluation
No currentevaluation
% o
f C
oa
litio
ns
![Page 19: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/19.jpg)
Survey Results : The majority of coalitions (85%) engaged in some program evaluation.
85%
13%
0%
20%
40%
60%
80%
100%
Some currentevaluation
No currentevaluation
% o
f C
oa
litio
ns
![Page 20: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/20.jpg)
Population
9%14%
19%22%
37%
0%
10%
20%
30%
40%
N W S NE SE
% o
f W
I po
pu
lati
on
Regional Distribution of Coalitions
Distribution of fully-funded coalitions by region
17%21% 21% 19%
24%
0%
10%
20%
30%
40%
N W S NE SE
% o
f C
oal
itio
ns
![Page 21: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/21.jpg)
Population
9%14%
19%22%
37%
0%
10%
20%
30%
40%
N W S NE SE
% o
f W
I po
pu
lati
on
Coalitions distributed geographically, rather than proportionate with population.
Distribution of fully-funded coalitions by region
17%21% 21% 19%
24%
0%
10%
20%
30%
40%
N W S NE SE
% o
f C
oal
itio
ns
![Page 22: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/22.jpg)
Number of full-time equivalent employees (FTEs) in Wisconsin:
33 FTEs statewide 0.82 per coalition
(33 hours per week) About 40¼ total
staff.
All fully-funded coalitions
33
0.820
5
10
15
20
25
30
35
FTEs in Wisconsin Average FTEs percoalition
![Page 23: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/23.jpg)
Total number of FTEs
5.1 4.7
8.5
6.9 7.5
0
2
4
6
8
10
N W S NE SE
FTEs by Region:
Number of residents per FTE
97,678
162,405
125,844
176,440
274,948
0
50,000
100,000
150,000
200,000
250,000
300,000
N W S NE SE
![Page 24: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/24.jpg)
Indication of urban-efficiency gains. Building partnerships is vital.
Total number of FTEs
5.1 4.7
8.5
6.9 7.5
0
2
4
6
8
10
N W S NE SE
Number of residents per FTE
97,678
162,405
125,844
176,440
274,948
0
50,000
100,000
150,000
200,000
250,000
300,000
N W S NE SE
![Page 25: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/25.jpg)
Most coalitions are funded entirely through a TPCP contract. This raises the potential for instability given a potential change
in state government. What steps are coalitions taking to diversify?
Grants Received by Funded Coalitions(Select all that apply)
87%71%
17% 10% 7% 2% 7%
0%
20%
40%
60%
80%
100%
Only TPCPand TTAfunding
Only TPCPcontractfunding
TTA CommunityFoundation
Federal AmericanLegacy
Othergrants
% o
f C
oal
itio
ns
![Page 26: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/26.jpg)
Priority Populations (Choose three)
Targeting youth has likely paid off.
After rising through 1999, youth cigarette use now drops every year.3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
![Page 27: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/27.jpg)
Priority Populations (Choose three)
More resources needed for 18-24 year olds.
3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
![Page 28: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/28.jpg)
Priority Populations (Choose three)
Less than 50% target policymakers.
“Other” populations reported include health care providers.
3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
![Page 29: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/29.jpg)
Coalition Objectives(Check all that apply)
“Other” reported goals:
Reducing tobacco use among pregnant women
Increasing the tobacco tax.
20%
33%
38%
58%
68%
73%
88%
90%
0% 20% 40% 60% 80% 100%
Other Goals
Smoke-free campuses
Smoke-free homes
Disparities
Treatment
Youth advocacy
Smoke-free worksites
Youth prevention
% of Coalitions
![Page 30: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/30.jpg)
Major Coalition Activities (Check all that
apply)
Coalitions selected an average of 10 activities.
All selected more than one activity.8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
![Page 31: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/31.jpg)
Major Coalition Activities (Check all that
apply)
Less than 100% recruit supporters and seek community partnerships.
Less than ¼ pursue alternative funds.
8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
![Page 32: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/32.jpg)
Major Coalition Activities (Check all that
apply)
“Other” activities: working with state
partners. helping worksites
create individual policies.
asthma workgroups. the N-O-T program. Oral health
screenings in disparate populations.
8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
![Page 33: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/33.jpg)
Back to Program Evaluation…
![Page 34: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/34.jpg)
Current Program
Evaluation Activities(Check all that
apply)
More than 50% document their basic program activities.
Herein lies a wealth of experience to share.
15%
15%
15%
18%
20%
30%
30%
30%
30%
38%
38%
40%
58%
58%
60%
65%
68%
0% 20% 40% 60% 80%
Other evaluation
Worksite survey
Survey of health professionals
Mail survey
Phone banking results
Observation data
Youth survey
Survey of general public
Track results of paid media
Solicit event feedback
Member survey
Pre and post tests
Quitline Stats
Track # meeting attendees
Track earned media
Lists of supporters
Track # event attendees
% of Coalitions
![Page 35: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/35.jpg)
15%
15%
15%
18%
20%
30%
30%
30%
30%
38%
38%
40%
58%
58%
60%
65%
68%
0% 20% 40% 60% 80%
Other evaluation
Worksite survey
Survey of health professionals
Mail survey
Phone banking results
Observation data
Youth survey
Survey of general public
Track results of paid media
Solicit event feedback
Member survey
Pre and post tests
Quitline Stats
Track # meeting attendees
Track earned media
Lists of supporters
Track # event attendees
% of Coalitions
Current Program
Evaluation Activities(Check all that
apply)“Other”: focus groups police reports CDC’s SPF
indicators WI Wins data First Breath
statistics parent surveys oral health
surveys data from local
agencies
![Page 36: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/36.jpg)
In your own words… Coalitions’ answers to open-ended
survey questions:
1. What questions would you want a program evaluation to answer?
2. Please describe any sources of information/data that might be available for use in program evaluation.
![Page 37: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/37.jpg)
1. What questions would you want a program evaluation to answer? Open-ended responses were grouped
into:
process questions outcome questions technical needs
![Page 38: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/38.jpg)
Process Questions: Is the program operating as intended?
Are training materials and meeting times best? Is the program structure optimal?
Are we reaching the intended population? Are our activities “best practice”?
Are we using best practices for helping pregnant women quit? Generally, what is the ranking of most- to least-
valuable practices? What is the need/demand for our program? How important are non-cigarette tobacco products?
![Page 39: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/39.jpg)
Process Questions: Is the program operating as intended?
Are training materials and meeting times best? Is the program structure optimal?
Are we reaching the intended population? Are our activities “best practice”?
Are we using best practices for helping pregnant women quit? Generally, what is the ranking of most- to least-
valuable practices? What is the need/demand for our program? How important are non-cigarette tobacco products?
![Page 40: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/40.jpg)
Outcome Questions: What difference has the program made?
Are we changing attitudes about second hand smoke?
Have our education materials impacted attitudes about smoke-free workplaces?
What behavior changes resulted? Have pledge cards increased smoke-free homes?
How many/who benefits? Progress on goals of multi-year action plan? Do results justify the costs?
![Page 41: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/41.jpg)
Technical needs: Where can we find detailed local statistics?
Where to find numbers of youth smokers, by school and by grade?
How can City of Milwaukee data be separated from Milwaukee county data?
How do we write survey questions that identify attitude and behavior changes? Where can we find template surveys that can be changed to
fit the needs of our community? How can we use program evaluation data to promote tobacco
control? What are indicators of whether progress is being made?
![Page 42: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/42.jpg)
2. Please describe any sources of information/data that might be available for use in program evaluation. Open-ended responses were grouped
into:
Qualitative Quantitative
![Page 43: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/43.jpg)
Qualitative Testimony from youth, school leaders, adults who work
with youth, partners, law enforcement. Documentation of earned media. Minutes of Board of Health meetings. Supporter data base. Mid-year and year-end reports on consolidated contract
tobacco objectives. Community needs assessment (every 5 years). Community Health Assessment data.
![Page 44: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/44.jpg)
Quantitative Statewide surveys/data: YRBS, BRFS, UW Burden of Tobacco, UW survey of smoke-free municipal
and government-owned buildings, Worksite survey, WIC data on tobacco use prevalence
Local surveys: Dane County Youth Assessment, PRIDE survey in county schools, South
West Youth Survey (SWYS), evaluation of local smoke-free ordinance, county smoke free air/tobacco use survey (400 residents surveyed), community health assessment data, survey data collected by community health improvement coalitions, oral health screening results
Program surveys/data: Friends Helping Friends (youth peer group) survey data, WI Wins
compliance checks data, First Breath data, Quit-line and Fax to Quit statistics, pre- and post-test data for participants, member survey
![Page 45: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/45.jpg)
Next Steps…
Potentially, there may be regional trainings in program evaluation methods in each of the DHFS service areas.
Show of hands: No PE training or experience Some PE training Some PE experience Some PE success
![Page 46: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/46.jpg)
Thanks! Local coalitions team:
Sue Marten, Chair Dianna Forrester Kathy Cahill Themis Flores Ramos Connie Olson Heidi Foster Deb Bruning Paula Silha Deb Gatzke Karen Hagemann Jody Moesch-Ebeling
T&TA Lynn Habrik Kathy Cahill Deb Bruning Mary Hilliker Sue Marten
Randy Glysch Cindy Musial Sue Marten
![Page 47: Program Evaluation Needs Assessment Survey of Local Coalitions.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb15503460f94bb6959/html5/thumbnails/47.jpg)