What Works in Multi-Site Evaluations of Nutrition Education Interventions?
description
Transcript of What Works in Multi-Site Evaluations of Nutrition Education Interventions?
What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?
What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?
Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie KeihnerCalifornia Nutrition Network for Healthy Active Families
Research and Evaluation Unit
California Nutrition Network California Nutrition Network for Healthy, Active Familiesfor Healthy, Active Families
Targets low-income families
Conducts a social marketing campaign to increase: fruit and vegetable
consumption physical activity promote the Food Stamp
Program Contracts over 170
agencies Receives almost 106
million from USDA food stamp program
Strategies
Factors thatInfluence
FVC
Change inBehavior
Evaluation for contractors Evaluation for contractors receiving >$350kreceiving >$350k
Process
Short-term
ImpactLong-term
impact
Network Developed Network Developed MethodologyMethodology
1. SOW Impact evaluation objective2. Compendium of Surveys 3. Impact Evaluation Handbook 4. Trainings and workshops 5. Data entry templates 6. Evaluation report template7. Monitoring
Scope of Work Impact Scope of Work Impact Evaluation ObjectiveEvaluation Objective
Impact objective: By Sep. 30 20xx, a sample of 4th and 5th grade students at intervention sites will report an increase in fruit and vegetable consumption and an increase in one or more factors related to fruit and vegetable consumption, such as knowledge, preferences, outcome expectations and self-efficacy.
Impact Evaluation Impact Evaluation HandbookHandbook
Compendium of SurveysCompendium of Surveys
TrainingsTrainings
WorkshopsWorkshops
One-on-one TAOne-on-one TA
Data Entry TemplatesData Entry Templates
Created in MS ExcelProvides same results as SPSSCalculate pretest mean, posttest
mean, difference and p-value for a paired t-test
Customized for each survey
Data Entry TemplatesData Entry Templates
Data Entry TemplatesData Entry Templates
Evaluation Report TemplateEvaluation Report Template
18 questionsDesignMethodsResultsReflectionSuccesses
Report Template - Report Template - DesignDesign
1. Please type your impact objective. 2. What were the factors you measured?3. What intervention and strategies were implemented
to change the factors? 4. What survey tool did you use? 5. Were there questions on the survey tool that were
not addressed in the intervention? If so, why?6. Describe challenges faced during the intervention
implementation. 7. What type of design was used, e.g., pretest and
posttest, pre-posttest with a control, posttest only or other?
Report Template - Report Template - Evaluation Survey ImplementationEvaluation Survey Implementation
8. What type of design was used, e.g., pretest and posttest, pre-posttest with a control, posttest only or other?
9. Who were the survey respondents? (please give age range)
10. Where were they administered (name and type of locations, e.g., health department, community center, mobile home park, etc.)
11. When were the pretest and posttest surveys administered? (Please give dates.)
12. How many were administered? (If a different number of pretests and posttest please explain.)
13. How long did it take the respondents to answer the survey? (Please give range in minutes.)
Report Template - Report Template - Evaluation ResultsEvaluation Results
14. What were the results? How would you interpret these results? How might these results be used?
15. How would you interpret these results?
16. How might these results be used?
Report Template - Report Template - ReflectionReflection
17. What were the big challenges you encountered during the evaluation process?
18. Now, reflect on the successes you’ve had over the past year. Describe the best moment you had as a nutrition educator this year. Think about a moment when you knew your nutrition education had made a difference in someone’s life or give an example of a time when you were proud to be a nutrition educator.
19. Share a story about a one of the best moments you had while conducting the evaluation this year? Think about one of the highlights?
20. If Aladdin appeared with his magic lamp and offered you a wish to have more of those best moments or highlights what would you ask for?
Monitoring sheet (in MS Monitoring sheet (in MS Excel) includes columns for:Excel) includes columns for:
Contractor Name Contact Name Region Channel Federal Share Program Manager Required/Volunteer Contract External Evaluator Number of years in impact evaluation
Monitoring sheet Monitoring sheet (continued)(continued)
Evaluation Objective Factors (to be)/(that were) measured Design (posttest only, posttest with control, pre-
posttest, pre-post with control) Participant’s age (grade, adults/youth, in years) Survey Used Have a copy of survey Pretest date(s) Posttest date(s) Number of Surveys expected (collected)
Monitoring sheet Monitoring sheet (continued)(continued)
Need Data Entry Template (date) Sent data entry template (date) Received Report (in email from & date) Report Late/Received Draft Extension Granted (notes) Data Received Need Training
What were the indicators of What were the indicators of success?success?
Factor
# that measured
change in factor
03-04 04-05 05-06
Knowledge 7 7 19
Preferences 10 13 22
Self-efficacy 3 4 18
Outcome expectations
2 2
Consumption 1 7 22
PA preferences 0 1
Who has participated?Who has participated?
Channel03-04
n=12 (44%)
04-05
n=24 (67%)
05-06
n=45 (98%)
Schools 8 12 24
Colleges and universities
2 3 4
Health departments
1 6 11
County Offices of Ed.
2 6
Cooperative Extension
1 1 1
First 5 1
Funding Funding (% of total budget)(% of total budget)
Federal Funding of Contractors that Participated in Impact Evaluation
16
30
45
0
10
20
30
40
50
2003-2004 2004-2005 2005-2006
Year
Fu
ndin
g
(In
Million
s o
f D
ollars
)
(19%) (34%) (47%)
ResourcesResources
www.ca5aday.comResearch and Evaluation
See impact evaluation section at the bottom
www.harvestofthemonth.com
Hoorah! – More MattersHoorah! – More Matters