Healthy Greenville 2036 Evaluation Planning

Melissa L Fair | Community Action Director

Institute for the Advancement of Community Health

Furman University

Today’s Agenda: o Welcome and Introductions

o What is program evaluation and why do we need to evaluate?

o What are the different types of evaluation?

o What is a logic model and how does it help me develop my evaluation plan?

o Review components of the evaluation template

WHAT IS PROGRAM EVALUATION

o “…application of evaluation approaches, techniques, and knowledge to systematically assess and improve the planning, implementation, and effectiveness of programs.”

WHY WE EVALUATE

o Evaluation findings should be used both to make decisions about program implementation and to improve program effectiveness

oNonprofits today are being pressed (by funders) to demonstrate the effectiveness of their program activities by initiating and completing outcome-oriented evaluation of projects

CDC: WHY WE EVALUATE? 1. Monitor progress toward the program’s goals

2. Determine whether program components are producing the desired progress on outcomes

3. Permit comparisons among groups, particularly among populations with disproportionately high risk factors and adverse health outcomes

4. Justify the need for further funding and support

5. Find opportunities for continuous quality improvement.

6. Ensure that effective programs are maintained and resources are not wasted on ineffective programs

TYPES OF EVALUATION 1. Formative evaluation: ensures that a program or program activity is feasible,

appropriate, and acceptable before it is fully implemented

2. Process/implementation evaluation: determines whether program activities have been implemented as intended

3. Outcome/effectiveness evaluation: measures program effects in the target population by assessing the progress in the outcome(s) that the program is designed to achieve

4. Impact evaluation: assesses program effectiveness in achieving its ultimate goals

PROCESS EVALUATION Determines whether program activities have been implemented as intended

Strengthen your ability to report on your program and use information to improve future activities.

It allows you to track program information related to Who, What, When and Where questions:

1. To whom did you direct program efforts? 2. What has your program done? 3. When did your program activities take place? 4. Where did your program activities take place? 5. What are the barriers/facilitators to implementation of program activities?

OUTCOME EVALUATION Measures program effects in the target population by assessing the progress in the outcomes that the program is to address

Selected measures should be guided by your program’s logic model (we will get to that)

Outcome Evaluation Questions: 1. Were medical providers who received intensive STD training more likely to effectively counsel,

screen and treat patients than those who did not? 2. Did the implementation of STD counseling in community-based organizations result in changes in

knowledge, attitudes, and skills among the members of the target population? 3. Did the program have any unintended (beneficial or adverse) effects on the target population(s)? 4. Do the benefits of the STD activity justify a continued allocation of resources?

When do I use what type of evaluation?

STEPS FOR ENGAGING IN EVALUATION oPlanning: Preparing for an Evaluation Identifying Stakeholders and Establishing an Evaluation Team Developing Evaluation Questions Budgeting for an Evaluation Selecting an Evaluator

oImplementation: Designing and Conducting an Evaluation Determining Data-Collection Methods Collecting Data Analyzing and Interpreting Data

oUtilization: Communicating Findings and Utilizing Results Communicating Findings and Insights Utilizing the Process and Results of Evaluation

LOGIC MODELS

oDescribe how a program works and to what end

oGuide to help connect your planned program activities to your anticipated outcomes

USING A SIMPLE LOGIC MODEL PRODUCES:

(1) An inventory of what you have and what you need to operate your program

(2) A strong case for how and why your program will produce your desired results

(3) A method for program management and assessment

WHY ARE LOGIC MODELS HELPFUL? oThe logic model approach helps create shared understanding of and focus on program goals and methodology, relating activities to projected outcomes

oUsing a logic model throughout your program helps organize and systematize program planning, management, and evaluation functions

How to Read Your Logic Model

LOGIC MODEL INPUTS AND ACTIVITIES o Resources or Inputs: include the human, financial, organizational, and community resources a program has available to direct toward doing the work

o May Include: Funding Existing organizations Potential collaborating partners Staff and volunteers Time Facilities, equipment, and supplies.

LOGIC MODEL ACTIVITIES

o Program Activities: what the program does with the resources Bring about the intended program changes or results Includes processes, tools, events, technology, and actions that are an intentional part of the program implementation

o May Include Products – promotional materials and educational curricula Services – education and training, counseling, or health screening Infrastructure – structure, relationships, and capacity used to bring about the desired results

LOGIC MODEL OUTPUTS o Outputs are the direct products of program activities and may include types, levels and targets of services to be delivered by the program Indicate if a program was delivered to the intended audiences at the intended “dose.”

oMeasured through process evaluation

oDoes not provide information on changes in KSA’s, health behaviors or outcomes

o May Include: The number of classes taught, meetings held, or materials produced and distributed Program participation rates and demography Hours of each type of service provided

LOGIC MODEL OUTCOMES o Outcomes are the specific changes in program participants’ behavior, knowledge, skills, status and level of functioning Short-term outcomes should be attainable within 1 to 3 years Longer-term outcomes should be achievable within a 4 to 6 year timeframe

o May Include: Changes in attitudes, behaviors, knowledge, skills, status (health), or level of functioning expected to result from program activities Most often expressed at an individual level

LOGIC MODEL IMPACTS

o Impact is the fundamental intended occurring in organizations, communities or systems as a result of program activities May take 7 to 10 years and may not be visible until after funding ends organizational, community, and/or system level changes expected to result from program activities,

o May Include: Improved conditions, increased capacity, and/or changes in the policy arena

SMART Outcomes and Impacts

o Outcomes and Impacts should be SMART:

Getting Started with Your Logic Model

Resources Activities Outputs Short- and Long-Term Outcomes

Impact

In order to accomplish our set of activities we will need the following

In order to address our problem we will conduct the following activities

We expect that once completed or under way these activities will produce the following evidence of service delivery

We expect that if completed or ongoing these activities will lead to the following changes in 1-3 and 4-6 years

We expect that if completed these activities will lead to the following changes in 7 to 10 years

A Logic Model Example: Healthy Vending in Schools

Inputs Project Activities Outputs

Short Term Outcomes

Long Term Outcomes Impacts

Inputs

• Funding • Staff • Partners • Existing data • Existing policies • Vending Contract • Scientific Literature

Project Activities

• Engage school leaders- identify barriers

• Engage vending companies • Implement point of sale

messaging for healthy vending items

• Implement Healthy School Vending Options

Outputs

• Number of school leaders engaged

• Number of vending companies engaged

• Number of point of sale messages implemented

• Number of healthy vending machines/items placed

Short Term Outcomes

• Increased access to heathy foods and beverages

• Increase awareness of healthy foods and beverages

• Increased consumption of healthy foods and beverages

• Decreased consumption of unhealthy snacks

Long Term Outcomes

• Reduced prevalence of youth obesity

Impacts

• Reduced rates of chronic disease

• Reduced premature morbidity and mortality

• Improved quality of life • Reduced medical costs

“Measuring Our Impact”

TODAY WE ARE GOING TO FOCUS ON: o The elements of you intended results: Outcomes and Impacts

o These components of your logic model give you an outline of what is most important to monitor and gauge to determine the effectiveness of your program

A Logic Model Example: Healthy Vending in Schools

Inputs Project Activities Outputs

Short Term Outcomes

Long Term Outcomes Impacts

Short Term Outcomes

• Increased access to heathy foods and beverages

• Increase awareness of healthy foods and beverages

• Increased consumption of healthy beverages and snacks

• Decreased consumption of unhealthy beverages and snacks

Can inform what outcomes you

intend to achieve

How We Measure

• Indicator: provides evidence that a certain condition exists or certain results have or have not been achieved

Short Term Outcomes

• Increased access to heathy foods and beverages

• Increase awareness of healthy foods and beverages

• Increased consumption of healthy beverages and snacks

• Decreased consumption of unhealthy beverages and snacks

Servings of healthy snacks and beverages

Servings of unhealthy snacks and beverages

Indicators

Youth BMI

Indicator Long Term Outcomes

• Reduced prevalence of youth obesity

Using the Logic Model to Develop the Evaluation Question

• Middle and High School Students Who • Decrease consumption of unhealthy beverages and

snacks • Increase consumption of healthy beverages and snacks What • Increased access to healthy foods and beverages

through healthy vending options How

Using the Logic Model to Develop the Evaluation Question

• Middle and High School Students Who • Decrease consumption of unhealthy beverages and

snacks • Increase consumption of healthy beverages and

snacks What

• Increased access to healthy foods and beverages through healthy vending options How

Do

After the implementation of

EVALUATION QUESTIONS Question 1: Do middle and high school students decrease the consumption of unhealthy beverages and snacks after the implementation of healthy vending machines to increase access to healthy foods and beverages?

Question 2: Do middle and high school students increase their consumption of healthy beverages and snacks after the implementation of healthy vending machines to increase access to healthy foods and beverages?

YOUR TURN….

Based on one of your proposed health outcomes, from the document provided, create your own evaluation question.

AN EXAMPLE… FARM TO SCHOOL

Using the Logic Model to Develop the Evaluation Question

• Middle and High School Students Who • Decrease consumption of unhealthy beverages and

snacks • Increase consumption of healthy beverages and

snacks What

• Increased access to healthy foods and beverages through healthy vending options How

Do

After the implementation of

NEXT STEP.. DESIGN YOUR EVALUATION PLAN

This plan will: 1. Guide you through each step of the process of evaluation 2. Help you decide what sort of information you and your stakeholders really need 3. Keep you from wasting time gathering information that isn't needed 4. Help you identify the best possible methods and strategies for getting the needed

information 5. Help you come up with a reasonable and realistic timeline for evaluation 6. Most importantly, it will help you improve your initiative!

FIRST SOME IMPORTANT RESEARCH TERMS • Independent variables: the program itself and/or the methods conditions that you want to evaluate

• They’re independent because their existence doesn’t depend on whether something else occurs: you’ve chosen them, and they’ll stay consistent throughout the evaluation period.

• If you’re evaluating a number of different methods or conditions, each of them is an independent variable

• Dependent variables: whatever may or may not change as a result of the presence of the independent variable(s).

• If you’re plan change in more than one behavior/outcome, each type of change is a different dependent variable • They’re called dependent variables because changes in them depend on the action of the independent variable

• Measures: measurements of the dependent variables • Usually refer to procedures that have results that can be translated into numbers • May take the form of community assessments, observations, surveys, interviews, or tests • May also count incidents or measure the amount of the dependent variable (#/% of children who are overweight or obese,

violent crimes per 100,000 population, etc.)

• Observations: might involve measurement, or • They might record what happens in specific circumstances: the ways in which people use a space, the kinds of interactions

children have in a classroom, etc.

COMPONENTS OF THE EVALUATION PLAN

1. Evaluation questions

2. Evaluation design

3. Instrument selection

4. Sampling plan

5. Data collection plan

6. Data analysis plan

EVALUATION QUESTIONS Program evaluations generally look for the answers to three basic questions: Was there any change – in participants’ or others’ behavior, in physical or social conditions, or in outcomes or indicators of success– during the evaluation period? Was whatever change took place – or the lack of change – caused by your program, intervention, or effort? What, in your program or outside it, actually caused or prevented the change?

You will be asked to develop three to five major evaluation questions for your program/intervention

You will be asked how your evaluation question contributes to the evidence base or program improvement

EXAMPLE

Evaluation Question How does the implementation of point of purchase labeling of healthy menu items influence the number of healthy items sold in restaurants?

Contribution to Evidence Base The average American now spends about half of their food dollars on food eaten outside of the home and consumes about 5 meals or snacks a week from restaurants. Additionally food consumption outside of the home is related to increased body weight due to the caloric density and low nutrient density of many restaurant and convenience foods. By labeling healthy food options the LWG on the menu initiative seeks to increase the number of healthy foods consumed as meals or snacks outside the home using easy to understand labeling in order reduce caloric intake and improve nutritional density of foods consumed.

EVALUATION DESIGN Every evaluation is essentially a research or discovery project Your research may be about determining how effective your program or effort is overall, which parts of it are working well and which need adjusting

If your results are to be reliable, you have to give the evaluation a structure that will tell you what you want to know.

That structure – the arrangement of discovery- is the evaluation’s design

Your ability to answer your research questions depends on the strength of your evaluation design

COMMON RESEARCH DESIGNS 1. Pre- and post- single-group design

2. Interrupted time series design with a single group (simple time series)

3. Control group design

PRE-POST SINGLE GROUP DESIGN The simplest design is also probably the least accurate and desirable: the pre (before) and post (after) measurement or observation.

Consists of: 1. Measuring whatever you’re concerned with in one group – the infant mortality rate,

unemployment, water pollution 2. Applying your intervention to that group or community 3. Observing again.

This type of design assumes that a difference in the two observations will tell you whether there was a change over the period between them also assumes that any positive change was caused by the intervention

INTERRUPTED TIME SERIES DESIGN WITH A SINGLE GROUP (SIMPLE TIME SERIES) An interrupted time series used repeated measures before and after delayed implementation of the independent variable (e.g., the program, etc.) to help rule out other explanations

The simplest form of this design is to: 1. take repeated observations 2. implement the program or intervention 3. Observe a number of times during the evaluation period, including at the close of the

intervention

Tracks the trend of change, and can therefore, help see whether it was actually the independent variable that caused any change

Can implement more than one independent variable Trying two or more, one after another (often with a break in between), or by adding each to

what came before

CONTROL GROUP DESIGN A common way to evaluate the effects of an independent variable is to use a control group This group is usually similar to the participant group, but either receives no intervention at all, or receives a different intervention with the same goal as that offered to the participant group.

A control group design is usually the most difficult to set up Have to find appropriate groups, observe both on a regular basis, etc. Generally considered to be the most reliable

Can be pre and post test or post test only

Can be randomized to condition

EVALUATION DESIGN CONSIDERATIONS

Will you use a single or multi-group design?

Will the design be cross-sectional or longitudinal?

Will you collect pre/post data?

Will you collect additional follow-up data? Single or multi-group/comparison group

You will also be asked for what is the rationale for your evaluation design

EXAMPLE Please describe the overall evaluation design, including the number and description of any comparison groups (if applicable) and the timing and frequency of data collection (e.g., before and after implementation).

The overall evaluation design is a quasi-experimental pre-test post-test with no comparison group. As part of a Memorandum of Agreement with each partnering food and beverage outlet baseline data will be collected using transactional level sales data obtained through the partner specific point of sale system. An attempt will be made to obtain one year of daily sales data prior to the implementation of labeling on the menu as baseline data. In addition the agreement will include quarterly obtainment of daily transactional level sales data through the partner specific point of sale system. This quarterly provision of daily transactional level sales data will commence with the initiation of the labeling on the menu effort and will continue until the end of the three year intervention period.

Will you collect data from the same individuals over time (longitudinal) or from independent samples at each time point (cross-sectional)?

Cross-sectional

INSTRUMENT SELECTION Your instrument is how you capture your desired outcome/variable of interest

May include surveys, focus groups, interviews, participants observations, scales, blood pressure cuff etc.

Use of established instruments is strongly recommended Demonstrated reliability and validity Ability to expand the evidence base through similar approaches to assessing outcomes

For each evaluation question you will be asked to list the instruments to be used and the outcome they are designed to assess You will be asked to provide references on reliability and validity of existing measures

Example

Instrument/ Data Source

Reference Please describe any modifications made to the original instrument

What key indicators are measured by this instrument/data source?

Daily transactional level sales data

Chu YH, Frongillo EA, Jones SJ, Kaye GL: Improving patrons' meal selections through the use of point-of-selection nutrition labels. Am J Public Health. 2009, 99: 2001-2005. 10.2105/AJPH.2008.153205.

none Proportion of healthy menu items sold, number of healthy menu items sold

Please list each of your evaluation instruments/data sources. If you know the indicators you want to measure, but have not yet selected the instruments, indicate "TBD" for the instrument. Please include available instruments as attachments when you submit your evaluation plan.

SAMPLING PLAN For each measure is a detailed outline of: Who will you collect data from? How will they be recruited? How will you ensure they are representative of the population you are intending to reach?

Random Sample vs Non-Random Sample

SAMPLING PLAN CONSIDERATIONS

oDetermine the sample size needed to detect effects Power analysis

oCollect sufficient cases at baseline to allow for attrition

oEnsure sufficient sampling of subgroups if examining subgroup differences

oIf you are not assessing all participants, consider random sampling to achieve a representative sample

oConsider potential biases sampling approach might introduce

EXAMPLE Describe the size and characteristics of your desired sample for this instrument. Please note any age, ethnic, gender, or other requirements for the sample.

By the end of the three year intervention period LiveWell Greenville aims to have 29 new restaurants, 28 food and beverage retail outlets, and 26 food and beverage concession stands participating in the LiveWell Greenville on the Menu Initiative.

We are unable to determine the specific demographics of our sample since we are collecting transactional level sales data. However careful attention will be given to obtaining a representative sample of the Greenville County population including disparate populations such as low income/SES, race/ethnicity, and gender by partnering with a number of restaurants, convenience stores, concession stands, and corner markets located within food deserts and target neighborhoods where these populations reside.

EXAMPLE CONTINUED What sample size is needed to have enough power to detect a significant outcome? Awardees are encouraged to conduct a power analysis to answer this question?

While we recognize that power and sample size are important, the data collected will be transactional data from restaurants participating in the LiveWell Greenville on the Menu Initiative. Therefore sample size will be limited by the consumer base of the participating outlets. LiveWell will work to ensure that enough restaurants with a large and varied consumer base are recruited to ensure a sufficient sample size.

How will you recruit and/or select participants to respond to this instruments?

Participating restaurants, food and beverage retail outlets, and concession stands will be recruited to participate in the LiveWell Greenville on the Menu initiative by the LiveWell Greenville At Mealtime Specialist. The specialist will work with each location to develop a Memorandum of Agreement, including expectations for labeling, advertising, and submission of sales data in order to analyze intervention effectiveness. The At Mealtime evaluator will provide assistance as needed to the At Mealtime specialist in order to include reporting requirements of sales data in the Memorandum of Agreement.

DATA COLLECTION PLAN

Influenced by your evaluation design

For each instrument includes: 1. Method and protocol (e.g. observation, survey, records transfer,

interviews, etc.) 2. Who will administer this instrument/collect the data? 3. How will you train data collectors to ensure data are collected accurately

and reliably? 4. How many times will you administer this instrument and when?

DATA COLLECTION PLAN CONSIDERATIONS

oDevelop and use standard data collection protocols

oTrain data collectors and conduct practice sessions

oDevelop clear protocols for data management Timing Pre/Post intervention

Frequency Additional follow-up recommended

Cross-sectional or longitudinal

EXAMPLE Describe the data collection method and protocol for this instrument (e.g., observation, survey, records transfer, interviews, etc.): Food and beverage outlets participating in the LiveWell Greenville on the Menu Initiative will sign a Memorandum of Understanding, which includes an agreement to provide a transfer of daily transactional sales data on a quarterly basis. The individual data collection instrument will vary depending upon the point of sale system utilized by each partnering food and beverage outlet.

Who will administer this instrument/collect the data? Sales data will be collected on a transactional level by each food retailer using partner specific point of sale system.

EXAMPLE CONTINUED How will you train data collectors to ensure data are collected accurately and reliably? No training will be provided to data collectors on the use of the point of sale system to obtain transactional level data. The evaluator will work with each partner to ensure that the data is accurately transferred on a quarterly basis.

How many times will you administer this instrument and when? Sales data will be collected quarterly on the daily transactional level. Quarterly provision of data will commence with the initiation of the labeling effort and will continue until the end of the 3 year intervention.

CHARACTERISTICS OF A GOOD EVALUATOR 1. Experience in the type of evaluation needed

2. Comfortable with quantitative data sources and analysis

3. Able to work with a wide variety of stakeholders, including representatives of target populations

4. Can develop innovative approaches to evaluation while considering the realities affecting a program (e.g., a small budget)

5. Incorporates evaluation into all program activities

6. Understands both the potential benefits and risks of evaluation

7. Educates program personnel in designing and conducting the evaluation

8. Will give staff the full findings (i.e., will not gloss over or fail to report certain findings)

DATA ANALYSIS For each evaluation question you will be asked to answer the following questions:

1. What indicators will be examined? 2. What types of variables will be examined (nominal, ordinal, ratio)? 3. What inferential analyses will you conduct to answer your evaluation

question? 4. What procedures will you conduct to test your assumptions? 5. What types of variables will be examined?

EXAMPLE Evaluation Question: How does the labeling of healthy menu items affect the number of healthy menu items sold?

What indicators will be examined?

Number (units) of healthy menu items sold

What types of variables will be examined? Ratio

What inferential analyses will you conduct to answer your evaluation question?

In order to assess actual use, descriptive statistics will be used to determine counts of the total number of healthy menu items (units) sold across all partners.

Appropriate inferential statistics, potentially one way ANOVAS and t-tests, will be used to determine the change in the proportion of healthy menu items sold before and after implementation.

EXAMPLE CONTINUED

What procedures will you conduct to test your assumptions?

1)Test significant outliers by creating box plots 2) Test that dependent variable approximately normally distributed for each group of the independent variable with Shapiro-Welk test 3) Test for equal variances using Levene’s test

Which variables will be examined?

Number of healthy menu items (units) sold, change in number of healthy menu items (units) sold post LiveWell Greenville on the Menu implementation

HOW TO GET BUY IN FOR YOUR EVALUATION 1. Clear expectations and benefits of community

evaluation requirements discussed up front and often

2. Offering of stipends or incentives to program participants when feasible

3. Buy-in from communities through hiring of community researchers and academic logistical support

4. Community data-driven dissemination with engagement

THERE ARE FOUR STEPS FOR BUILDING STRONG EVALUATION PLANS: 1. Establish partnerships

2. Use a collaborative process to develop/revise the logic model

3. Use your logic model to build the evaluation plan

4. Sustain your program and evaluation partnerships

EVALUATION RESOURCES

1. CDC’s Framework for Evaluation

2. CDC’s Introduction to Program Evaluation for Public Health Programs

3. W.K. Kellogg Foundation Logic Model Development Guide

4. Innovation Network Evaluation Capacity Building Resources