Tools to evaluate policy and environmental changes:

Tools to evaluate policy and environmental changes:Opportunities for growing the field

Presented by:

Laura K. Brennan, PhD, MPH

Our Team & Partners

Robert Wood Johnson Foundation– Tracy Orleans, Laura Leviton (and Punam Ohri-Vachaspati)

Active Living By Design– Sarah Strunk, Phil Bors, Rich Bell, Fay Gibson, Joanne Lee, Mary Beth

Powell, Tim Schwantes, Risa Wilkerson– Community Partnerships (25 ALbD, 50 Healthy Kids, Healthy Communities)

Washington University Institute for Public Health (St. Louis)– Ross Brownson, Cheryl Carnoske, Christy Hoehner, Peter Hovmand,

Timothy Hower

Saint Louis University School of Public Health– Elizabeth Baker, Cheryl Kelly

Transtria LLC – Tammy Behlmann, Sarah Castro, Julie Claus, Peter Holtgrave, Courtney

Jones, Allison Kemner, Laura Runnels (many part-time staff and interns)

HKHC Leading Site Communities

Healthy Kids,Healthy Communities

Seattle/King County, WA

Oakland, CA

Central Valley, CA

Baldwin Park, CA

Columbia, MO

Chicago, IL

Louisville, KY

Washington, DC

Somerville, MA

Benton County, OR

Watsonville/Parajo Valley, CA

Rancho Cucamonga, CA

Phoenix, AZ

Cuba, NM

San Felipe Pueblo, NM

Grant County, NM

El Paso, TX

San Antonio, TX Houston, TXNew Orleans, LA

Jackson, MS

Desoto/Marshall/ Tate Counties, MS Jefferson County, AL

Boone/Newton Counties, AR

Kansas City, MO

Omaha, NE

Denver, CO

Kane County, IL

Milwaukee, WI

Houghton, MI

Flint, MI

Hamilton County, OH

Knoxville, TNChattanooga, TN

Moore/Montgomery Counties, NC

Greenville, SCSpartanburg, SC

Milledgeville, GA

Cook County, GA

Duval County, FL

Lake Worth/Greenacres/ Palm Springs, FL

Caguas, PR

Charleston, WV

Philadelphia, PA

Kingston, NYBuffalo, NY

Fitchburg, MARochester, NY

Healthy Kids, Healthy Communities (50 Grantees)

Nash/Edgecombe Counties, NC

Portland/Multnomah County, OR

Our Advisors

National Research Advisory Group Elizabeth Baker Rachel Ballard-

Barbash Frank Chaloupka William Dietz Lawrence Green Terry Huang Shiriki Kumanyika Marc Manley Robin McKinnon Shawna Mercer Meredith Reynolds Barbara Riley Eduardo Sanchez Loel Solomon

National WorkingGroup Karen Glanz Debra Haire-Joshu Laura Kettel-Khan Maya Rockeymoore James Sallis Janice Sommers Mary Story Sarah Strunk Antronette Yancey

National Policy/Practice Advisory Group

Don Bishop Elaine Borton Leah Ersoylu Steve Farrar Harold Goldstein Dean Grandin James Krieger Elizabeth Majestic Jacqueline Martinez Malisa McCreedy Leslie Mikkelsen Joyal Mulheron Thomas Schmid Marion Standish Ian Thomas Mildred Thompson

Our Projects

Projects Brief Description

1. Social determinants of health (2003-2010)

Funding: Centers for Disease Control & PreventionProcess: Forum, Presentation, ApplicationProducts: Workbook, Training, Train-the-trainer

2. Evaluation of Active Living by Design (2006-2010)

Funding: Robert Wood Johnson FoundationProcess: Interviews/Site visitsProducts: Journal Supplement, Case Reports

3. Review of environment & policy interventions for childhood obesity prevention (2008-2011)

Funding: Robert Wood Johnson FoundationProcess: Advisors, Resource reviewProducts: Intervention strategy summaries, Gaps

4. Evaluation of Healthy Kids, Healthy Communities (2009-2014)

Funding: Robert Wood Johnson FoundationProcess: Technical assistance, Interviews/Site visitsProducts: Articles, Policy Briefs, Resources, Tools

5. System dynamics modeling to inform overweight and obesity-relevant policy (2009-2011)

Funding: National Institutes of HealthProcess: Group model buildingProducts: System dynamics models

Connecting Projects

1) Bridge research/evaluation and policy/practice

2) Evaluate system, policy and environment change impacts and outcomes

3) Assess reach, adoption, implementation and sustainability of policy and environment interventions

4) Accelerate translation of replicable, evidence-based policy and environment interventions

5) Model the complex pathways from community contextual factors to behaviors and health

Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Family Practice 2008; 1-5.

Why Evaluate?

Why Evaluate?

Building the scientific evidence

Why Evaluate?

Shaping, creating policiesand practices for the field

Stevens B, Peikes D. When the funding stops: Do grantees of the Local Initiative Funding Partners Program sustain themselves? Evaluation and Program Planning 2006; 29 (2): 153-161.

Chances of sustainability are higher for projects that have been evaluated

Why Evaluate?

Why Evaluate?

To determine the effectiveness of local policy, environment, and system changes

To identify the changes with the greatest impact, relevance, feasibility and sustainability

To inform local decision-making, document successes & obtain more funding

THESE ARE NOT MUTUALLY EXCLUSIVE…

Why Evaluate?


Short-, intermediate- & long-term health behavior or obesity outcomes

Reliable & valid quantitative tools & measures

Study design and execution to ensure confidence in the findings

Why Evaluate?





OPPORTUNITY:What is a meaningful decrease in BMI? Increase in activity?

Decrease in sugar consumption?

Why Evaluate?





OPPORTUNITY:Should we use height/weight (BMI)? skinfold thickness? waist

circumference? accelerometers? 3-day food recall interviews? surveys?

Why Evaluate?





OPPORTUNITY:Prospective cohort designs? Complex time series designs?

Natural experiments? What is the intervention population? What is a representative evaluation population?

Why Evaluate?

To identify approaches with the greatest impact, relevance, feasibility and sustainability

What works, where it works, when it works, how it works & why it works (or why not)

Multi-method quantitative & qualitative measures

Local representation and participation to ensure confidence in the findings

Why Evaluate?





OPPORTUNITY:Partnership or coalition formed? New decision-making body or

position created? Policy developed? Policy adopted? Funds appropriated? New/improved structures, facilities or conditions? Policy compliance? Policy enforcement? Use of new facilities? Maintenance of new facilities?

Why Evaluate?





OPPORTUNITY:Should we use policy assessment? environmental audits?

direct observation? key informant interviews? focus groups? surveys? photovoice or digital storytelling? GIS mapping? web-based tracking systems?

Why Evaluate?





OPPORTUNITY:(Perspective of community members) Does the policy or

environmental change increase access to healthy foods or opportunities for physical activity? Did community members participate in planning, implementation and evaluation? Can the community sustain the changes over time?

Why Evaluate?


Track unintended results & practical considerations (resources, costs, assets & challenges)

Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings)

Findings respond to the interests of local audiences (decision-makers, business owners)

Why Evaluate?





OPPORTUNITY:Who did the work? For how long? Who provided resources?

How much did it cost? What funding was provided/leveraged? What were unanticipated benefits/challenges?

Why Evaluate?





OPPORTUNITY:What policy assessment tools inform policy development? What

audit tools can be used to increase awareness? Can digital storytelling be used for advocacy? What surveys help to estimate costs?

Why Evaluate?




Findings respond to the interests of local audiences (decision-makers, businesses, schools, residents)

OPPORTUNITY:What are the costs? What are the impacts on economic

development, academic performance, community safety or air quality? Are changes implemented and enforced equitably throughout the community?

Evidence

With expanded definitions of evidence, answers to these questions help to

bridge the gap between

research/evaluation and

policy/practice efforts…

Shaping the Field

Evaluation: Beginning to End


Connect the intervention and evaluation– Work plan and logic model

Determine the evaluation approach– Participatory, # communities, # strategies

Use multi-method quantitative & qualitative measures– Reliability, validity, feasibility

Seek partners for data collection & data analysis– Existing data, new data

Translate and disseminate findings– Audience, content, medium

Work Plan

Planning Goals Objectives Activities

Implementation Events Actions

Evaluation Outcomes Impacts Processes

Goal: Broad outcomes partners hopes to achieve aligned with the overall purpose of the project.

Lead Assessment/

Evaluation

Objective: More specific strategies/ methods aligned with and used to help achieve a goal (or related goals).

Individual or group to complete a goal, objective or activity.

Indicators that the goal has been reached, the objective accomplished or the activity implemented.Activity: Planned

events or efforts to help achieve the objective(s).

Work Plan

Goal: Adopt and implement a Complete Streets policy

Lead Assessment/ Evaluation

Objective: By June 2010, develop the components of a Complete Streets policy (e.g., sidewalk, bike lane, and crosswalk parameters).

Pedestrian/ Bicyclist committee

Complete Streets policy adopted by City/ implemented by x% of neighborhoods

Condition of sidewalks, bike lanes, and crosswalks (before and after policy)

Number and type of participants attending the Summit

Activity: Host a Complete Streets Summit that invites stakeholders from neighborhoods




Work Plan

Goal: Plan/organize/ implement a Corner Store Initiative

Lead Assessment/ Evaluation

Objective: By October 2010, identify 5 local farmers (producers) and 5 convenience stores (retailers) to improve access to healthy foods/beverages

Food Policy Council

Sales and profits on healthy, locally grown foods/beverages

Number of corner stores and local farms participating

Cost, placement and marketing of foods/beverages

Start-up and maintenance costs (e.g., refrigeration)

Activity: Create a plan for food distribution and storage (farmers to stores) and a business model for profit sharing




Logic Models

What is a logic model? Description of how activities to be carried out during a project

are related to the expected outcomes

Five core components: Inputs: resources, contributions, investments Activities: actions, events Outputs: immediate products (trained staff, meeting attendees) Outcomes: changes related to your objectives Impacts: changes related to your goals

Other considerations: Assumptions, External or Contextual Factors: beliefs about the

people involved, interactions and influence of the environment, political context, social determinants of health

Evaluation Approach

Participatory evaluation

Externalevaluation

Who?All partners engaged in planning, implementation, enforcement & sustainability

Objective, third party typically having some research or evaluation expertise

How?

• Collective planning and decision-making

• Simple, interactive and engaging process

• Tracking and monitoring to inform local efforts

• Reliable and valid measurement tools

• Multiple methods and measures

• High-quality design and execution of methods

When?Throughout the entire process, meaningful findings for all partners

Strategic points during intervention (baseline, midpoint, follow-up)

Evaluation Approaches

ExampleEvaluationMethods

Example Strategies

CompleteStreets

Farmer’sMarkets

Childcare Curriculum

PolicyAnalysis

SITESTOOLS

SITESTOOLS

SITESTOOLS

EnvironmentalAudits

SITESTOOLS

SITESTOOLS

SITESTOOLS

DirectObservation

SITESTOOLS

SITESTOOLS

SITESTOOLS

GISMapping

SITESTOOLS

SITESTOOLS

SITESTOOLS

Understanding system, policy and environment impacts

Similar settings, approaches and strategies

Common tools and/or measures

Multi-method

Methods

Description (examples)

Direct observation

Record of people, selected characteristics (age, gender), and behaviors (use of facilities, food purchases) for specific locations and time periods

Audit/Photo/Video

Unobtrusive assessment of factors in the physical or social environment that can hinder or facilitate behavior (playground, menu boards)

Policy analysis

Inventory of existing policies, modified policies or new policies (nutrition standards, time for activity), also implementation and enforcement

GIS mapping

Spatial relationships using surveillance data and sophisticated software programs (location of fast food restaurants, streets with bike lanes)

Interview/ Focus group

Open-ended questions to partners, staff, decision-makers and community members (access to healthy foods or places to be active)

SurveyClose-ended questions to partners, staff, decision-makers and community members (perceptions of policy and environment changes)

Direct Observation

Strengths(+) Pre/post comparison(+) Evaluates the impact of physical changes or improvements on behavior

Challenges(-) May depend on external factors (e.g., weather, special events)(-) Requires many observations (times of day, days of week)

Environmental Audits

Strengths(+) Validated tools(+) Pre/post comparison(+) Impact of policies or physical projects on environmental conditions

Challenges(-) May not compare across different communities or physical projects(-) May not have facilities or environments to audit at baseline

Photos & Videos

Strengths(+) Provides visual representation of project impacts(+) Conveys project impacts to diverse audiences

Challenges(-) May be expensive depending on equipment and production(-) Requires consent for photo release

Policy Analysis

Strengths(+) Identifies policies that hinder or support healthy eating or active living(+) Tools provide potential policies for planning/ implementation

Challenges(-) May not capture practices (informal policies)(-) May not track policy development, implementation or enforcement

http://ww2.wkkf.org/DesktopModules/WKF.00_DmaSupport/ViewDoc.aspx?LanguageID=0&CID=6&ListID=28&ItemID=5000564&fld=PDFFile

W. K. Kellogg Food & Fitness Planning Guide

CDC Community Health Assessment aNd Group Evaluation (CHANGE) Tool

http://www.cdc.gov/healthycommunitiesprogram/tools/change.htm

http://www.ymca.net/communityhealthylivingindex/tools.html

YMCA’s Community Healthy Living Index (CHLI)










GIS Mapping

Strengths(+) Provides visual representation multiple data sources (triangulation)(+) Conveys project impacts to diverse audiencesChallenges(-) May be expensive (data & software)(-) May be difficult to access data(-) May not have current or accurate data

Interview

Strengths(+) Gathers what, who, where, when, how, and why responses(+) Captures emotional responses(+) Offers flexibility to clarify or probe in areas of interest

Challenges(-) May be time intensive(-) Reflects only one perspective(-) Requires expertise or experience in areas of interest

Has their been support from

elected officials or community

leaders?

What plans have been made sustain

the efforts over time?

What staff (time, expertise, training), resources (space, equipment) and

funding were needed?

Focus Group

Strengths(+) Gathers what, who, where, when, how, and why responses(+) Captures social and emotional responses(+) Offers flexibility to clarify or probe in areas of interest(+) Obtains multiple perspectives(+) Generates new ideas or questions

Challenges(-) May be time intensive(-) May require travel (in-person)(-) May be restricted to only a few topics rather than a broad spectrum of topics

What are barriers to healthy eating and active living

in your community?

What has made the initiative successful in your community?

How did your partnership get the community involved

in its efforts?

Survey

Strengths(+) May require few resources for data collection or analysis(+) Enables site and cross-site analysis of responses

Challenges(-) Limits the amount of detail and nuances across efforts that can be captured(-) May require multiple or long surveys for the range of activities conducted

http://www.activelivingresearch.org/

http://www.healthyeatingresearch.org/

Active Living Research

Healthy Eating Research

http://www.activelivingresearch.org/

http://www.healthyeatingresearch.org/

Web-based tracking system

Strengths(+) Focuses on goals, tactics, and benchmarks created by the communities(+) Keeps a log of all activities conducted

Challenges(-) May be time intensive(-) Development can be expensive(-) Depends on quality/ complete entries(-) Requires expertise for categorizing entries

http://www.healthykidshealthycommunities.org

Evaluation Partners

Increase evaluation capacity Co-design evaluation and dissemination activities Build on existing social relationships to solicit participation from

people and organizations Determine assets/barriers of people (e.g., knowledge, skills),

organizations (e.g., public health agencies) and infrastructures (e.g., parks, grocery stores) in the community

Create collaborative processes to engage communities in joint decision-making

Support for tool selection, data collection & data analysis Seek technical assistance for assessment & evaluation Document intended/unintended consequences Identify pathways, strengths and challenges

Partnership

Base Model

Utilitarian ModelLead Agency

Model

CollaborationModel

With Partners

Reassess your partnership structure

Create local awareness of the partnership

Bring in new partners

Ensure active participation

Encourage shared leadership and decision-making

Develop a strong sense of group identity

Create partnership principles or agreements

Develop partners skills and leverage resources

Recognize the efforts of your partners

With Communities

Increase community awareness of the policy or environment changes

Share lessons learned with similar groups

Gain support for future efforts (political or community support)Build on existing efforts and recognize new effortsIdentify potential new sources of funding

Other Related Efforts

• Measures Registry (subcontract with Mathematica)

• Energy Gap Modeling (Gortmaker & Wang)

• Developing an Evidence-Based Intervention Planning System for Obesity Prevention (NIH SBIR)

Laura K. Brennan, PhD, [email protected]

mailto:[email protected]

Washington StateHealthy Communities Assessment Workbook

Marilyn Sitaker, MPH

What is the Washington Healthy Communities Project?

Who: Washington State Department of Health programs that address chronic diseases and their risk factors

What: Build capacity of local health jurisdictions (LHJ’s) to address primary risk factors for chronic diseases: tobacco, nutrition, and physical activity

How: Change local policies, environments, and systems where people live, work, play, and go to school

5 LHJs were chosen based on their high levels of risk, low resource level, and willingness.

The Washington Healthy Communities Initiative: building local capacity

Capacity building consists of :

•Technical assistance tailored to each LHJ

•Series of skill-building trainings: •Policy & environmental change approach•Community assessment•Building local partnerships

•Developing action plans

•Data & tools to support assessment and planning

DOH PlanningBegins

July 2009 – December 2010

6-09

Project Launch!

Project Complete

LHJs Accept grants

BeginAssessment

Planning Training forAction Planning

Assessment Training

4-1012-098-09 10-09 6-102-10

LHJs CompleteAssessment

Timeline for HC Initiative

Assessment Package: Factors to Consider

1. Communities did not have time or skills to collect and analyze lots of data—therefore, DOH had to provide a clear assessment framework and give communities as much pre-analyzed data as possible.

2. Identify a set of core indicators for critical features of the built environment & data sources available state-wide

3. Main focus: building skills in interpreting, synthesizing, and connecting data to appropriate evidence-based intervention strategies.

Choosing Indicators, Finding Data

• Recommended Community Strategies & Measurements to Prevent Obesity in the United States MMWR July 24, 2009 / 58(RR07);1-26

• Advisory Group•University of Washington, Center for Public Health Nutrition and College of Urban Planning

•DOH spatial analysis group

•State Indicator Report on Fruits and Vegetables, 2009

Ch 1: Describing the Burden

Ch 2: Physical Activity

Ch 3: Nutrition

Ch 4: Tobacco

Ch 5: Focus Group Guide

Appendices: Menu of Strategies

Glossary

Healthy Community Assessment Workbook

Features of the Built Environment

• Chronic diseases, health conditions, risk factors and socio- demographic data

• Graphs & interpretation; poverty maps

Chapter 1: Describing the Burden

In Grays Harbor County

•Over a third of household have income less than 200 percent of poverty.

•3 out of 4 adults 25 and older do not have a college degree.

•About a 14 percent of the population is non-white or Hispanic.

Slightly less than a third of adults have no medical insurance.

A. Maps of built environment features

B. Data collection tools & instructions• Physical Activity• Nutrition• Tobacco

C. Worksheets for Chapters 2 -4• Summarize, synthesize & interpret findings

Chapters 2-4: Local Environment

Chapters 2-4, Built Environment:A. Maps & Statistics

Physical Activity

• % of city residences within ½ mile of a park, trail or school playground

Nutrition • % residences in a census block within

½ mile of healthy food retailer

Tobacco • Tobacco retailers within 1,000 feet of

school, park or clinic

Each map included a key, tech notes & interpretation

Physical Activity • Walkability/bikeability audit• Inventory of low-cost programs (adapted from RALA)Nutrition • Grocery store assessment (adapted from NEMS)• Inventory of community gardens, CSAs & food banksTobacco • Policies for tobacco-free campus

Chapters 2-4, Built Environment:B. Data Collection Tools

Each tool included instructions for collecting, analyzing & interpreting the data

Chapters 2-4, Built Environment:C. Summary Worksheets

A. Data Summary, Physical Activity Environments

1.What areas in your town lack parks, trails, and playgrounds?

2.Summarize gaps in the availability of physical activity programming.

3.Summarize problems identified during your walkability assessment

B. How data from three sources fit together:

1.How does the map of outdoor recreational facilities correspond to the poverty map in Chapter 1?

2.What areas in your town are far away from low-cost physical activity programs? Are these areas also far away from parks, trails and playgrounds?

3.Are there any other barriers to accessing these resources? This might be a good question to explore in your community focus group.

LHJ Response to Assessment Workbooks

Assessment Training

• Assessment Workbook for each LHJ

Training for Action Planning

• LHJ’s shared their A-ha moments

Formal Evaluation in progress

Acknowledgements

Washington State Department of Health:

Dennis McDermot , Mike Boysun, Nguyet Tran, Shanae Williams, Angela Kemple, and Craig Erickson

University of Washington

Colin Rehm and Phil Hurvitz

Centers for Disease Control and Prevention

Dr. Latetia Moore

THANKS!

Marilyn Sitaker

Washington State Department of Health

Chronic Disease Prevention Unit

360-236-3463

[email protected]

mailto:[email protected]

Tools to evaluate policy and environmental changes:

Documents

Transcript of Tools to evaluate policy and environmental changes: