LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009.
-
Upload
chaya-lively -
Category
Documents
-
view
214 -
download
1
Transcript of LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009.
LiveWell Colorado Evaluation
LiveWell Colorado Taskforce Meeting
July 16, 2009
LWCO Evaluation Team: Mission
To guide and support LiveWell Colorado
communities in evaluation and to report the overall outcomes and impact of
LiveWell initiatives.
LWCO Evaluation Team: Who We Are
Sr. Research Asst.Deanna McQuillan, MA
Sr. Research SpecialistErica Morse, MA
Sr. Research Asst.Lisa Harner, MA
Tristan Sanders, BS
Sr. Evaluation ManagerBonnie Leeman-Castillo, PhD
Data AnalysisData Collection
Community Reports & TAData Management
& Reports
Statistical AnalystSophia Newcomer, MPH
KPCO Community & Local Gov’t Relations
Corina Lindley, MPHMonica Buhlig, MPH
Principal Investigator Diane King, PhD
What We Do
Collect Enter Analyze Report
…………………………………………………Data
Primary Sources of Data
Demographic & Process Data Community Action Plan (year end) Continuation Progress Report (year end) Annual Site Visits (summer) Key informant interviews with key community
coalition members (fall) TA Reports (monthly)
Outcomes Data Continuation Progress Report (year end) IVR Surveys (varies by community)
Other Sources of Data
Media and Website Hits Community Highlights Local Evaluators’ Reports School surveys CDPHE surveys
e.g., BRFSS and Child Health Survey
Today’s Plan
LWC IVR Survey: Analytic Update Progress Milestones
Current indicators of future success Community Reports
Small group discussion
LiveWell Colorado IVR Survey:Analytic Update
Sophia Raff Newcomer, MPHBiostatisticianInstitute for Health Research, Kaiser Permanente Colorado
Agenda
What’s new with the IVR survey? IVR survey updates Timing of community surveys
Analytic update Discussion of raw vs. weighted data Examples
What’s new with the IVR survey? Timing of survey
Survey will be conducted: M&P Year Year 1 (first year of implementation) Year 5 Year 10
Questions Consistent “core” questions Limiting number of questions on survey
Unweighted data In the LiveWell Colorado IVR survey, a sample (n) of
community members participate in a phone survey.
The intent of the survey is to use the responses from that sample of community members (n) to better understand key health and behavior indicators of the entire community (N).
Community (N)
Sample (n)
Unweighted data (cont.) The data collected is raw data
Raw data: the actual responses from survey participants Sometimes refered to as “unweighted”
If we use the raw data from the sample (n) to draw conclusions about the community (N), then we assume that: Non coverage and non response bias is the same
throughout all demographic groups in the community The sample (n) is representative of the community
These assumptions are violated in the LWCO IVR
Example: Park Hill 2006Question: # of fruits and vegetables/dayN=19,315 (2000 Census) n= 321
18-24 25-34 35-44 45-54 55-64 65+
Population (N) 963 2003 1947 1769 1009 1159Population (%) 4.99% 10.37% 10.08% 9.16% 5.22% 6.00%
Population (N) 989 1941 2184 2201 1359 1791Population (%) 5.12% 10.05% 11.31% 11.40% 7.04% 9.27%
MA
LE
FE
MA
LE
18-24 25-34 35-44 45-54 55-64 65+
Population (N) 963 2003 1947 1769 1009 1159Population (%) 4.99% 10.37% 10.08% 9.16% 5.22% 6.00%
Sample (n) 1 7 10 20 13 20Sample (%) 0.31% 2.18% 3.12% 6.23% 4.05% 6.23%
Population (N) 989 1941 2184 2201 1359 1791Population (%) 5.12% 10.05% 11.31% 11.40% 7.04% 9.27%
Sample (n) 4 19 44 42 56 85Sample (%) 1.25% 5.92% 13.71% 13.08% 17.45% 26.48%
MA
LE
FE
MA
LE
18-24 25-34 35-44 45-54 55-64 65+
Population (N) 963 2003 1947 1769 1009 1159Population (%) 4.99% 10.37% 10.08% 9.16% 5.22% 6.00%
Sample (n) 1 7 10 20 13 20Sample (%) 0.31% 2.18% 3.12% 6.23% 4.05% 6.23%
ICW = Population (N) / Sample (n) 963.00 286.14 194.70 88.45 77.62 57.95
Population (N) 989 1941 2184 2201 1359 1791Population (%) 5.12% 10.05% 11.31% 11.40% 7.04% 9.27%
Sample (n) 4 19 44 42 56 85Sample (%) 1.25% 5.92% 13.71% 13.08% 17.45% 26.48%
ICW = Population (N) / Sample (n) 247.25 102.16 49.64 52.40 24.27 21.07
MA
LE
FE
MA
LE
Weighted survey analysis (Park Hill data) Weighted survey analysis adjusts the raw,
collected data from the sample (n) to try to “look” like the population (N)
Weighting on age and sex, the individual case weights (ICW) are: Small cell sizes; collapse
with next age group
Definitions…and then some data! Survey sample size: number of respondents that started
the IVR survey (provided a valid age and pressed a key for the first question)
Item sample size: number of respondents who provided a valid response for that question
95% confidence interval: range of values in which the “true” weighted percentage is expected to occur, with 95% probability If we drew 100 samples from that same community, we would
expect the weighted percentage from 95 of those samples to fall within the reported 95% confidence interval
Have you heard of Park Hill Thriving Communities?
2006 2007 2008
Survey Sample Size 420 524 529
Item Sample Size n=408 n=514 n=521
Response= Yes 20.1% 37.0% 41.8%
Response= Yes
Weighted %(95% C.I.)
17.0% ( 12.4% - 21.6% )
33.4% ( 28.0% - 38.7% )
39.3% ( 33.8% - 44.8% )
RAW DATA
WEIGHTED DATA
Have you heard of Park Hill Thriving Communities?
Weighted percentage for response=“Yes”
2006
10%
20%
30%
40%
17.0%
12.4% 21.6%
2007
33.4%
28.0% 38.7%
2008
39.3%33.8% 44.8%
Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?PARK HILL 2006 2007 2008
Survey Sample Size 420 524 529
Item Sample Size n=360 n=440 n=476
Meeting guidelines 33.1% 41.4% 31.9%
Meeting guidelines
Weighted %(95% C.I.)
37.3%( 25.4% - 49.2% )
39.9%( 33.7% - 46.0% )
34.9%( 28.7% - 41.1% )
RAW DATA
WEIGHTED DATA
Meeting guidelines of 5 or more servings of fruits and vegetables per day?
PARK HILL 2006 2007 2008
Survey Sample Size 420 524 529
Item Sample Size n=321 n=469 n=463
Meeting guidelines 47.4% 44.1% 42.3%
Meeting guidelines
Weighted %(95% C.I.)
44.7%( 37.4% - 51.9% )
41.3%( 35.5% - 47.2% )
38.7%( 32.6% - 44.7% )
RAW DATA
WEIGHTED DATA
Have you heard of LiveWell Commerce City?
2006 2007 2008
Survey Sample Size 305 320 354
Item Sample Size n=281 n=315 n=352
Response= Yes 20.6% 15.9% 20.5%
Response= Yes
Weighted %(95% C.I.)
14.0% ( 9.36% - 18.7% )
15.1% ( 10.1% - 20.0% )
19.1%( 13.7% - 24.4% )
RAW DATA
WEIGHTED DATA
Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?COMMERCE CITY 2006 2007 2008
Survey Sample Size 305 320 354
Item Sample Size n=234 n=243 n=305
Meeting guidelines 24.4% 27.2% 31.8%
Meeting guidelines
Weighted %(95% C.I.)
26.7%( 18.9% - 34.6% )
27.3%( 20.4% - 34.2% )
35.5%( 27.9% - 43.1% )
RAW DATA
WEIGHTED DATA
Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?
Weighted percentage of respondents meeting moderate PA guidelines
COMMERCE CITY
2006
20%
30%
40%
26.7%18.9% 34.6%
2007
27.3%20.4% 34.2%
2008
35.5%27.9% 43.1%
Meeting guidelines of 5 or more servings of fruits and vegetables per day?
COMMERCE CITY 2006 2007 2008
Survey Sample Size 305 320 354
Item Sample Size n=192 n=248 n=301
Meeting guidelines 33.9% 39.9% 29.6%
Meeting guidelines
Weighted %(95% C.I.)
29.1%( 21.5% - 36.8% )
38.8%( 31.0% - 46.6% )
25.8%( 19.6% - 31.9% )
RAW DATA
WEIGHTED DATA
Data weighting summary Advantages:
Allows the collected IVR data to “look” more like the population from which it is sampled
Limitations 2000 Census data Weighting on other sociodemographic factors,
such as race, is impractical Small sample sizes and “extreme” ICWs
Any questions?
Progress Milestones
How do we answer that question????? Collaborative Partnerships & Community
Capacity
4 Intermediate Outcomes that Predict Impact Volume of changes: number of new or modified community &
systems changes. Intensity of changes: modifying access & removing barriers
more intense than providing information about services. Permanence (durability) of changes: ongoing versus once Penetration of changes: was contact made (reach) through
multiple sectors & settings (adopters) or geographic areas.
Collaborative Partnerships & Capacity Building
Are Partners Broadly Representative? e.g., 4 core sectors, parents, policy makers, police,
target populations, non-profit organizations
Are Partners (including residents) Engaged? e.g., shared leadership & work teams
Is Community Capacity Increasing? e.g., amount of resources leveraged
Volume & Intensity
How many organizational systems changes adopted?
How many environmental changes made?
How many sectors are you working in?
Permanence (a.k.a “maintenance”)
Are strategies durable? e.g., policy change > curriculum change > events
Has HEAL mission been adopted by partner organizations?
Has HEAL language been incorporated into planning & policy documents?
Broad Penetration (a.k.a. “reach”)
What proportion of target population(s) have been reached?
Are residents being reached at multiple levels of the ecological model? e.g., cooking class (individual / program);
corner store/farmer’s markets (family / environment); school rules involving snacks (organization / policy)
Broad Penetration (a.k.a. “adoption”) Are changes holistic (i.e., inter-linked)?
e.g., park improvements + safe routes to park > park improvements alone
Are single goals addressed in multiple sectors? e.g., breastfeeding addressed through:
healthcare (training providers to counsel, changing formula bags for lactation kits),
social marketing (norms & attitudes),
workplaces (space, time, supervisor awareness)
Community Reports
Community Reports: ActivityQuestions for Discussion:
Who would use this information: partners, stakeholder, and/or residents?
How might it be used? Is it a good way to display this information?
Does it need more/less text?
What would you add or subtract on this topic/subsection of the report?