Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst David P. Weikart Center for Youth Program Quality Charles Smith Vice President for Research Forum for Youth Investment Executive Director David P. Weikart Center for Youth Program Quality April 18, 2012
#readyby21
Agenda • Welcome • Opening Activity • Important Questions
– The Why – The What – The How
• Site-Level Improvement • System-Level Planning
– The Where • Looking Forward/Next Steps • Questions
Objectives
• Learn how collecting program data from staff, youth and parents help to tell the overall story about Quality for a single site.
• See how this data can be used to identify low capacity programs and support resources targeting decisions.
Opening Activity Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
Quality of Instruction (Point of Service Setting)
YouthVoiceandProgramGovernanceStructures
ENGAGEMENT
INTERACTION
SUPPORTIVEENVIRONMENT
SAFEENVIRONMENT
Higherorderengagementthroughchoice,planning,andreflec4on.
Peerinterac4onthroughgroupingandcoopera4velearning.
Suppor4veenvironmentthroughwelcoming,conflictresolu4on,ac4velearning,andskillbuilding.
Physicalandemo4onalsafetyisprovided.
Why Were the Leading Indicators Developed?
Continuous Improvement Practices by Site Teams (Organizational Setting)
StandardizedAssessmentofInstruc4on
Team‐basedPlanningwithData
CoachingandPerformanceFeedback
TrainingforInstruc4onalSkills
Why Were the Leading Indicators Developed?
Objec4veData
MeaningfulInforma4on
Ac4on/Exper4se
ImprovedOutcomes
LowerStakesAccountabili4es
Interpre4veCommunity
• TeamSelfAssessment• Reviewexternalscores
TeamPlanningandImplemen4ng
• Improvementplanning• Performancecoaching
HigherStakesAccountabili4es
Why Were the Leading Indicators Developed?
Why Were the Leading Indicators Developed?
PointofServiceSeOng
Organiza4onalSeOng
PolicySeOng
Con%nuousImprovementPrac%cesforSiteTeams
QualityInstruc%on&ProximalChildOutcomes
LowStakesAccountabilityandSupports
TheoryofChange:Mul%pleLevelsofSe]ng
Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
Why Were the Leading Indicators Developed?
LeadingIndicator5.1–
FamilySa4sfac4on
LeadingIndicator4.2–AcademicEfficacy
LeadingIndicator4.1–Socioemo4onalDevelopment
LeadingIndicator2.2–
EngagingInstruc4on
LeadingIndicator1.1–StaffingModel
LeadingIndicator3.4–CommunityResources
LeadingIndicator3.3–
SchoolAlignment
LeadingIndicator3.2–
FamilyEngagement
LeadingIndicator2.1–AcademicPress
LeadingIndicator1.2–
YouthGovernance
LeadingIndicator3.1–SystemNorms
LeadingIndicator1.4–Enrollment
Policy
LeadingIndicator1.2–Con4nuousImprovement
PointofServiceSeOng
Organiza4onalSeOng
PolicySeOng
Con%nuousImprovementPrac%cesforSiteTeams
QualityInstruc%on&ProximalChild
Outcomes
LowStakesAccountabilityandSupports
Why Were the Leading Indicators Developed?
Why Were the Leading Indicators Developed?
• Isn’t this just more data?
13 composite measures categorized into five different contexts:
- Organizational Context - Instructional Context - External Relationships - Youth Characteristics - Family Satisfaction
What Are the Leading Indicators? WhataretheLeadingIndicators?
Wheredidtheycomefrom?
- Grantee Director/Site Coordinator Surveys - Afterschool teacher/Youth Worker Surveys
- Youth Surveys (grades 4-12)
- Parent Surveys
- PPICS data
Howdowemeasurethem?
- Youth Program Quality Intervention (YPQI) - California Outcomes Measures (Vandell)
- PPICS data
• Sample Report What Are the Leading Indicators?
What Are the Leading Indicators?
A
C
B
• Itemsaresimplythequestionsthatweaskonthesurveys,demographicandenrollmentdata,orYouthPQAscores.OnFigure1below,theItemscorrespondwithletterA. • ScalesaremadeupofgroupingsofdifferentItemsthatgotogetherwell.AScaleisdesignatedbyletterBinFigure1below. • LeadingIndicatorsaremadeupofgroupingsofdifferentScales,muchliketheScalesthemselvesaremadeupofItems.Intheexamplebelow,the“Accountability”Scale(alongwith“Collaboration”)makeuptheLeadingIndicator3.1–SystemNorms,whichisrepresentedbyletterCinFigure2below.[JB1] • Finally,alloftheLeadingIndicatorsaregroupedintoLiveoverarchingDomainsbasedonthecontextthattheyrepresent.TheseDomainsarecolor‐codedforeasydistinction,andinclude:OrganizationalContext(red),InstructionalContext(green),ExternalRelationships(blue),YouthCharacteristics(purple)andParentSatisfaction(brown).
Figure1
Figure2
Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
What Are the Leading Indicators?
How Have the Leading Indicators Been Used? OklahomaExemplar–SystemCharacteris4cs
• History of the system • Integration of QIS and required evaluation
efforts • 75 grantees in the first year, 77 this year • Timeline • Data Collection Methods • Outputs
Site-Level Improvement
• The report is… – A tool to help you identify the
strengths of your program – A tool to help you identify the
weaknesses of your program
• The report is not… – A mechanism to induce
evaluative comparisons or competitions across grants
– Something to be scared of
Site-Level Improvement
• Get a feel for the layout of the report • Study the graphs - In what areas are you
doing comparatively well? In what areas does it look like your site could improve?
• Celebrate your strengths • What could you work on? • Do some thinking • Prepare to make a plan!
Howtoreadandinterpretyourreport
Site-Level Improvement
Priority Assessment Form: Leading Indicators
1. Create the story of your data (column one) – What is the message or story of your data?
What do the numbers tell you? – What’s missing from the data? What
important things about program quality do not come through?
– Where are the gaps between what you want to provide and what the data says you’re providing?
2. Brainstorm ideas for improvement (column two)
Site-Level Improvement
The profiles (clusters) in Figure A-8 may be interpreted as follows. • Cluster 3: “High quality”
Thirty-three percent of grantees fall into Cluster 3, where programs show high quality in all areas.
• Cluster 2: “High with low -- growth/mastery and family communication” This cluster represents 24% of grantees. These programs show relatively high quality in most areas, but low school alignment and parent communication.
• Cluster 1: “Medium” Cluster 1 represents 21% of grantees. These programs have medium levels of supervision quality, high academic press & school alignment, low program quality (growth & mastery), and low family communication.
• Cluster 4: “Low with high -- school alignment & family communication” Four percent of grantees fall into Cluster 4, where programs show low quality in supervision, growth & mastery, and academic press, but high school alignment & family communication.
• Cluster 5: “Low quality” Eleven percent of grantees fall into Cluster 5, where programs appear to demonstrate low quality in all areas.
System-Level Planning
Intermediate & Academic Outcomes
• Table A-8 provides means of the satisfaction variables across cluster groups. The omnibus ANOVA indicates that the clusters produce significantly different scores in the Intermediate Outcomes set (staff job satisfaction, parent satisfaction, and parent reports of the program supporting academics). The highest quality group of grantees/sites (Cluster 3) produces the highest staff satisfaction, youth engagement, and academic efficacy, whereas the lowest quality group of grantees/sites (Cluster 5) exhibits the lowest or nearly the lowest in each area.
Table A-8 Means Scores for Intermediate Outcomes by Level of Quality
System-Level Planning
Cluster Staffjobsa%sfac%on
(S) ParentSa%sfac%on
(P) YouthEngagement
(Y) Homework
comple%on(Y) Prog.Supportsacademics(P)
3:Highquality 4.5 4.7 4.0 4.4 4.2
2:Highwithlow 4.2 4.7 3.8 4.0 3.9
1:Mediumquality 4.2 4.6 3.8 4.3 3.8
4:Lowwithhigh 3.5 4.8 3.7 4.2 4.2
5:Lowquality 3.6 4.5 3.7 4.2 3.6 Omnibusdifferenceacrossclusters(ANOVAF)
8.3***
2.6*
1.4
2.3+
5.9***
Where Have the Leading Indicators Been Used?
Looking Forward/Next Steps
• Further Validation Work – Exploration of Leading Indicators Framework –
Theoretical and Statistical
• Expansion to New Networks • Integration of Quality Improvement Systems
Questions?
Thank You!
Top Related