The Results are In: Using Early Childhood Outcome Data Kathy Hebbeler Early Childhood Outcomes...
-
date post
19-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of The Results are In: Using Early Childhood Outcome Data Kathy Hebbeler Early Childhood Outcomes...
The Results are In: Using Early Childhood Outcome Data
Kathy HebbelerEarly Childhood Outcomes Center at SRI International
August, 2011
What we will cover
• Quick review of the reporting requirement and state approaches
• Share the national data• Describe how the national data were computed• Discuss the quality of the national data• Discuss the meaning of the numbers• Framework and self-assessment (maybe…)
2Early Childhood Outcomes Center
3Early Childhood Outcomes Center
OSEP Reporting Requirements: Child Outcomes
• Positive social emotional skills (including positive social relationships)
• Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])
• Use of appropriate behaviors to meet their needs
4Early Childhood Outcomes Center
OSEP Reporting Categories
Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to move nearer
to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged
peers but did not reach itd. Improved functioning to reach a level comparable to
same-aged peerse. Maintained functioning at a level comparable to same-
aged peers
3 outcomes x 5 “measures” = 15 numbers
Illustration of 5 Possible Paths
0
10
20
30
40
50
60
70
1 6 11 16 21 26 31 36 41 46 51 56
Age in Months
Sco
re
Maintained functioning comparable to age peers
Achieved functioning comparable to age peers
Moved nearer functioning comparable to age peers
Made progress; no change in trajectory
Did not make progress
The Summary Statements
1. Of those children who entered the program below
age expectations in each outcome, the percent who
substantially increased their rate of growth by the
time they turned 3 years of age or exited the
program.
2. The percent of children who were functioning within
age expectations in each outcome by the time they
turned 3 years of age or exited the program.
6Early Childhood Outcomes Center
State Approaches to Outcomes Data
Approach Part C (56
states/jurisdictions)
Preschool(59
states/jurisdictions)
COS 7 pt. scale 41/56 (73%) 37/59 (63%)
One tool statewide 7/56 (13%) 9/59 (15%)
Publishers’ online analysis
3/56 (5%) 6/59 (10%)
Other 5/56 (9%) 7/59 (12%)
Early Childhood Outcomes Center 8
Note: Based on 29 States with highest quality data
Social relationships Knowledge and skills Action to meet needs
a 0.0165481765021159 0.0148974177446063 0.0145225115393538
b 0.179791044501508 0.199542744006336 0.174681665026758
c 0.185932320590048 0.247993957138401 0.214074779812331
d 0.294138593838746 0.369311305766878 0.370708975497373
e 0.323587775084597 0.168236271568793 0.226016769460911
3%
8%
13%
18%
23%
28%
33%
38%
Estimated Data for Part C, 2009-10
Early Childhood Outcomes Center 9
Note: Based on 29 States with highest quality data
Social relationships Knowledge and skills Action to meet needs
SS1 0.70973347275746 0.742180533377054 0.755546347980436
SS2 0.61772636892334 0.537547577335669 0.596725744958282
5%
15%
25%
35%
45%
55%
65%
75%
Estimated Summary Statement Data for Part C, 2009-10
Early Childhood Outcomes Center 10
Social relationships Knowledge and skills Action to meet needs
a 0.0173423475321513 0.0182367034048671 0.0164717096861928
b 0.114520004403404 0.133727833654954 0.108153036923834
c 0.281561785450637 0.32470410889358 0.208844242808853
d 0.343437919521324 0.34397760817479 0.356458315050575
e 0.243152978902516 0.179335862613809 0.310081127789363
3%
8%
13%
18%
23%
28%
33%
38%
Estimated National Data for Early Childhood Special Education, 2009-2010
Note: Based on 33 States with highest quality data
11Early Childhood Outcomes Center
Note: Based on 33 States with highest quality data
Social relationships Knowledge and skills Action to meet needs
SS1 0.825777563121165 0.81482333429858 0.819365394291995
SS2 0.586590898423841 0.523313470788598 0.666539442839937
10%
30%
50%
70%
90%
Estimated National Summary Statements for Early Childhood Special Education, 2009-2010
Criteria for States with Quality Data
1. Low percentage of missing data
2. No odd patterns in “a” or “e” categories
3. Did not use questionable data collection methods
12Early Childhood Outcomes Center
Calculating Missing Data for Part C
Proxy for missing data =
Number with data for C3/
Exiting Data (618)
13Early Childhood Outcomes Center
• Do not expect this number to be 100%
• ..but we don’t expect it to be 10% either
Part C: Percent of Exiters included in Outcomes Data
08-09
<10% = 10*
10- 20% = 4
20- 30% = 8
30- 40% = 11
40- 50% = 8
50- 60% = 8
60- 70% = 4
70- 80% = 2
>80% = 1
09-10
<10% = 5*
10- 20% = 4
20- 30% = 6
30- 40% = 8
40- 50% = 5
50- 60% = 11
60- 70% = 9
70- 80% = 1
>80% = 0*3 states are sampling for Part C. Excluded states with <28%.
Calculating Missing Data for 619
Proxy for missing data =
Number with data for B7/
Child count
15
• Do not expect this number to be 100%
• ..but we don’t expect it to be 10% either
Percent of Child Count included in Outcomes Data for ECSE
08-09
<10= 11*
10- 20%= 15
20- 30%= 12
30- 40%= 12
40-50% =1
>50% = 2
09-10
<10= 6*
10- 20%= 11
20- 30%= 12
30- 40%= 16
40-50% =4
>50%= 0
*4 States are sampling for 619Excluded states with <12% of child count
Problem with Missing Data
• We don’t know how well the data the state has represent the entire state.
• If the data are representative, the percentages for the a to e Progress Categories and the Summary Statements won’t change as data are added on more children.
17Early Childhood Outcomes Center
Early Childhood Outcomes Center 18
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
0%
2%
4%
6%
8%
10%
12%
14%
16%
Percent Reported in "a“ for Knowledge and Skills for ECSE by State
Exclusion criteria: a<10% in any outcomes e<65% in any outcomes
Pattern checking for validity
• Checking across years–How do the 2009-10 compare to the
data for 2008-09?• Checking across methods
–How do the data for all states compare to states with highest quality data?
20Early Childhood Outcomes Center
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
64.570.2
64
71
SS1: % who IncreasedGrowth Rates
Part C, Outcome A: Social Relationships
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
6761.3
64.761.8
SS2: % who Exited at Age Expectations
Part C, Outcome B: Knowledge and Skills
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
8070.4
76.6
68.1
74.2
SS1: % who Increased Growth Rates
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
5953.9 55.8 53.8
SS2: % who Exited at Age Expectations
Part C, Outcome C: Meets Needs
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
69.7
76
68.1
75.6
SS1: % who Increased Growth Rates
All states 19 best All states 29 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
64.660.5 61.5 59.7
SS2: % who Exited at Age Expectations
Part B Preschool: Social Relationships
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
77.282.7
79.7 82.6
SS1: % who Increased Growth Rates
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
61.758.7
62.158.7
SS2: % who Exited at Age Expectations
Part B Preschool: Knowledge and Skills
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
76.282.7
78.381.5
SS1: % who Increased Growth Rates
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
55.851.2
55.552.3
SS2: % who Exited at Age Expectations
Part B Preschool: Meets Needs
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
75.381.6
78.181.9
SS1: % who Increased Growth Rates
All states 15 best All states 33 bestFFY 08-09 FFY 09-10
0
10
20
30
40
50
60
70
80
90
67.8 67.2 66.7 66.7
SS2: % who Exited at Age Expectations
What to these data tell us?
• Nationally, a high proportion of children who receive Part C and ECSE services are showing greater than expected progress
• Nationally, many (over half) are exiting the program functioning like same age peers in at least one of the outcomes.
29Early Childhood Outcomes Center
Should your state’s data look like the national data?
• Probably not• More important that each state continue to
focus on the quality of its own data– Getting outcomes data on all children who exit– Working with programs whose data look
unusual to address possible data quality issues
30Early Childhood Outcomes Center
Framework and Self-Assessment
• FRAMEWORK– Set of components and quality indicators– Provides the structure for the self-assessment
• SELF-ASSESSMENT – Scale that provides criteria for levels of
implementation within each quality indicator– Rating assigned based on level of
implementation within each indicator 32Early Childhood Outcomes Center
Purpose of the Self Assessment
• Assist states in setting priorities for improving their COMS
• Provide information to assist states in advocating for resources for systems development
33
• Provide guidance to states on what constitutes a high quality child outcomes measurement system.
Early Childhood Outcomes Center
Additional information
For information on improving data quality and using data for program improvement
www.the-eco-center.org
35Early Childhood Outcomes Center
Measuring and Improving Child and Family Outcomes ConferenceSeptember 18-21, 2011
New Orleans, LA