The Current Status of States' Early Childhood Outcome Measurement Systems

Kathy Hebbeler, SRI International

Lynne Kahn, FPG Child Dev Inst

October 17, 2009

DEC's 25th Annual International Conference on Young Children with Special Needs and Their Families 1

We’ve come a long way, BABY!

2

In the Beginning- Federal Accountability

• 1994 Government Performance and Results Act (GPRA)

• 2002 Performance Assessment Rating Tool (PART)– Both Part C and Part B Preschool categorized as

“Results Not Demonstrated” due to lack of outcome data.

– OMB recommended OSEP develop a strategy for collecting outcome data

The Development of Outcome Statements

• 2004-early 2005: ECO generates discussion and gathers input on child and family outcomes

• Summer 2005: OSEP announces the child and family outcomes States must report on through their SPP/APRs

Early Childhood Outcomes Center4

Three Child Outcomes

Percent of children who demonstrate improved:– Positive social emotional skills (including

positive social relationships)– Acquisition and use of knowledge and skills

(including early language/ communication [and early literacy])

– Use of appropriate behaviors to meet their needs

Early Childhood Outcomes Center5

Key Concepts to Communicate

• Outcomes are functional– term has multiple definitions

• Each outcome is integrated across domains

• Challenge: Reporting language does not clearly define the outcomes

Early Childhood Outcomes Center6

Reporting Child Progress

• December 2005: States submit plans via SPP on how they will collect outcome data

• September 2006: OSEP finalizes the child outcome reporting categories

Early Childhood Outcomes Center7

Child Progress- the % of children who

a. did not improve functioning

b. improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers

c. improved functioning to a level nearer to same-aged peers but did not reach it

d. improved functioning to reach a level comparable to same-aged peers

e. maintained functioning at a level comparable to same-aged peers

Early Childhood Outcomes Center8

Key Concepts to Communicate

• Progress categories require 2 data points for each child,

• are based on growth trajectories,

• compare a child to him or herself over time, and also compare each child to age expectations

Early Childhood Outcomes Center9

Early Childhood Outcomes Center

10

Entry Exit

States Submit Data to OSEP in their Performance Reports

• Feb 2007: “status at entry” data

• Feb 2008: child progress data for children who exited 7/1/06 through 6/30/07

• Feb 2009: child progress data for children who exited 7/1/07 through 6/30/08

Early Childhood Outcomes Center11

State Approaches to Measuring Child Outcomes

Approach

Part C

(56 states/jur)

Preschool

(59 states/jur)

One tool statewide 7/56 (13%) 9/59 (15%)

Publishers’ online analysis

3/56 (5%) 6/59 (10%)

COSF 7 pt. scale 41/56 (73%) 38/59 (64%)

Other 5/56 (9%) 7/59 (10%)

Number of Children Included in Feb ‘09 SPP/APR Data

Part C (56)

Range: 5-6452

<30 = 3

30-99= 10

100-499= 25

500-999= 6

1000+ = 12

Preschool (59)

Range: 3-10157

<30 = 3

30-99 = 9

100-499= 14

500-999= 10

1000+ = 22

Key Concepts to Communicate

• Meaningfulness of the progress categories

• Establishing expectations for how the percentages “should” look

• Exploring relationships between state eligibility definitions and expectations for outcomes

Early Childhood Outcomes Center18

Setting Targets for Improving Child Outcomes

• 2008 and 2009: State input into summary statements to be the basis of target setting

• Currently: summary statements have been out for public comment, and are in draft form in SPP Measurement tables

• February 2010: Baseline data and target setting on summary statements

Summary Statement Data

• Required Summary Statement 1:

Of those children who entered or exited the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.

•  Required Summary Statement 2:

The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.

20

Key Concepts to Communicate

• The meaning of the summary statements

• Ensuring that data is clean before making program changes based on it

• Choosing/ developing strategies that will improve child outcomes

Early Childhood Outcomes Center21

So where are the states now?

States and Outcomes Measurement

• Many states have embraced the ideas of measuring outcomes– Want information for own purposes– Program improvement

• The power of GOOD outcomes data is widely recognized

States and Outcomes Measurement

• States have invested resources and time in building an outcomes measurement system.– Involving stakeholders– Identifying an approach– Training providers in approach– Building a data system

And collecting data!

States have made incredible progress

• States started at various times

• Have varying amounts of data

• Examining their data (and procedures)– The Quality Question??

This is VERY DIFFICULT work

Some of the Challenges

• Need for good assessment tools– Need for more providers skilled in assessment

• Need strong grounding in typical child development• Adequate training statewide in data collection

procedures– Understanding and commitment

• Resources for new data systems • Capability to analyze, report and interpret data

– Generating reports for local programs

• Resources in general

Next steps:

Putting it all together

Early Childhood Outcomes Center28

Making Good Use of Data: Making Good Use of Data: Some Necessary Ingredients Some Necessary Ingredients

• Set of good questions

• Data set with the required elements

• Analytic capability to analyze the data to address the question

• Commitment to use the information for program improvement

Examining and Tweaking the Service System

Plan (vision) Program characteristics

Child and family outcomes

Implement

Check(Collect and analyze data)

ReflectAre we where we

want to be?

Is there a problem?

Why is it happening?

What should be done?

Is it being done?

Is it working?

Early Childhood Outcomes Center30

Information Infrastructure: Information Infrastructure: Data Needed for Program ImprovementData Needed for Program Improvement

WHO SERVICES

COSTQUALITY

OUTCOMES

Early Childhood Outcomes Center31

Where is your state (or program) now?Where is your state (or program) now?

WHO SERVICES

OUTCOMES

What do you have?

How much is linked?

WHOSERVICES

COSTQUALITY

OUTCOMES

WHOSERVICES

COST

QUALITY

OUTCOMES

WHOSERVICES

COSTQUALITY

OUTCOMES

PreschoolPreschool

KindergarteKindergartenn

Grade 1Grade 1

Grade 2Grade 2

WHOSERVICES

COSTQUALITY

OUTCOMES

Tracking outcomes over time

An outcomes measurement system

is more than hardware, software, and data

Framework Development

• ECO is working with 7 partner states over the next two years to develop a framework for outcomes measurement.

• Framework provides the basis for a self assessment to allow states to track their own progress in building an outcomes measurement system.

Areas of DRAFT Outcomes Measurement System Framework

Data Collection Data Collection and and

TransmissionTransmission

Data Collection Data Collection and and

TransmissionTransmission AnalysisAnalysisAnalysisAnalysis Information Information SharingSharing

Information Information SharingSharing ActionActionActionAction

Vision for What Is to Be LearnedVision for What Is to Be LearnedVision for What Is to Be LearnedVision for What Is to Be Learned

EvaluationEvaluationEvaluationEvaluation

Data Collection and Data Collection and TransmissionTransmission

• State has a method capable of accurately capturing outcomes information

• State has effective procedures for professional development around data collection.

• State has effective procedures for moving data from the local to the state level

• Etc.

State Self Assessment

• Is each area of the outcomes measurement system implemented?

• Is the area implemented in a way that reflects high quality?

We’ve come a long way, BABY!

39

…and we still have a long way to go…

...but won’t it be great when we get there?

For more information

(and for a copy of this presentation)

www.the-eco-center.org