Call in number 888-674-0222 or 201-604-0498

79
1 Call in number 888-674-0222 or 201-604-0498 Improving the Quality of Improving the Quality of Child Outcome Data Child Outcome Data Materials at www.the-eco- center.org

description

Welcome to the National ECO TA Call. Improving the Quality of Child Outcome Data. Call in number 888-674-0222 or 201-604-0498. Materials at www.the-eco-center.org. Reminder. ECO looking for states to partner in framework development activities - PowerPoint PPT Presentation

Transcript of Call in number 888-674-0222 or 201-604-0498

Page 1: Call in number 888-674-0222  or  201-604-0498

1

Call in number

888-674-0222

or

201-604-0498

Improving the Quality of Improving the Quality of Child Outcome DataChild Outcome Data

Materials at www.the-eco-center.org

Page 2: Call in number 888-674-0222  or  201-604-0498

2

Reminder

• ECO looking for states to partner in framework development activities

• Call for states interested in the partner state application on March 20, 3 pm EDT/ 2 p.m.CDT/1 p.m. MDT/Noon PDT.

• See www.the-eco-center.org for application and call in information.

Page 3: Call in number 888-674-0222  or  201-604-0498

3

Today’s Presenters

• Christina Kasprzak, ECO at FPG

• Lynne Kahn, ECO at FPG

• Kathy Hebbeler, ECO at SRI

• Lisa Backer, Minnesota

Page 4: Call in number 888-674-0222  or  201-604-0498

4

To ask a question during the presentation

• Use the chat box

• If you can’t see the chat box, click on the triangle in front of “Chat” to expand the box

• Type your question in the box “Type chat message here”

• Send to All Participants.

Page 5: Call in number 888-674-0222  or  201-604-0498

5

Key to Good Data

Have a good outcome

measurement

Page 6: Call in number 888-674-0222  or  201-604-0498

6

Examples of Components of an Outcomes Measurement System

• Data collection procedures• Professional development around data collection

--- and data analysis• Ongoing supervision and monitoring of data

collection

• Ongoing analyses to check on the quality of the data

• Etc.

Page 7: Call in number 888-674-0222  or  201-604-0498

7

Building quality into your outcomes measurement system

• Occurs at multiple steps

• Requires multiple activities

Page 8: Call in number 888-674-0222  or  201-604-0498

8

Building quality into your outcomes measurement system

• Keep errors from occurring in the first place

• Develop mechanisms to identify weaknesses that are lessening the quality of the data

• Provide ongoing feedback including reports of the data to programs and providers

Page 9: Call in number 888-674-0222  or  201-604-0498

9

Different approaches present different kinds of challenges to quality data

• For states using COSF– Are all professionals trained in the process?– Are all professionals applying the rating criteria

consistently?• For states deriving OSEP data from an

assessment– Are all professionals trained in the assessment and

administering it properly?– Are the appropriate items/domains being used for

each outcome? – Are the appropriate “cut points” or criteria for age

appropriate and moved nearer to same age peers being used?

Page 10: Call in number 888-674-0222  or  201-604-0498

10

Today’s Focus: Using data analysis to check on the quality of your data

• Remember this is only weighing the pig

• Weighing the pig does not make it fatter

• Need to take what you learn from the analysis and do something with it.

Page 11: Call in number 888-674-0222  or  201-604-0498

11

Child Outcomes Data Quality

• So what do you look at to know?

• Our game plan– Walk through a series of expected patterns

and look at the corresponding analyses– These data are being shared as a teaching

tool. Do not cite the data.– Do consider the analyses as a way to

examine your own state data.

Page 12: Call in number 888-674-0222  or  201-604-0498

12

THIS IS A DATA “SAFE ZONE”…

Page 13: Call in number 888-674-0222  or  201-604-0498

13

Using data for program improvement = EIA

Evidence

Inference

Action

Page 14: Call in number 888-674-0222  or  201-604-0498

14

Evidence

• Evidence refers to the numbers, such as

“45% of children in category b”

• The numbers are not debatable

Page 15: Call in number 888-674-0222  or  201-604-0498

15

Inference• How do you interpret the #s?

• What can you conclude from the #s?

• Does evidence mean good news? Bad news? News we can’t interpret?

• To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

Page 16: Call in number 888-674-0222  or  201-604-0498

16

Inference• Inference is debatable --

even reasonable people can reach different conclusions from the same set of numbers

• Stakeholder involvement can be helpful in making sense of the evidence

Page 17: Call in number 888-674-0222  or  201-604-0498

17

Action

• Given the inference from the numbers, what should be done?

• Recommendations or action steps

• Action can be debatable – and often is

• Another role for stakeholders

Page 18: Call in number 888-674-0222  or  201-604-0498

18

Quality Checks

• Missing Data

• Pattern Checking

Page 19: Call in number 888-674-0222  or  201-604-0498

19

Missing Data - Overall

• How many children should the state be reporting to OSEP in the SPP/APR table?– i.e., how many children [had entry data,] exited in the

year, and stayed in the program 6 months?– Do you have a way to know?

• What percentage of those children do you have in the table?

• These questions apply whether or not you are sampling.

Page 20: Call in number 888-674-0222  or  201-604-0498

20

Are you missing data selectively?

• By local program

• By child characteristic – Disability?– Type of exit? (children who exit before 3)

• By family characteristic– Families who are hard to reach (and may

leave unexpectedly)

***Which of these can you check on?***

Page 21: Call in number 888-674-0222  or  201-604-0498

21

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

Page 22: Call in number 888-674-0222  or  201-604-0498

22

Pattern Checking

3 Possible Sets of Numbers1. OSEP Progress Categories

2. Entry Data

3. Exit Data

Page 23: Call in number 888-674-0222  or  201-604-0498

23

OSEP Progress Categories

a. Did not improve functioning.

b. Improved functioning but not enough to move closer to same-age peers.

c. Improved functioning to a level nearer to same-age peers but did not reach it.

d. Improved functioning to reach a level comparable to same-age peers.

e. Maintained functioning at a level comparable to same-age peers.

Page 24: Call in number 888-674-0222  or  201-604-0498

24

Looking for Sensible Patterns in the Data

• Putting together your “validity argument.”

• You can make a case that your data are valid if …..they show certain patterns.

• The quality of your data is not established by one or two numbers.

• The quality of the data is established by a series of analyses that demonstrate the data are showing predictable patterns.

Page 25: Call in number 888-674-0222  or  201-604-0498

25

“Invalid Outcomes Data?”

Page 26: Call in number 888-674-0222  or  201-604-0498

26

Predicted Pattern #1

1a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle). .

1b. Children will differ from one another in their exit scores in reasonable ways.

1c. Children will differ from one another in their OSEP progress categories in reasonable ways.

Page 27: Call in number 888-674-0222  or  201-604-0498

27

Rationale

Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).

Page 28: Call in number 888-674-0222  or  201-604-0498

28

Predicted Pattern #1 (cont’d)

Analysis1. Look at the distribution of rating/scores at

entry and exit and the data reported to OSEP.

2. Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit.

Question: Is the distribution sensible? What do you expect to see?

Page 29: Call in number 888-674-0222  or  201-604-0498

29

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

Page 30: Call in number 888-674-0222  or  201-604-0498

30

Entry &

Exit Data

Page 31: Call in number 888-674-0222  or  201-604-0498

31

MN: Outcome 1 Entrance: 07-08

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5 6 7

Part C Part B

Page 32: Call in number 888-674-0222  or  201-604-0498

32

State with Scores: Distribution of entry scores on Outcome 1

Page 33: Call in number 888-674-0222  or  201-604-0498

33

MN: Outcome 2 Exit: 07-08

0

200

400

600

800

1000

1200

1 2 3 4 5 6 7

Part C Part B

Page 34: Call in number 888-674-0222  or  201-604-0498

34

OSEP Categories

Page 35: Call in number 888-674-0222  or  201-604-0498

35

MN: Outcome 3 OSEP Categories: 07-08

0%

5%

10%

15%

20%

25%

30%

35%

40%

A B C D E

Part C Part B

Page 36: Call in number 888-674-0222  or  201-604-0498

36

Fake Data: OSEP progress categories

• Possible Problems:

– Too many children in “a”

– Too many children in “e”

20

15 15

25 25

0

5

10

15

20

25

30

a b c d e

310 10

15

60

0

10

20

30

40

50

60

70

a b c d e

Page 37: Call in number 888-674-0222  or  201-604-0498

37

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

Page 38: Call in number 888-674-0222  or  201-604-0498

38

Predicted Pattern #2

2. Functioning in one outcome area will be related to functioning in the other outcome areas.

Analyses: Look at the relationship across the outcomes at entry, at exit, across the OSEP progress categories. 1. Crosstabs 2. Correlation coefficient

Question: What do we expect to see?

Page 39: Call in number 888-674-0222  or  201-604-0498

39

Rationale

For many, but not all, children with disabilities, progress in functioning in the

three outcomes proceeds together

Page 40: Call in number 888-674-0222  or  201-604-0498

40

Outcome 1

Outcome 2 1 2 3 4 5

1

2

3

4

5

Page 41: Call in number 888-674-0222  or  201-604-0498

41

MN: Crosstabulation with Progress Categories: 619: Know/Skills to Soc/Emot

AA BB CC DD EE

AA 14 4 3 1

BB 5 230 116 36 67

CC 6 102 664 172 37

DD 45 227 517 179

EE 44 53 52 186 440

Page 42: Call in number 888-674-0222  or  201-604-0498

42

Correlation Coefficient

• Useful statistic

• Range: 0 to 1

• Can be negative

• Measure of extent of a relationship between 2 sets of numbers

• Closer to 1, stronger the relationship

• Negative correlation means as one set of numbers goes up, the other goes down.

Page 43: Call in number 888-674-0222  or  201-604-0498

43

MN619: Correlation coefficients between exit scores for the 3 outcomes (N=3,160)

 

Soc Emot_exit Knowledge_exit

 

Knowledge_exit

.724

Action_exit

.737 .691

Page 44: Call in number 888-674-0222  or  201-604-0498

44

MNPart C: Correlation coefficients among entry scores for the 3 outcomes

 

Soc Emot_entry Knowledge_entry

 

Knowledge_entry

.660**

Action_entry

.604** .594**

**. Correlation is significant at the 0.01 level (2-tailed).

Page 45: Call in number 888-674-0222  or  201-604-0498

45

Predicted Pattern #3

3. Functioning at entry within an outcome area will be related to functioning at exit (or – children who have higher functioning at entry in an outcome area will be the ones who are high functioning at exit in that outcome area).

Analyses: 1. Correlation coefficients between entry and exit scores for each outcome2. Crosstabs between entry and exit scores for each outcome

Question: What do we expect to see?

Page 46: Call in number 888-674-0222  or  201-604-0498

46

MN 619: O2 Entry X Exit Ratings

1 2 3 4 5 6 7

1 11 4 1 2 0 0 0

2 16 50 5 4 2 0 0

3 33 108 68 15 7 3 0

4 18 121 151 70 14 5 2

5 21 148 213 233 170 18 12

6 16 70 138 172 282 217 30

7 3 15 39 47 130 219 257

Page 47: Call in number 888-674-0222  or  201-604-0498

47

MN Part C: Correlation coefficients between entry and exit scores

 O1_exit O2_exit O3_exit

O1_entry.660 .505 .499

O2_entry.539 .612 .491

O3_entry.512 .471 .649

**. Correlation is significant at the 0.01 level (2-tailed).N=1,060  

Page 48: Call in number 888-674-0222  or  201-604-0498

48

“Any Requests?

Page 49: Call in number 888-674-0222  or  201-604-0498

49

Predicted Pattern #4

4. Most children will either hold their developmental trajectory or improve their trajectory from entry to exit.

Analyses: 1. Comparison of distributions of COSF ratings, standard scores, or some other metric that takes age into account. (Why can’t we use raw scores on an assessment for this?) at entry and exit.Question: What do we expect to see?

Page 50: Call in number 888-674-0222  or  201-604-0498

Illustration of 5 Possible Develomental Trajectories

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peersAchieved functioning comparable to age peersMoved nearer functioning comparable to age peersMade progress; no change in trajectoryDid not make progress

Page 51: Call in number 888-674-0222  or  201-604-0498

51

Entry & Exit Ratings MN: C-O2

0

50

100

150

200

250

300

1 2 3 4 5 6 7

Entry Exit

Page 52: Call in number 888-674-0222  or  201-604-0498

52

Entry & Exit Ratings MN: B-O1

0

50

100

150

200

250

300

1 2 3 4 5 6 7

Entry Exit

Page 53: Call in number 888-674-0222  or  201-604-0498

53

Predicted Pattern #4b

4b. Children will not show huge changes in a year (or between entry and exit??).

Analyses:

1. Time 2 scores minus Time 1 scores 2. 2. Crosstabs of scores at each time point

Question: What do we expect to see?

Page 54: Call in number 888-674-0222  or  201-604-0498

54

Distribution: Exit - Entrance Ratings Minnesota Part B Know/Skills n=3160

0

200

400

600

800

1000

1200

-3 -2 -1 0 1 2 3 4 5 6

Page 55: Call in number 888-674-0222  or  201-604-0498

55

MN Part C: O3 Entry X Exit

1 2 3 4 5 6 7

1 5 2 3 1 0 1 0

2 18 32 13 3 3 1 0

3 23 59 40 14 14 2 2

4 6 34 67 42 15 5 1

5 5 16 49 49 66 20 1

6 3 17 30 28 80 94 13

7 2 5 18 10 27 48 73

Page 56: Call in number 888-674-0222  or  201-604-0498

56

Predicted Pattern #5

5. Entry, exit, and OSEP progress category distributions from year to year should be similar (assuming the same kinds of children are being served).

Analysis: 1. Frequency distributions of entry data in 2007, 2008, etc. 2. of exit data3. of OSEP Categories

Question: What do we expect to see?

Page 57: Call in number 888-674-0222  or  201-604-0498

57

Entry Ratings MN: B-O1

0%

5%

10%

15%

20%

25%

1 2 3 4 5 6 7

2005-06 2006-07 2007-08

Page 58: Call in number 888-674-0222  or  201-604-0498

58

Exit Ratings MN: C-O2

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4 5 6 7

2006-07 2007-08

Page 59: Call in number 888-674-0222  or  201-604-0498

59

OSEP Progress Categories MN: B-O2

0%

5%

10%

15%

20%

25%

30%

35%

A B C D E

2006-07 2007-08

Page 60: Call in number 888-674-0222  or  201-604-0498

60

Predicted Pattern #6

6. If local areas are serving similar kinds of children, scores at entry should be similar.

Analysis:

1. Frequency distributions of entry by local areas (Use the big programs.)

2. Means and standard deviations (and Ns!) by local area.

Question: What do we expect to see?

Page 61: Call in number 888-674-0222  or  201-604-0498

61

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

Page 62: Call in number 888-674-0222  or  201-604-0498

62

B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

0%

20%

40%

60%

80%

100%

R A S M

7654321

Page 63: Call in number 888-674-0222  or  201-604-0498

63

B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

0%

20%

40%

60%

80%

100%

R A S M

At age levelBelowSig. Below

Page 64: Call in number 888-674-0222  or  201-604-0498

64

Predicted Pattern #7

7. Entry and exit scores and OSEP categories should be related to the nature of the child’s disability.

Analyses: 1. Frequency distributions for each disability group2. Means and standard deviations for each disability group

Question: What do we expect to see?

Page 65: Call in number 888-674-0222  or  201-604-0498

65

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

Page 66: Call in number 888-674-0222  or  201-604-0498

66

MN: Entry Ratings by Disability: Part B, Soc/Emot Skills

0%

20%

40%

60%

80%

100%

DD S/ L ASD EBD

7654321

Page 67: Call in number 888-674-0222  or  201-604-0498

67

MN: Entry Ratings by Disability: Part B, Know/Skills

0%

20%

40%

60%

80%

100%

DD S/ L ASD EBD

7654321

Page 68: Call in number 888-674-0222  or  201-604-0498

68

MN: Mean and Standard Deviation by Disability

Soc-Emot. Soc-Emot. skillsskills

Knowledge/Knowledge/SkillsSkills

Action to MeetAction to Meet NeedsNeeds

ASDASD 2.88

1.15

3.63

1.48

3.56

1.38

DDDD 3.43

1.35

3.44

1.27

4.09

1.47

Sp/LangSp/Lang 5.24

1.44

5.25

1.62

5.84

1.66

Page 69: Call in number 888-674-0222  or  201-604-0498

69

Predicted Pattern #8

8. Scores at entry (and exit) should not be related to certain characteristics (e.g., race/ethnicity).

Analyses: 1. Frequency distributions for each group2. Means and standard deviations for group

Question: What do we expect to see?

Page 70: Call in number 888-674-0222  or  201-604-0498

70

MN Pt B: Outcome 2 Entry Ratings by Gender

0

5

10

15

20

25

1 2 3 4 5 6 7

BoysGirls

Page 71: Call in number 888-674-0222  or  201-604-0498

71

MN: Mean & SD by Race/EthnicityRaceRace Soc-EmotSoc-Emot Know/SkillsKnow/Skills Action/NeedAction/Need

American American Indian Indian n=24n=24

3.00

1.45

4.00

1.38

4.00

1.52

Asian/Pac. Asian/Pac. Islander Islander n=52n=52

4.04

1.55

3.83

1.61

4.37

1.73

HispanicHispanicn=81n=81

3.60

1.62

3.48

1.53

4.17

1.79

BlackBlackn=95n=95

3.38

1.50

3.95

1.35

3.47

1.54

WhiteWhiten=907n=907

4.07

1.72

3.96

1.67

4.11

1.72

Page 72: Call in number 888-674-0222  or  201-604-0498

72

Wrap-up

Page 73: Call in number 888-674-0222  or  201-604-0498

73

Page 74: Call in number 888-674-0222  or  201-604-0498

74

Drilling down: Looking at data by local program

• All analyses that can be run with the state data can be run with the local data

• The same patterns should hold and the same predictions apply.

• Need to be careful about the size of N with small programs.

Page 75: Call in number 888-674-0222  or  201-604-0498

75

Are your data high quality?

1. Are the missing data less than X% with no systematic biases?

• Systematic bias = some LEA/EIS or sub-groups are missing far more data than others (you have non-representative data).

2. Do your state’s data support the predicted patterns?

• If not, where are the problems?• What do you know or can you find out about

why they are occurring?

Page 76: Call in number 888-674-0222  or  201-604-0498

76

Adapting the definition of insanity…

“The definition of insanity is doing the same thing over and over again and expecting different results.”

Einstein

Page 77: Call in number 888-674-0222  or  201-604-0498

77

Data Insanity

…..is doing nothing over and over again and expecting your data to get better.

Page 78: Call in number 888-674-0222  or  201-604-0498

78

Achieving high quality data is a process that takes time – and intentional action

Quality of Child Outcomes Data

Time

Page 79: Call in number 888-674-0222  or  201-604-0498

79

Reminder

• ECO looking for states to partner in framework development activities

• Call for states interested in learning about the partner state application on March 20, 3 pm EDT/2 p.m.CDT/1 p.m. MDT/Noon PDT.

• See www.the-eco-center.org for application and call-in information.