FosteringCourtImprovement.org

Data: Telling a Story …A Walk Through Meaningful Data

Analysis

Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results

Andy BarclayStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law

Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006

FosteringCourtImprovement.org2

Overview How Knowledge of the Past and Present

can Help you Shape the Future• Data, Data, Data Isn’t Yada, Yada, Yada• Court Performance Measures• AFCARS

The Easy Win vs. Sea Change Getting Real, Getting Local, Getting

Context Results that Matter to Kids & Families

FosteringCourtImprovement.org3

Shaping the Future

Why Data, Data, DataIsn’t

Yada, Yada, Yada

FosteringCourtImprovement.org4

Lessons learned or missteps repeated?

Lunchroom lore or documented results?

Shaping the Future

FosteringCourtImprovement.org5

Shaping the Future

“They always say that time changes things, but you actually have to change them yourself.”

Andy Warhol

FosteringCourtImprovement.org6

Do you know where your caseload has been?

Do you know where it is?

Do you know where it is going?

Shaping the Future

FosteringCourtImprovement.org7

Shaping the Future

Am I on the road to success or on a “frolic and detour”?

(Only the data know.)

FosteringCourtImprovement.org8

Shaping the Future

“Georgia Judge leads stampede out of homes.”

FosteringCourtImprovement.org9

Shaping the Future

“Illinois Judge ‘doesn’t have time’ to return children to parents.”

FosteringCourtImprovement.org10

Shaping the Future

Take off the blinders.Or

Load and Learn.

FosteringCourtImprovement.org11

Shaping the Future

Where to Start?

FosteringCourtImprovement.org12

Court Performance Measures

The three leading legal/ judicial organizations, with a grant from the Packard Foundation, have created and pilot-tested court performance measures that complement those used in the CFSR process so that agencies and courts can collaborate in the measurement of their performance and tracking of their own progress in improving safety, permanence, and timeliness for children coming before them.

FosteringCourtImprovement.org13

Court Performance Measures

(Building a Better Court) Reinforce current federal reform

initiatives, including CFSR and PIP’s Helps to identify areas of court reform

essential to better outcomes Assist the CIP by providing courts with

tools to engage in systematic process of court performance measurement

FosteringCourtImprovement.org14

Court Performance Measures

“Performance measures are the yardstick by which courts and child welfare agencies can measure the success they are achieving and the progress they are making.”

Building a Better Court

FosteringCourtImprovement.org15

Court Performance Measures

Safety: Are there new allegations while in foster care?

Permanency: What was the exit from foster care and how many came back?

Due Process: Is there notice and representation?

Timeliness: How long between hearings?

FosteringCourtImprovement.org16

Already Measurable and Measured (AFCARS)

Safety Permanency Timeliness

FosteringCourtImprovement.org17

AFCARS

Adoption and Foster Care Analysis and Reporting System

FosteringCourtImprovement.org18

Uses of AFCARS

Award Adoption Incentive bonuses Allot Chafee Independence $$$ Produce annual reports to Congress

on child welfare outcomes Conduct IV-E eligibility reviews Assess state’s compliance with federal

CFSR performance standards

FosteringCourtImprovement.org19

Platform for Collaboration

Longitudinal AFCARS provides a platform for rolling out the due process & timeliness measures recommended by Building a Better Court

Promotes working partnership between the courts, child welfare agencies and universities,

Provides 9 of the 24 measures defined in Building a Better Court, and 9 of the 30 in ToolKit.

FosteringCourtImprovement.org20

Fostering Court Improvement

What gets measured gets done,

IfYou know it’s being measured.

FosteringCourtImprovement.org21

Stories from Medicine : An Easy Win

Diagnostic Imaging (Cardiac PET/SPECT) Pre-Screen

Very simple questionnaire Predictive Value of the Test

• Surgery: y/n• Angioplasty: y/n

Easy sell to cardiologists Very easy sell to insurance

companies

Bayesian Pre-Screening Tool

FosteringCourtImprovement.org22

Stories from Medicine : A Sea Change

Major resistance, even when shown computer was right more often, lower mortality

Over time (3 years) older docs modified their risk tolerance, less invasive, better outcomes, learned from computer

Tripled business, more patients living longer, much lower costs ...

Automated Reading of Scans

FosteringCourtImprovement.org23

Court Stakeholders Have a Right to be Skeptical

In medicine, I see examples of better outcomes at lower cost quite often.

Have yet to see one in child welfare. Many claims, little evidence Judges, attorneys, CASAs, and caseworkers

have good reason to be skeptical:• Complex Business• Spectacular IT Failures• Terabytes of Unused Data

FosteringCourtImprovement.org24

Get Real Data reports are only meaningful to you in the

context of your job, your reality. We (FCI) have been successful* using data to

develop a shared context among local court stakeholder groups.

Very different jobs, very different realities, but we need to have some common understandings to do successful* work together (and occasionally alter some realities).

*successfully improve an outcome

FosteringCourtImprovement.org25

Diverse Realities: Dependency Court

Caseloads

3,140 US counties + DC

70% of counties remove less than 1 child per week

• 50% less than 1 per 2 weeks

7% of counties (234) average more than 1 child removal per business day

FosteringCourtImprovement.org26

Get Local Much as I’d like to, we

probably can’t do this with just a website.

• our web reports are not for a general audience.

County by county …• Judges convene• Georgia’s CIP buys lunch• I walk through a

Powerpoint of charts & numbers from the website.

FosteringCourtImprovement.org27

Broad Topic Areas of Court Stakeholder Meetings

Removal Rates Reasons for Removal Kids in Care Group Settings IV-E Reimbursement Kids Headed for

Adoption Reunification Rate Time to Adoption

FosteringCourtImprovement.org28

A Simple Temporal Context

Emphasize the Context for every number, nothing without Context.

Temporal: Interpret current measures in the context of the past.

FosteringCourtImprovement.org29

The Context of Comparable Jurisdictions

Emphasize the Context for every number, nothing without Context.

Comparables: Interpret in context of comparable counties.

FosteringCourtImprovement.org30

The Context of Statewide Rank

Emphasize the Context for every number, nothing without Context.

Rank: Interpret in context of state or other ranking.

FosteringCourtImprovement.org31

The Systemic, Process Context

Flow of cases through the system

FosteringCourtImprovement.org32

Outcome Framework Context

Safety• Maltreatment

does not occur.Measures: Proportion of children reported maltreated.

• Maltreatment does not recur.Measures: Proportion of children re-reported within 3, 6, and 12 months; Proportion re-entering foster care.

Permanency• Children remain in

their homes.Measures: Proportion removed from their home.

• Bonded attachments are maintained.Measures: Proportion of siblings placed together; Frequency of visitation with kin; Proportion in a stable placement.

• Permanency is timely.Measures: Proportion reunified with family within 12 months; Proportion adopted within 24 months.

Well Being• Children are

healthy.Measures: Proportion with physical health needs met; Proportion with mental health needs met.

• Children are educated.Measures: Proportion performing at age-appropriate grade level.

Note that these outcomes are from the child’s perspective, not the system’s.

FosteringCourtImprovement.org33

Track Record: Real Work Done in a Shared Context

Douglas County, GA:• Removals cut 50%• Relative placements

increased 10-fold• Adoption rate

doubled Floyd County, GA:

• Removals cut 30%• Pre-teens in group

care down 80%

Hall County, GA:• Adoptions under 24

months tripled Bartow County, GA:

• Removals cut 50% Most counties

unchanged, but actively dissecting their processes

FosteringCourtImprovement.org34

Improving the Product

Additional Reports:• Safety Report from linked NCANDS-AFCARS data• Children Remaining Safe with Family• CFSR 2001, CFSR 2006• Illinois Signs of Progress Reports• Toolkit (9 Measures)

Discovering Best Practices:• Composite Measures• Causal Linkages from Process Measures to

Outcome Measures

FosteringCourtImprovement.org35

Loose Ends Process Measures (caseloads, due process, …) are crucial,

but only when put in the context of Outcome Measures.

Judges care deeply about outcomes. Need bottom-lines to keep us aligned and on task, I

propose:• Children safe with family• Failing that, timely permanency

Shift spending from data acquisition to data analysis. Don’t call GeekSquad™ for data analysis.

(Don’t equate technology and data analysis)

FosteringCourtImprovement.org

Questions, Comments ...

Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results

Andy Barclay, Andy@FosteringCourtImprovement.orgStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law

Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006