Data: Telling a Story … A Walk Through Meaningful Data Analysis

FosteringCourtImprovement.org

Data: Telling a Story …A Walk Through Meaningful Data

Analysis

Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results

Andy BarclayStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law

Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006

FosteringCourtImprovement.org2

Overview How Knowledge of the Past and Present

can Help you Shape the Future• Data, Data, Data Isn’t Yada, Yada, Yada• Court Performance Measures• AFCARS

The Easy Win vs. Sea Change Getting Real, Getting Local, Getting

Context Results that Matter to Kids & Families


Shaping the Future

Why Data, Data, DataIsn’t

Yada, Yada, Yada


Lessons learned or missteps repeated?

Lunchroom lore or documented results?

Shaping the Future


Shaping the Future

“They always say that time changes things, but you actually have to change them yourself.”

Andy Warhol


Do you know where your caseload has been?

Do you know where it is?

Do you know where it is going?

Shaping the Future


Shaping the Future

Am I on the road to success or on a “frolic and detour”?

(Only the data know.)


Shaping the Future

“Georgia Judge leads stampede out of homes.”


Shaping the Future

“Illinois Judge ‘doesn’t have time’ to return children to parents.”


Shaping the Future

Take off the blinders.Or

Load and Learn.


Shaping the Future

Where to Start?


Court Performance Measures

The three leading legal/ judicial organizations, with a grant from the Packard Foundation, have created and pilot-tested court performance measures that complement those used in the CFSR process so that agencies and courts can collaborate in the measurement of their performance and tracking of their own progress in improving safety, permanence, and timeliness for children coming before them.



(Building a Better Court) Reinforce current federal reform

initiatives, including CFSR and PIP’s Helps to identify areas of court reform

essential to better outcomes Assist the CIP by providing courts with

tools to engage in systematic process of court performance measurement



“Performance measures are the yardstick by which courts and child welfare agencies can measure the success they are achieving and the progress they are making.”

Building a Better Court



Safety: Are there new allegations while in foster care?

Permanency: What was the exit from foster care and how many came back?

Due Process: Is there notice and representation?

Timeliness: How long between hearings?


Already Measurable and Measured (AFCARS)

Safety Permanency Timeliness


AFCARS

Adoption and Foster Care Analysis and Reporting System


Uses of AFCARS

Award Adoption Incentive bonuses Allot Chafee Independence $$$ Produce annual reports to Congress

on child welfare outcomes Conduct IV-E eligibility reviews Assess state’s compliance with federal

CFSR performance standards


Platform for Collaboration

Longitudinal AFCARS provides a platform for rolling out the due process & timeliness measures recommended by Building a Better Court

Promotes working partnership between the courts, child welfare agencies and universities,

Provides 9 of the 24 measures defined in Building a Better Court, and 9 of the 30 in ToolKit.


Fostering Court Improvement

What gets measured gets done,

IfYou know it’s being measured.


Stories from Medicine : An Easy Win

Diagnostic Imaging (Cardiac PET/SPECT) Pre-Screen

Very simple questionnaire Predictive Value of the Test

• Surgery: y/n• Angioplasty: y/n

Easy sell to cardiologists Very easy sell to insurance

companies

Bayesian Pre-Screening Tool


Stories from Medicine : A Sea Change

Major resistance, even when shown computer was right more often, lower mortality

Over time (3 years) older docs modified their risk tolerance, less invasive, better outcomes, learned from computer

Tripled business, more patients living longer, much lower costs ...

Automated Reading of Scans


Court Stakeholders Have a Right to be Skeptical

In medicine, I see examples of better outcomes at lower cost quite often.

Have yet to see one in child welfare. Many claims, little evidence Judges, attorneys, CASAs, and caseworkers

have good reason to be skeptical:• Complex Business• Spectacular IT Failures• Terabytes of Unused Data


Get Real Data reports are only meaningful to you in the

context of your job, your reality. We (FCI) have been successful* using data to

develop a shared context among local court stakeholder groups.

Very different jobs, very different realities, but we need to have some common understandings to do successful* work together (and occasionally alter some realities).

*successfully improve an outcome


Diverse Realities: Dependency Court

Caseloads

3,140 US counties + DC

70% of counties remove less than 1 child per week

• 50% less than 1 per 2 weeks

7% of counties (234) average more than 1 child removal per business day


Get Local Much as I’d like to, we

probably can’t do this with just a website.

• our web reports are not for a general audience.

County by county …• Judges convene• Georgia’s CIP buys lunch• I walk through a

Powerpoint of charts & numbers from the website.


Broad Topic Areas of Court Stakeholder Meetings

Removal Rates Reasons for Removal Kids in Care Group Settings IV-E Reimbursement Kids Headed for

Adoption Reunification Rate Time to Adoption


A Simple Temporal Context

Emphasize the Context for every number, nothing without Context.

Temporal: Interpret current measures in the context of the past.


The Context of Comparable Jurisdictions


Comparables: Interpret in context of comparable counties.


The Context of Statewide Rank


Rank: Interpret in context of state or other ranking.


The Systemic, Process Context

Flow of cases through the system


Outcome Framework Context

Safety• Maltreatment

does not occur.Measures: Proportion of children reported maltreated.

• Maltreatment does not recur.Measures: Proportion of children re-reported within 3, 6, and 12 months; Proportion re-entering foster care.

Permanency• Children remain in

their homes.Measures: Proportion removed from their home.

• Bonded attachments are maintained.Measures: Proportion of siblings placed together; Frequency of visitation with kin; Proportion in a stable placement.

• Permanency is timely.Measures: Proportion reunified with family within 12 months; Proportion adopted within 24 months.

Well Being• Children are

healthy.Measures: Proportion with physical health needs met; Proportion with mental health needs met.

• Children are educated.Measures: Proportion performing at age-appropriate grade level.

Note that these outcomes are from the child’s perspective, not the system’s.


Track Record: Real Work Done in a Shared Context

Douglas County, GA:• Removals cut 50%• Relative placements

increased 10-fold• Adoption rate

doubled Floyd County, GA:

• Removals cut 30%• Pre-teens in group

care down 80%

Hall County, GA:• Adoptions under 24

months tripled Bartow County, GA:

• Removals cut 50% Most counties

unchanged, but actively dissecting their processes


Improving the Product

Additional Reports:• Safety Report from linked NCANDS-AFCARS data• Children Remaining Safe with Family• CFSR 2001, CFSR 2006• Illinois Signs of Progress Reports• Toolkit (9 Measures)

Discovering Best Practices:• Composite Measures• Causal Linkages from Process Measures to

Outcome Measures


Loose Ends Process Measures (caseloads, due process, …) are crucial,

but only when put in the context of Outcome Measures.

Judges care deeply about outcomes. Need bottom-lines to keep us aligned and on task, I

propose:• Children safe with family• Failing that, timely permanency

Shift spending from data acquisition to data analysis. Don’t call GeekSquad™ for data analysis.

(Don’t equate technology and data analysis)

FosteringCourtImprovement.org

Questions, Comments ...

Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results

Andy Barclay, [email protected] and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law

Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006

Data: Telling a Story … A Walk Through Meaningful Data Analysis

Documents

Transcript of Data: Telling a Story … A Walk Through Meaningful Data Analysis