1

Evidence Based Practice:One Agency’s Journey

Dr. Shirley Hunt

Director, Research Division

Orange County Probation Department

Presented to

American Probation and Parole Association

February 2, 2010

Orange

County

CA

2

OC FAST FACTS

• 3rd largest county in CA (~3.1 million)

• Exceptional wealth and extreme poverty

within 798 square miles

• County oversight of probation (no direct

ties to state)

• Supervision of adults (~ 16,000) and

juveniles (~6,000)

3

OC Probation’s EBP Journey

• 1980s Decade – Implemented NIC Probation Model

• Validated Risk-Need Assessment Instruments • Initiated Supervision by risk & Case Plan Objectives by needs• Began tracking workload & staffing (paper and pencil)• Set up a PC-database for assessments

• 1990s Decade – Building & Expanding Capacity• Growth in use of OC data to inform strategic planning and fiscal decisions• Began formal program evaluations (“8%” repeat offenders)• Implemented automated “Case Management System” (Individual-level +

Unit workload)

• 2000s Decade – Revisiting Goals & EBP • Expanded internal research capacity • Continued use of OC data + “what works” to inform OCPD Operations• 2008 – selected as 4th NIC EBP site• Expansion of automated case management system (“windows-based”)

4

Measuring EBP ……

from the Inside/Out

• What data do you have now? What do

you need?

• Where is it? How will you get it?

• Once you have it, what do you do with it?

5

What Data?

“Assess agency data using EBP template”

• Demographic • Age (DOB), gender, race/ethnicity, residence, citizenship

• Supervision • Presenting offense, conviction date, supervision start date • Risk/Need assessment scores (changes over time – overall/subgroups)• Changes in supervision type and/or level (dates occurred)• Officer-Offender Interaction (contacts - #, type, quality of..)

• Treatment/Sanctions• Treatment Type/Dosage (agency, referral date, completion status, exit date)• Sanctions placed (causal event, date, custody days imposed, other,

completion status)

• Outcomes • Arrests, violations, incarceration (type of offense, dates)• Length of supervision (probation termination date, type)

6

Where are the Data?

“Map your system to identify key points of service”

• What information is available at each point?• Who is the “holder” of the data? One agency/multiple agencies.

• Probation/Court/DA/Social Services/Health Care/Treatment Provider/• What additional data can/should be captured at each point?

• How will you access the data? • How is the existing data currently maintained (paper; automated, etc.)• What barriers exist to exchanging or sharing of data? (confidentiality,

technology, etc.)

• How will you track information over time?• What’s needed for a comprehensive “data record” from entrance to exit• What data elements must be permanent (never overridden)

• What is the plan to ensure reliability and validity of information over time? (very important question!)• Who will be the “responsible” owner?

7

Challenges: Creating an EBP data record

Low-Risk High-Risk� 49 yr. old male convicted driving

under the influence (heroin)

� 2 prior convictions (DUI-alcohol)

� Assessed “High risk/Low Needs”

� Risk: Freq. moves; Unstable

emplymt; Serious drug/alcohol

us; Priors; < 19 at lst offense;

� Needs: Lower skill level; unsatisf.

Emplmt; Situational finan. issues

� Supervision: Prop 36 -> Reg Sup

-> Drug court

� Treatment: 4 alco/drug

treatment programs (failed)

� TERMINATED PRISON:

THREE TECH. VIOLATIONS

� 43 yr. old male convicted driving

under the influence of meth.

� No prior convictions

� Initially placed on Informal/cont’d

meth use -> Formal/Prop 36

� Assessed “Low risk/Low needs”

� Risk: (1) recent unstable

employment - emplyd 11 years until

incarcerated; (2) serious drug abuse

� Needs: (1) Drug problem; (2)

Emot. issues (abused by alcoholic

father; ADHD); (3) Finan. issues

� Min. Sup./ Enrolled in drug trtmt

� TERMINATED NO NLV;

COMPLETED TREATMENT

8

How do you use data to measure

progress and Improve Outcomes?

(or)Now you have the data, what do you

do with it?

“ Begin asking/answering this

question from day one and

continuously revisit”

9

EBP MEASURES:

INFORMATION/ANALYSIS MATRIX

COMMUNITY OFFENDER OFFENDER

PROTECTION ACCOUNT. COMPETENCY

PROCESS

INTERMEDIATE

LONG-TERM

Short-term measures of progress

Long-term measures of success in

meeting goals

Counting of numbers

10

……..PROCESS COUNTING…….

COMMUNITY OFFENDER OFFENDER

PROTECTION ACCOUNT. COMPETENCY

• # on probation

by risk level

• # of contacts by

risk level

• # of prob. Out

to warrants

• # of prob. drug

tested

administered

• # of prob. With

positive drug

results

• # referred to

drug counseling

by risk level

• # of juv prob

attending school

regularly

What are we doing? OCPD

____________

11

Unit Summary Report (Example)

GSUSUM1UNIT STATISTICAL

SUMMARYPage: 1

5/11/04 9:14:27

Unit: 059 JUVENILE DECENTRALIZED IN

Month: April 2004

Supervisor:

(Cases) Total

Total Reso Prog W/L

Off # Officer Name Cases New High Med Low Unc Cus FM Ref PT Dispo Rpt Points

0591 58 30 28 112.48

0592 57 35 22 116.72

**Report Totals** 115 65 50 229.2

CASE MOVEMENTSTATISTICS

1. In Cases: New- 12 Transfer- 0

2. Out Cases: Trm- 11 Transfer- 3

Warrants- 0

3. % Cases Turn Over- 22.61

12

Intermediate Measures

• % of population with completed r/n assessment (set goal)

• Gain scores: Changes in protective measures

• # & type of technical violations & dispositions

• % of medium/high risk offenders with case plans developed within specified timeframe

• % of case plans that address the top 3 identified criminogenicneeds based on assessment

• Average length of supervision sorted by risk level.

13

Employment Status:

Percent with satisfactory employment

Intermediate measure example

28%

40%

0%

10%

20%

30%

40%

50%

1% at Entry % at One Year

14

Technical Violations at First Reassessment:

% With No Tech. Violations by Initial Risk Level

Intermediate measure example – Opportunity to do QAS

70%

83%92%

0%

20%

40%

60%

80%

100%

High Medium Low

Initial Risk Classification

% W

ith

No

Tech

Vio

lati

on

s

15

Long-Term Outcomes

How well are we doing?

1. % Terminating Satisfactorily

2. # of probationers terminating with no new law

violations (while under probation supervision?

3. Violation outcomes by initial risk level/scores

16

Adult Termination Outcomes FY 08/09

34%30%

12%17%

7%

0%

10%

20%

30%

40%

50%

Normal Prison/Jail Cond/ROS Early Term Other Term

FY 08/09 Adult Termination Status

17

Terminations with Zero New Law:

Ten Year Outcomes

Adult Probationers Terminating

w ith No New Law Violations

69%71%

69%71%

66%

71%69%

67%66%

74%

56%

60%

64%

68%

72%

76%

80%

FY 99-

00FY 0

0-01

FY 01-

02FY 0

2-03

FY 03-

04FY 0

4-05

FY 05-

06FY 0

6-07

FY 07-

08FY 0

8-09

% o

f T

erm

inate

d P

rob

ati

on

ers

18

Violation Outcomes by Initial Risk

38%

23%

39%

67%

12%

20%

87%

5%8%

0%

20%

40%

60%

80%

100%

High Medium Low

Initial Risk Classification

Adult FY 08/09 Terminations:

Violation Outcomes by Initial Risk level

New Law

Tech Only

None

19

Adult Terminations with New Law Violation by Initial Risk Score

(2008-09 Terminations, n=4292)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Total Initial Risk Score

% 1

+ N

LV

Medium HighLow

20

OC Probation’s EBP Journey:

What have we learned?

• What have we done well?• Maintaining formal risk/need assessment as cornerstone of operations

for 2+ decades (supervision & case plan)

• Integrating research/evidence (data) into organizational fabric

• Automating functions and production of process reports

• Focusing on agency-level performance measures

• What are we doing….medium?• Case plans are “barebones”

• Minimal attention to quality control (e.g. interrater reliability)

• Measures of officer performance still traditional (i.e. contacts, testing)

• Limited use of data to show outcomes at the officer/unit level

• What still remains to be done……?

21

2010’s Decade:

Coming EBP attractions in OCPD

• Building next generation of continuous quality improvement

• Developing automated outcome reports

• Expanding performance measures (from agency-wide out to staff level)

• Strengthening Measurement of “officer-offender”relationship quality

22

For more information

• Dr. Shirley Hunt

Orange County Probation DepartmentPO Box 10260Santa Ana, CA 92701714-569-2174shirley.hunt@prob.ocgov.com

OrMicheal CollinsOrange County Probation Department714-667-7723micheal.collins@prob.ocgov.com

Orange

County

CA