Results-Based Accountability

I N

N O

V A

T I

O N

Results-Based AccountabilityResults-Based Accountability

Discovery Communities TA InstituteDiscovery Communities TA Institute

April 1, 2008April 1, 2008

2

2

Results Accountability Decision-making

and Strategic Planning

Fiscal Policy Studies InstituteFiscal Policy Studies Institute

Santa Fe, New MexicoSanta Fe, New Mexico

WEBSITESWEBSITES

www.resultsaccountability.comwww.resultsaccountability.com

www.raguide.orgwww.raguide.org

www.charteroakgroup.comwww.charteroakgroup.com

BOOK ORDERS www.trafford.com

www.amazon.com

3

3

3

Why Are We Here?Why Are We Here?

4

4

SIMPLE

COMMON SENSE

PLAIN LANGUAGE

MINIMUM PAPER

USEFUL

5

5

Concepts to Take Away TodayConcepts to Take Away Today

Population accountability v. performancePopulation accountability v. performance accountability accountability

Ends v. meansEnds v. means

How to choose indicators and measuresHow to choose indicators and measures

The importance of a data development agendaThe importance of a data development agenda

How to move from talk to actionHow to move from talk to action

6

6

Results Accountabilityis made up of two parts:


Performance Accountabilityabout the well-being of

CLIENT POPULATIONSFor Programs – Agencies – and Service Systems

Population Accountabilityabout the well-being of

WHOLE POPULATIONSFor Communities – Cities – Counties – States - Nations

7

7

7

Results and Performance Accountability

Results and Performance Accountability

COMMON LANGUAGE

COMMON SENSE

COMMON GROUND

8

8

THE LANGUAGE TRAPToo many terms. Too few definitions. Too little discipline

Benchmark

Target

Indicator Goal

Result

Objective

Outcome

Measure

Modifiers Measurable Core Urgent Qualitative Priority Programmatic Targeted Performance Incremental Strategic Systemic

Lewis Carroll Center for Language Disorders

9

9

The Humpty Dumpty Approach to LanguageThe Humpty Dumpty Approach to Language

““When When II use a word,” Humpty Dumpty said, use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just in rather a scornful tone, “it means just what I choose it to mean—neither more nor what I choose it to mean—neither more nor less.” “The question is,” said Alice, less.” “The question is,” said Alice, “whether you “whether you cancan make words mean so make words mean so many different things.” “The question is,” many different things.” “The question is,” said Humpty Dumpty, “which is to be said Humpty Dumpty, “which is to be master—that’s all.”master—that’s all.”

LEWIS CARROLL, LEWIS CARROLL, Through the Looking-Through the Looking-Glass,Glass, chapter 6, p. 205 (1934). First chapter 6, p. 205 (1934). First published in 1872.published in 1872.

10

10

DEFINITIONS

Children born healthy, Children succeeding in school, Safe communities, Clean Environment, Prosperous Economy

Rate of low-birthweight babies, Rate of high school graduation, crime rate, air quality index, unemployment rate

1. How much did we do? 2. How well did we do it? 3. Is anyone better off?

RESULT

INDICATOR

PERFORMANCE MEASURE

A condition of well-being for children, adults, families or communities.

A measure which helps quantify the achievement of a result.

A measure of how well a program, agency or service system is working. Three types:

= Customer Outcomes

Popu

latio

nPe

rfor

man

ce

11

11

From Ends to Means

ENDS

MEANS

From Talk to ActionPo

pula

tion

Perf

orm

ance

RESULT

INDICATOR

PERFORMANCEMEASURE

Customer outcome = EndsService delivery = Means

12

12

Connecticut Glossary of RBA TermsConnecticut Glossary of RBA Terms

The Appropriations Committee standardized the The Appropriations Committee standardized the terms we use in Connecticut. You have a copy of the terms we use in Connecticut. You have a copy of the glossary.glossary.

We could have chosen other terms consistent with We could have chosen other terms consistent with Friedman’s RBA approach so this list is somewhat Friedman’s RBA approach so this list is somewhat arbitrary.arbitrary.

Its virtue is that everyone in Connecticut doing this Its virtue is that everyone in Connecticut doing this work – both executive branch and legislative branch – is work – both executive branch and legislative branch – is using the same vocabulary and meaning the same thing using the same vocabulary and meaning the same thing by it. At least we should be able to understand each by it. At least we should be able to understand each other.other.

13

13

Some Connecticut Early Childhood IndicatorsSome Connecticut Early Childhood Indicators

% Infants born at low birth weight% Infants born at low birth weight

% Births to mothers without a high school degree% Births to mothers without a high school degree

% of kindergartners with all or most pre-literacy and personal % of kindergartners with all or most pre-literacy and personal skillsskills

% 4% 4thth grade reading scores at mastery or above grade reading scores at mastery or above

14

14

IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE?IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE?

1. Safe Community

2. Crime Rate

3. Average Police Dept response time

4. A community without graffiti

5. % of surveyed buildings without graffiti

6. People have living wage jobs and income

7. % of people with living wage jobs and income

8. % of participants in job training who get living wage jobs

15

15

IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE?IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE?

9. % HS graduates enrolling in college

10. Traffic-related death rate

11. Clean environment

12. Air pollutants in parts per million

13. % participating cities fixing treatment plants

14. All children eat healthy

15. % students eligible for free lunch participating in free lunch

16

16

POPULATIONACCOUNTABILITY

POPULATIONACCOUNTABILITY

For Whole PopulationsFor Whole Populationsin a Geographic Areain a Geographic Area

17

17

17

MarylandResults for Child Well-Being

● Babies born healthy

● Healthy children

● Children enter school ready to learn

● Children successful in school

● Children safe in their families and communities

● Stable and economically independent families

● Communities that support family life

18

18

18

Connecticut Results StatementsConnecticut Results Statements

• A clean and healthy Long IslandA clean and healthy Long Island Sound Sound

• All children healthy and ready toAll children healthy and ready to learn by age 5 learn by age 5

• All children ready by five andAll children ready by five and fine by nine fine by nine

19

19

Leaking Roof(Results thinking in everyday life)

Experience:

Measure:

Story behind the baseline (causes):

Partners:

What Works:

Action Plan:

Inches of Water

? Fixed

Not OK

Turning the Curve

20

20

20

Seven Population Accountability QuestionsSeven Population Accountability Questions

What are the quality of life conditions we want for the What are the quality of life conditions we want for the children, adults and families who live in our community?children, adults and families who live in our community?

What would these conditions look like if we could see them?What would these conditions look like if we could see them?

How can we measure these conditions?How can we measure these conditions?

How are we doing on the most important of these How are we doing on the most important of these measures?measures?

Who are the partners that have a role to play in doing Who are the partners that have a role to play in doing better?better?

What works to do better, including no-cost and low-cost What works to do better, including no-cost and low-cost ideas?ideas?

What do we propose to do?What do we propose to do?

21

21

“We haven’tgot the money, so we’ve gotto think.”

Lord Rutherford1871 - 1937

22

22

22

Connecticut Population TemplateConnecticut Population Template Quality of Life Result All Connecticut children are healthy and ready for school success at age 5, contributing to a reduction over time in Connecticut’s “achievement gap” at Grade 4. Indicator 1: Infants born at Low Birth Weight (LBW) Indicator 2: Births to Mothers Without a High School Degree

% Births to Mothers without a High School Degree

0

5

10

15

20

25

2000 2001 2002 2003 2004 2005

Priority Distircts Other SRP Balance of Towns

Indicator 3: HUSKY A Enrollment and Participation Rates

Story Behind the Baselines 41,719 babies were born in CT in 2005. About 6,000 (14%) are at risk because their family income is at or below the Federal Poverty Level. About 28% of young children at each age (~12,000 children) are at risk of school un-readiness because their family income is at or below 185% of the Federal Poverty Level. Nearly eight in ten of these “at risk” children (78%) live in just 19 towns, CT’s Priority School Districts. Another 15% live in the balance of School Readiness Towns (39 towns). These 58 towns are listed at the back of this template. The rest of CT’s at risk children, defined (for now) as living in poverty, (7%) live in the remaining 111 communities. Other indicators point to developmental challenges for many of these same children: Seven in ten of all low birth weight babies

(71%) live in the 58 School Readiness towns, and half of them (52%) live in the 19 Priority School Districts. Of note, while the average rate of low birth weight babies is stable (but too high), it is rising among African American families.

About a quarter (23%) of mothers with young

children who live in Priority School Districts have not attained a high school degree, a rate much higher than for other communities in CT. The proportion of mothers without a high school degree is increasing in these 19 towns as well.

Nearly two-thirds (64%) of children from

Priority School District towns enter kindergarten without the pre-literacy skills needed for early school success.

Key Funding Information Total Current Funding 533.3 million Funding Distribution Total Federal Funds 262.9 million Total State Funds 266.3 million Capital Projects Subtotal Other Funding 4.1 million Percent of Total Funding Contracted to Third Parties

Dis tr ibution o f Bir ths and LBW In fan ts , 2003

40%

52%

22%

19%

35%

29%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Tot al B irt hs LB W

P rior it y Distric ts Ot her S R Distr ic ts A ll other Towns

HUSKY A Enrollment and Well-Care Participation, Children Under Six Years

65%

70%

75%

80%

85%

Per

cen

t

Enrollment of Eligible Children At Least 1 Well-care visit

Enrollment ofEligible Children

78% 80% 82% 83%

At Least 1 Well-

care visit

72% 74% 76% 80%

2002 2003 2004 2005

23

23

Criteria for

Choosing Indicators

as Primary vs. Secondary Measures

Communication Power

Proxy Power

Data Power

Does the indicator communicate to a broad range of audiences?

Does the indicator say something of central importance about the result?

Does the indicator bring along the data HERD?

Quality data available on a timely basis.

24

24

Choosing IndicatorsWorksheet

Outcome or Result_______________________

Candidate IndicatorsCommunication

PowerProxyPower

DataPower

H M L

H

Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 6

Measure 7

Measure 8

HData

Development

Agenda

Safe Community

H M L H M L

H H

H L

25

25

Three Part Indicator List for each Result

Part 1: Primary Indicators

Part 2: Secondary Indicators

Part 3: Data Development Agenda

● 2 or 3 or 4 “Headline” Indicators● What this result “means” to the community● Meets the Public Square Test

● Everything else that’s any good (Nothing is wasted.)● Used later in the story behind the baseline

● New data● Data in need of repair (quality,timeliness etc.)

26

26

The Matter of Baselines

Baselines have two parts: history and forecast

H

M

L

History Forecast

Turning the CurvePoint to Point

OK?

27

27

Performance AccountabilityPerformance Accountability

For Programs, Agencies and Service For Programs, Agencies and Service SystemsSystems

28

28



Performance Accountabilityabout the well-being of

CLIENT POPULATIONSFor Programs – Agencies – and Service Systems

Population Accountabilityabout the well-being of

WHOLE POPULATIONSFor Communities – Cities – Counties – States - Nations

29

29

“All Performance Measures

that have ever existed

for any program

in the history of the universe

involve answering two sets of

interlocking questions.”

30

30

HowMuchdid we do?

( # )

HowWell

did we do it?

( % )

Quantity Quality

Program Performance Measures

31

31

EffortHow hard did we try?

EffectIs anyone better off?


32

32

Effort

Effect

HowMuch

HowWell


33

33

How much service did we deliver?


How welldid we

deliver it?

How much change / effect

did we produce?

What quality of change / effect

did we produce?

Quantity Quality

E

ffect

Ef

fort

Out

put

Inpu

t

34

34

How much did we do?


How welldid we do it?

Is anyonebetter off?

Quantity Quality

E

ffect

Ef

fort

# %

35

35

How much did we do?

Education

How well did we do it?

Is anyone better off?

Quantity Quality

Effe

ct

E

ffort Number of

studentsStudent-teacher

ratio

Number ofhigh schoolgraduates

Percent ofhigh schoolgraduates

36

36

How much did we do?

Education



Quantity Quality

Effe

ct

E

ffort Number of

studentsStudent-teacher

ratio

Percent of 9th graders who

graduate on timeand enter college or

employment after graduation

Number of 9th graders who

graduate on timeand enter college or

employment after graduation

37

37

How much did we do?

Health Practice



Number ofpatientstreated

Percent ofpatients treated

in less than1 hour

Incidence ofpreventable

disease(in the practice)

Rate ofpreventable

disease(in the practice)

Quantity Quality

E

ffect

Effo

rt

38

38

How much did we do?

General Motors



# of production hrs

# tons of steel

# of cars produced

Employees pervehicle

produced

# of cars sold

$ amount of Profit

$ car value after 2 years

Quantity Quality

Effe

ct

E

ffort

Source: USA Today 9/28/98

% market share

Profit per share

% car value after 2 years

39

39

39

● # of people served

● % participants who got jobs

● staff turnover rate

● # participants who got jobs

● % of children reading at grade level

● cost per unit of service

● # applications processed

● % patients who fully recover

What Kind of PERFORMANCE MEASURE?

Upper Left

Lower Right

Upper Right

Lower Left

Lower Right

Upper Right

Upper Left

Lower Right

40

40

RBA Categories Account for All Performance Measures(in the history of the universe)

Quantity Quality

Efficiency, Admin overhead, Unit costStaffing ratios, Staff turnoverStaff morale, Access, Waiting time, Waiting lists, Worker safety

Customer Satisfaction(quality service delivery& customer benefit)

Cost / Benefit ratioReturn on investment

Client results or client outcomes

EffectivenessValue-addedProductivity

Benefit value

Product Output Impact

Process Input

Effe

ctEf

fort

Cost

TQM

Effectiveness

Efficiency

41

41

42

42

How much did we do?

The Matter of Control



Quantity Quality

Effe

ct

E

ffort

LeastControl

PARTNERSHIPS

MostControl

43

43

The Matter of Use

1. Fundamental Purpose is to Improve Performance as a contribution to improving results

2. Avoid the Performance Measurement Equals Punishment Trap

● Acknowledge the experience as real.

● Work to create a healthy organizational environment

● Start small.

● Build bottom-up and top-down simultaneously.

44

44

How much did we do?


How welldid we do it?

Is anyonebetter off?

Quantity Quality

E

ffect

Ef

fort

# %

45

45

45

Connecticut Program TemplateConnecticut Program TemplateConnecticut Appropriations Committee RBA Template

Part II, Program/Agency/System Accountability Summary Special Supplemental Nutrition Program for Women, Infants, and Children (WIC)

Program/Agency/System Purpose The purpose of the WIC Program is to provide supplemental foods, nutrition education and referrals for health and support services to eligible, low income pregnant, postpartum and breastfeeding women, and children 0-5.

Program/Agency/System The Special Supplemental Nutrition Program for Women, Infants and Children (WIC Program) / Department of Public Health, Public Health Initiatives Branch

Performance Measure 1 The incidence of low birth weight (LBW) among infants whose mothers were on the WIC Program for at least 6 months during pregnancy does not exceed 6%.

Performance Measure 2 At least 50% of infants whose mothers were enrolled in the WIC Program during pregnancy breastfeed.

Performance Measure 3 The prevalence of anemia among children enrolled in the WIC Program for at least one year does not exceed 10%.

Key Budget Information Total Current Program Year Funding Admin $9.38 M

Food $27.68 M

Funding as Percent of All Funding for Population Result

Program Funding As Percent of Total Agency Budget

Admin 4% Food 11%

Funding Distribution Total Federal Funds $37.06 M

Total State Funds $0

Capital Projects Subtotal

Other Funding $0

Percent of Total Current Funding Contracted to Third Parties

Admin 77%

Story Behind the Baselines Participation in the WIC Program is associated with Performance Measure 1: a reduction in rates of low birth weight (LBW) and Performance Measure 3: childhood anemia. It also is associated with better cognitive performance. Early and continuous enrollment in the program may result in further improvement in LBW and anemia rates and, hence, school readiness. Performance Measure 2: Children who were breastfed score higher on cognitive and IQ tests at school age. Breastfed infants experience fewer and less severe cases of infectious and non-infectious illnesses. Therefore, children who were breastfed are more likely to be healthy and ready to learn. Turning The Curves: What do you propose to do over the next two years and why? 1. Prepare and implement the USDA Value Enhanced

Nutrition Assessment (VENA) state plan to improve client services; enhance local agency program monitoring to improve program operations, nutrition counseling and breastfeeding promotion.*

2. Contingent upon the receipt of additional funding, conduct formative research to identify why

eligible women do not routinely enroll in WIC during their first trimester of pregnancy and

eligible children leave the program before age 5, and develop a targeted outreach initiative to address. 3. Contingent upon the receipt of additional funding,

increase WIC local agency staffing levels to the state-recommended staffing pattern to enhance outreach activities and nutrition and breastfeeding education.

4. Contingent upon the receipt of additional funding, replicate the existing model breastfeeding peer counseling program in a selected WIC local agency.

*Indicates, low-cost, no-cost action steps, including reallocation of existing resources.

46

46

How Population

&Performance Accountability

FIT TOGETHER

How Population

&Performance Accountability

FIT TOGETHER

47

47

POPULATION ACCOUNTABILITY

Healthy Births Rate of low birth-weight babiesStable Families Rate of child abuse and neglectChildren Succeeding in School Percent graduating from high school on time

Contributionrelationship

Alignmentof measures

Appropriateresponsibility

Child Welfare Program

THE LINKAGE Between POPULATION and PERFORMANCE

CUSTOMEROutcomes

POPULATIONRESULTS

# Foster ChildrenServed

% withMultiple

Placements

# RepeatAbuse/Neglect

% RepeatAbuse/Neglect

PERFORMANCE ACCOUNTABILITY

48

48

Contributionrelationship

Alignmentof measures

Appropriateresponsibility

THE LINKAGE Between POPULATION and PERFORMANCE

POPULATION ACCOUNTABILITY

Healthy Births Rate of low birth-weight babiesChildren Ready for School Percent fully ready per K-entry assessmentSelf-sufficient Families Percent of parents earning a living wage

CUSTOMEROutcomes

# personsreceivingtraining

Unit costper person

trained

# who getliving wage jobs

% who getliving wage jobs

PERFORMANCE ACCOUNTABILITY

POPULATIONRESULTS

Job Training Program

49

49

50

50

50

What is Happening in the Legislature?What is Happening in the Legislature?

Institutionalizing RBA in the legislature:Institutionalizing RBA in the legislature:

• The Appropriations Committee has created a new RBA sub-The Appropriations Committee has created a new RBA sub-committeecommittee

Early Childhood Cabinet must develop an accountability plan Early Childhood Cabinet must develop an accountability plan and make other recommendations for changes necessary to and make other recommendations for changes necessary to ensure coordination, service integration, and accountabilityensure coordination, service integration, and accountability

The 2007 budget requires new and expanded programs to The 2007 budget requires new and expanded programs to report to the General Assembly and OPM using an approved report to the General Assembly and OPM using an approved RBA framework, an effort lead by the Office of Fiscal RBA framework, an effort lead by the Office of Fiscal Analysis (OFA) and OPMAnalysis (OFA) and OPM

The co-chairs and rankings members of the other sub-The co-chairs and rankings members of the other sub-committees of Appropriations are identifying particular committees of Appropriations are identifying particular public policy issues (result statements) for RBA public policy issues (result statements) for RBA development development

All this month, the early childhood agencies that were part All this month, the early childhood agencies that were part of the RBA Phase II initiative last session reported back to of the RBA Phase II initiative last session reported back to the legislature on the progress they have madethe legislature on the progress they have made

51

51

What is Happening at State Agencies and in Communities?What is Happening at State Agencies and in Communities?

The Child Poverty and Prevention Council and The Child Poverty and Prevention Council and the Juvenile Justice Policy and Operating the Juvenile Justice Policy and Operating Coordinating Committee are both using RBA to Coordinating Committee are both using RBA to guide their implementation effortsguide their implementation efforts

Municipalities and non-profit groups, e.g., Municipalities and non-profit groups, e.g., workforce boards, are embracing RBA in areas workforce boards, are embracing RBA in areas beyond early childhoodbeyond early childhood

State agencies will be developing program State agencies will be developing program quality and outcome measures in response to the quality and outcome measures in response to the new Appropriations Committee initiativesnew Appropriations Committee initiatives

They will be passing the requirements along to They will be passing the requirements along to their vendorstheir vendors

52

52

52

What is Happening at State Agencies and in Communities?What is Happening at State Agencies and in Communities?

• Organizations that contract with the State will Organizations that contract with the State will have new data collection and reporting have new data collection and reporting requirements, including outcome measuresrequirements, including outcome measures

• Managing performance will be more Managing performance will be more complicatedcomplicated

• Likely to be greater requirements for Likely to be greater requirements for integration of programs and collaboration with integration of programs and collaboration with other agenciesother agencies

• Opportunity for contractors to have input on Opportunity for contractors to have input on measurementmeasurement

• Opportunity for program improvementOpportunity for program improvement

53

53

53

“If you do what you always did,

you will get what you always got.”

“If you do what you always did,

you will get what you always got.”

Kenneth W. JenkinsPresident, Yonkers NY NAACP

54

54

Thank YouThank You

The Charter Oak Group, LLCThe Charter Oak Group, LLC

www.charteroakgroup.comwww.charteroakgroup.com

Ron SchackRon Schack

[email protected][email protected]

(860) 478-7847(860) 478-7847

Bennett PudlinBennett Pudlin

[email protected]@charteroakgroup.com

(860) 324-3555(860) 324-3555

55

55

Turn the Curve Exercise: Population Well-being

5 min: Starting Points - timekeeper and reporter - two hats (yours plus partner’s)

5 min: Baseline

- forecast: Where is the trend line going? - turn the curve: Is forecast OK or not OK?

15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes

15 min: What works? (What would it take?) - what could work to do better

- each partner’s contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works

10 min: Report: Convert notes to one page

Two pointers to action

56

56

ONE PAGE Turn the Curve Report

Result: _______________Indicator

(Lay Definition)Indicator Baseline

Story behind the baseline --------------------------- --------------------------- (List as many as needed)

Partners --------------------------- --------------------------- (List as many as needed)

Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost ---------Off the Wall

SharpEdges

4.

Results-Based Accountability

Documents

Transcript of Results-Based Accountability