Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental...

47
Managing Cross-Departmental Collaboration: A Performance Scorecard for Boston’s Mayoral Sub-Cabinets Part One By Devin Lyons-Quirk and Meghan Haggerty (MPP 2010) Harvard Kennedy School September 2010 PERFORMANCE MANAGEMENT RAPPAPORT INSTITUTE for Greater Boston

Transcript of Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental...

Page 1: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

Managing Cross-Departmental Collaboration:

A Performance Scorecard

for Boston’s Mayoral Sub-CabinetsPart One

By Devin Lyons-Quirk and Meghan Haggerty (MPP 2010) Harvard Kennedy School

September 2010

PERFORMANCE MANAGEMENT

RAPPAPORT INSTITUTEfor Greater Boston

Page 2: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

About This Paper

This working paper, which was originally prepared as a Policy Analysis Exercise, Harvard Kennedy School’s equivalent of a master’s thesis, was named one of the best PAEs written by an HKS student in the 2009-2010 academic year. Devin Lyons-Quirk and Meghan Haggerty, the PAE’s authors, were Rappaport Public Policy Fellows in the summer of 2009 and graduated from the Kennedy School in May 2010.

The Rappaport Institute for Greater Boston

The Rappaport Institute for Greater Boston aims to improve the governance of Greater Boston by fostering better connections between scholars, policy-makers, and civic leaders. The Rappaport Institute was founded and funded by the Phyllis and Jerome Lyle Charitable Foundation, which promotes emerging leaders in Greater Boston. More information about the Institute is available at http://www.hks.harvard.edu/rappaport.

The Taubman Center for State and Local Government

The Taubman Center and its affi liated institutes and programs are the Kennedy School of Government’s focal point for activities that address urban policy, state and local governance and intergovernmental relations.

The Policy Analysis Exercise

The Policy Analysis Exercise (PAE), which is the capstone of the Kennedy School’s Master in Public Policy (MPP) curriculum, is a professional product, meant to clarify and address a practical policy or management problem for a real-world client. The clients for this PAE were three senior offi cials in the City of Boston’s Mayor’s Offi ce: Judith Kurland, Chief of Staff, Barbara Berke, Special Advisor to the Mayor, and Liz Walczak, Human Services Policy Advisor. The faculty advisors were Professor Robert Behn, Lecturer in Public Policy and Professor Mary Ruggie, Adjunct Professor of Public Policy at Harvard’s Kennedy School of Government. More information about PAEs is available at http://www.hks.harvard.edu/degrees/oca/students-alumni/pae.

© 2010 by Devin Lyons-Quirk and Meghan Haggerty. The contents of this paper refl ect the views of the authors (who are responsible for the facts and accuracy of the research herein) and do not represent the offi cial views or policies of the Rappaport Institute for Greater Boston, the Taubman Center for State and Local Government, the Harvard Kennedy School, or Harvard University.

Page 3: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

Managing Cross-Departmental Collaboration:A Performance Scorecard for Boston’s Mayoral Sub-Cabinets

Harvard Kennedy SchoolPolicy Analysis Exercise

Submitted By:

Meghan HaggertyDevin Lyons-Quirk

Master Degree Candidates 2010Public Policy and Urban Planning

Submitted To:

Judith KurlandBarbara Berke Liz Walczak

City of Boston, Mayor’s Offi ce

Supported By:

Professor Bob BehnFaculty Advisor

Professor Mary RuggiePAE Seminar Leader

April 13, 2010 Revision

Page 4: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar t Nov DecFeb Apr

10

100%89%

83%74% 7700%%

4 %

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

4ug S4uu0uugg%gg Sep Octg40gg%gg4 t4 %

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8

u

0

40

Page 5: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

Table of ContentsExecutive Summary. ......................................................................................................... iIntroduction. .....................................................................................................................1Research Methodology. ....................................................................................................5Defi ning Sub-Cabinet Success . .......................................................................................7

The Public Safety Sub-Cabinet . ............................................................................................................................................................ 7The Development Sub-Cabinet . .......................................................................................................................................................... 9The EHHS Sub-Cabinet . ......................................................................................................................................................................10

Selecting Scorecard Metrics. .........................................................................................13Recommended Indicators & Analysis. ..........................................................................21

Metric #1: School Attendance - Number of Students below 80% attendance for the Month . ..........................................21Metric #2: School Attendance – Number of Monthly Absences by Absence Classifi cation . ..............................................25Metric #3: School Attendance – Percentage of Successful Absence Interventions . ..............................................................27Metric #4: After School Program Attendance - Monthly BCYF Attendance by Facility . ......................................................28Metric #5: Library Program Attendance – Monthly BPL Attendance by Facility . ..................................................................31Metric #6: Student Performance - Average Monthly BPS Learning Index Score . ..................................................................34Metric #7: Youth Health – Percentage of Medicaid Eligible Youth Receiving Screenings . .....................................................37Metric #8: Youth Criminal Activity - Interactions with Boston Police Department . ............................................................40Metric #9: Facility Financial Effi ciency – Monthly Upkeep Cost per Square Foot . ................................................................42Metric #10: Citizen Service Operations – Monthly CRM Service Referral Counts by Department . ..............................45

Scorecard Implementation. ..........................................................................................49Supporting Policy Recommendations . ........................................................................55Appendix A: Data Tables. ...............................................................................................58Appendix B: End Notes. .................................................................................................62

Page 6: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 7: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

i

Executive Summary

In a time of limited resources and economic insecurity, the citizens of Boston increasingly demand results from city government. However, the City’s sub-cabinets lack the cross-departmental performance information necessary to (1) describe the benefi ts of their collaborative efforts to citizens, (2) identify and learn from successes, and (3) analyze and improve upon failures. While comprised of exceptionally dedicated public managers, the sub-cabinets lack a critical policy making tool – regular, systematic access to performance data. Therefore, we recommend that the sub-cabinets deploy performance scorecards to explicitly link their monthly efforts to the citywide outcomes they aim to achieve: safer city streets, more affordable living, and better educated residents.

Mayor Menino’s sub-cabinets set city policy in the areas of public safety, economic development, and education and human services. To inform policy choices, troubleshoot problems, and identify innovations, the sub-cabinet leaders require regular access to results-orientated information. Simply put, these leaders need a way to answer the questions: “Did our efforts last month improve services for the City of Boston? If so, how? If not, why?” Performance scorecards will help provide answers.

A successful cross-departmental scorecard targets only a limited set of strategically selected performance metrics. These metrics must apply across all sub-cabinet departments and relate to monthly operations. To prevent scorecards from becoming useless and irrelevant data dumps, they must be built using the following fi ve criteria.

• Focus Efforts: Limit the scorecard to 10 performance metrics. The sub-cabinet meetings are held monthly and last only 90 minutes. In order to stimulate actionable, strategic policy discussion, the scorecard should contain no more than 10 metrics. Any more than 10 metrics may spread attention and resources too thinly, wasting precious meeting time and limiting momentum for service improvement.

“Did our efforts last month improve services for the City of Boston? If so, how? If not, why?”

Managing Cross-Departmental Collaboration:A Performance Score-Card for Boston’s Mayoral Sub-Cabinets

Page 8: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

ii

Keys To Scorecard Success:

Focus Efforts:by limiting the scorecard to 10 metrics

Make the Analysis Count:by selecting only “consequential” metrics

Generate Momentum: by deploying the scorecard monthly

Facilitate Collaboration:by focusing on cross-departmental metrics

Achieve Balance:by analyzing performance through multiple lenses

• Make the Analysis Count: Utilize “consequential” performance measures. Consequential measures are: (1) based on reliable quantitative information; (2) have diagnostic and/or predictive relevance; and (3) can be compared against relevant benchmarks to create goals.

• Generate Momentum: Update the sub-cabinet measures monthly. Performance measures are most useful if they give the group monthly feedback at the sub-cabinet meeting. Therefore, it is important that the data have the capacity to change on a monthly basis. Without operational, monthly information, the sub-cabinet may lose momentum.

• Facilitate Collaboration: Select cross-departmental metrics that all sub-cabinet departments can impact. Shared measures support the sub-cabinets’ function – to catalyze cross-departmental initiatives – and enable accountability and critical thinking at the cabinet level. A focus on common goals requires performance measures that all the sub-cabinet departments can potentially infl uence. While performance metrics tied to the performance of single departments are important in other venues, the sub-cabinets’ purpose is to support cross-departmental strategies, and therefore require cross-departmental performance analysis.

• Achieve Balance: Analyze performance through multiple lenses. Performance information comes from a variety of sources including operational statistics, fi nancial records, employee inputs and customer experiences. Using lessons from the balanced scorecard, the set of metrics on the sub-cabinet scorecard should speak to multiple aspects of performance. While it is appropriate to focus on service delivery to citizens, the internal and fi nancial processes of sub-cabinet departments should not be ignored in the scorecard.

While each sub-cabinet can benefi t from a scorecard, this report begins by focusing on the Education, Health and Human Services (EHHS) sub-cabinet, which will serve as a roadmap for producing similar scorecards for the Public Safety and Economic Development sub-cabinets.

The performance of Boston’s Education, Health and Human Service departments depends on cross-departmental collaboration. Data-driven scorecard analysis will strengthen the partnerships necessary to meet the needs of Boston’s most underserved. Given the tremendous potential impact that service delivery improvement could yield, we choose the EHHS sub-cabinet as our pilot study.

ii

Page 9: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

iii

Over the past four months, we conducted qualitative and quantitative research to uncover the best set of EHHS sub-cabinet measures. We interviewed delegates from each EHHS sub-cabinet department and paid careful attention to department-specifi c views on using performance metrics for collaboration. Guided by these interviews, we collected available data and catalogued data needs. We analyzed data trends both geographically and chronologically to uncover the stories this information told about access to successful city services.

Based on our research, we recommend the following set of 10 performance measures for the EHHS scorecard, which we believe have the potential to change the way the sub-cabinet approaches performance:

1) School Attendance: The number of students in Boston Public Schools who have a less than 80% school attendance rate for the previous month, reported by school.

2) School Attendance: The number of student monthly absences from Boston Public Schools, reported by school, broken down by absence classifi cation (e.g., illness, behavior problems, poor academics, unstable housing).

3) School Attendance: Monthly percentage of chronically absent youth receiving intervention and case management services that led to improved school attendance.

Executive Summ

ary

The performance of Boston’s Education, Health and Human Service departments depends on cross-departmental collaboration.

Data-driven scorecard analysis will strengthen the partnerships necessary to meet the needs of Boston’s most underserved.

Page 10: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

iv

4) After School Programming Attendance: Monthly Boston Center for Youth and Families (BCYF) attendance by facility.

5) Library Programming Attendance: Monthly Boston Public Library (BPL) attendance by program type (child, young adult and adult) by library branch.

6) Student Performance: Average monthly score on Boston Public Schools Student Learning Index. (This score quantifi es student performance and predicts dropout risk by student.)

7) Youth Health: Percentage of children eligible for Medicaid primary care screening who received their allotted screening (currently, this is only an annual number; see Chapter 5 for recommendations on creating a future monthly health metric).

8) Youth Criminal Activity: Number of monthly youth negative interactions with police (negative interactions are defi ned as either arrest or fi eld intelligence write-up).

9) Facility Financial Effi ciency: Monthly facility upkeep cost per square foot broken down by EHHS department (currently monthly data does not exist - annual data will be used in the interim until monthly collection is achieved).

10) Citizen Service Operations: Monthly Citizen Request Management (CRM) service requests that come through the Mayor’s Hotline and are referred to each EHHS department.

Exec

utiv

e Su

mm

ary

April 2010 EHHS Sub-Cabinet MeetingLaunch EHHS Scorecard and hold fi rst • discussion based on available dataEHHS departments give feedback and make • recommendations for fi nal scorecardIncorporate scorecard performance metrics • into the FY11 Budget where possible

May 2010 EHHS Sub-Cabinet MeetingHold second scorecard discussion• Identify person to fi ll EHHS “Scorecard • Coordinator” roleDesignate departmental leads to collect • currently unavailable data

EHHS Scorecard Implementation Timeline

June 2010 EHHS Sub-Cabinet MeetingThird scorecard discussion• Set specifi c annual goals for each • scorecard metricEnsure EHHS Scorecard Coordinator • is receiving adequate departmental support

iv

The scorecard must be built around a limited

set of strategically selected performance metrics. These metrics must apply across all

of the departments and relate to monthly

operations.

Page 11: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

v

This scorecard and its associated measures should not be static – as the sub-cabinets evolve, so should their measures. Once the EHHS scorecard is institutionalized, the group should continue to analyze and refi ne the measures. The metrics above represent the best information available now or in the near future, but as new data systems and initiatives come online in Boston, new metrics may advance the sub-cabinet’s capacity to manage performance.

The following report is both a roadmap for implementing performance scorecards and an analysis of currently available cross-departmental EHHS performance data. This report explains the purpose underlying cross-departmental scorecards and the logic that led to the development of the current EHHS measures. Other sub-cabinets can refer to the principles developed here when creating scorecards for themselves, as can the EHHS sub-cabinet as it creates future iterations of the EHHS scorecard.

Below is a timeline for taking the policy recommendations in this report and implementing them as a fully functioning scorecard program. While this report includes an analysis of the currently available scorecard metrics, this is only a snapshot in time. The real impact will come from continual cross-departmental performance discussions at each monthly sub-cabinet meeting.

Ultimately, instituting a regular and systematic discussion of performance data within the existing sub-cabinet structure will provide managers with the information they need to provide effi cient, innovative, and exceptional services to the citizens of Boston.

Executive Summ

ary

Mid Fall EHHS Sub-Cabinet MeetingsRecruit new Kennedy School PAE team to • evaluate scorecard use to date or establish scorecard metrics for other sub-cabinetsContinue presenting scorecards and • discussing potential for new metrics

Early Fall EHHS Sub-Cabinet MeetingsContinuing holding regular scorecard • discussions at each meetingEvaluate progress towards goals• Use scorecard to identify and publicize an • EHHS program success

Winter 2011 EHHS Sub-Cabinet MeetingsCompare scorecard performance against • annual strategic goalsEvaluate how nearly a year of scorecard use • has aided performance

v

Page 12: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

vi

Exec

utiv

e Su

mm

ary Current Scorecard Data

Metric #1: School Attendance - the number of BPS students below 80% school attendance:

The monthly number of students below 80% attendance has increased every month, peaking in February (the most recent data). 9th grade students have the worst attendance rates, while 4th graders have the best. See maps and Ch. 5 for more details.

Metric #2: School Attendance - The number of student monthly absences from Boston Public Schools, reported by school, broken down by absence classifi cation (e.g., illness, behavior problems, poor academics, unstable housing).

This metric is still in development, see Ch. 5 for recommendations

Metric #3: School Attendance - Monthly percentage of chronically absent youth receiving intervention and case management services that lead to their improved school attendance.

This metric is still in development, see Ch. 5 for recommendations

Metric #4: After School Programming Attendance - Monthly BCYF gate counts

BCYF monthly visits have remained near 20,000 throughout the winter. More signifi cant is the change from center to center which is illustrated in Map 2. See Ch. 5 for discussion.

Metric #5: Library Programming Attendance - Monthly BPL attendance by program type

While the number of children’s programs has increased since the summer, the number of children attending has declined. Adult attendance increased in January but declined in February. Map 4 shows 2009 gate counts by branch. More discussion in Ch. 5

0

1000

2000

3000

4000

5000

6000

7000

Sept Oct Nov Dec Jan Feb

System-wide Number of Students below 80% Monthly Attendance

0

50000

100000

150000

200000

250000

300000

350000

Jul 09 Aug 09 Sep 09 Oct 09 Nov 09 Dec 09 Jan 10

BCYF Monthly Visits Jul 09 to Jan 10

0

5000

10000

15000

20000

July Aug Sept Oct Nov Dec Jan

Pro

gra

m A

tte

nd

ee

s

Number of Persons Attending BPL Programs

Adult Attended Young Adult Attended Children Attended

No datayet available

No datayet available

Map 1 (above) shows the schools with improving (green) and deteriorating (red) attendance rates for February 2010

Map 2 (below) shows the BCYF centers with improving (green) and deteriorating (red) gate counts for January 2010

Page 13: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

vii

Current Scorecard Data Executive Summ

ary

Metric #6: Student Performance - Average monthly score on Boston Public Schools Student Learning Index (which quantifi es student performance and predicts dropout risk by student).

This metric is still in development, see Ch. 5 for recommendations

Metric #7: Youth Health - Percent of eligibles receiving Medicaid primary care screening

The percent of eligible children who receiving their allotted Medicaid health screenings declines with age from nearly 100% of infants to 40% of 13 to 20 year olds. This data is for 2008. Monthly data is recommended, see Ch. 5 for details.

Metric #8: Youth Criminal Activity - Monthly number of negative youth interactions with police (negative interactions are defi ned as either arrest or fi eld intelligence write-up).

This metric is still in development, see Ch. 5 for recommendations

Metric #9: Facility Financial Effi ciency - Monthly facility upkeep cost per sq. ft., by department

For FY09 BCYF spent much less per square foot to maintain their facilities. This may be because BCYF often collaborates with other departments to use their space - an example of a money-saving consolidation. Monthly data not yet available, see ch. 5.

Metric #10: Citizen Service - Monthly CRM citizen service requests for EHHS depts

2009 data indicates that service requests for EHHS departments are highly seasonal - homelessness issues rise in the summer, elderly issues in the winter, and school issues in the fall. Numbers for some departments are extremely low, indicating the need for advertising. See Ch. 5.

0

50

100

150

200

250

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

2009 Total EHHS Service Requests

No datayet available

No datayet available

Total Cost per Sq. Ft.:BPS = $3.28BPL = $3.94BCYF = $1.44

Map 3 (above) shows the number of students below 80% attendance by school for February 2010

Map 4 (below) shows the number of library visits by branch for 2009

100%89%

83%74% 70%

57%

40%

0%

20%

40%

60%

80%

100%

120%

Age <1 Ages 1-2 Ages 3-5 Ages 6-9 Ages 10-14

Ages 15-18

Ages 19-20

2008 % of Total Eligibles Receiving Screening

Page 14: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

viii

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 15: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

1

IntroductionChapter 1:

In recent years, Mayor Menino’s administration has been increasingly focused on using performance measures and sophisticated technology to enhance city services. However, this focus on performance management is not refl ected in the way the sub-cabinets hold meetings. While the sub-cabinets have successfully implemented new citywide strategies, the meetings are generally limited to a series of briefi ngs aimed at improving cross-departmental communication. The sub-cabinets have not explicitly linked their outputs to citywide outcomes.

The sub-cabinets need a tool to transform their meetings from information-sharing sessions into action-orientated discussions that yield new, results-based strategies. Sub-cabinet members need to be able to leave the monthly meetings with a shared understanding of cross-departmental priorities and a specifi c knowledge of who is accountable for what action steps before the next sub-cabinet meeting. They must be able to speak to the public with common and consistent messages on where the sub-cabinet departments are investing their resources, the goals they hope to achieve, and the successes they have delivered. Monthly scorecards will bring this needed focus to the sub-cabinet discussions, allowing them to be conduits for true performance-based leadership.

This Policy Analysis Exercise (PAE) was developed in response to a request by the City of Boston Mayor’s Offi ce to research the use of performance scorecards for the three policy sub-cabinets. In 2008, Mayor Menino established three cross-departmental groups called sub-cabinets in order to form citywide strategies for Public Safety, Economic Development, and Education and Human Services. City departments with relevant policy mandates send representatives to the sub-cabinet meetings. Sub-cabinet leaders meet monthly to discuss shared responsibilities and policy issues that require a cross-departmental approach.

The sub-cabinets need a tool to transform their meetings from information-sharing sessions into action-orientated discussions

Page 16: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

2

Prior Performance Management Efforts in BostonCity of Boston leaders are well versed in the value of performance management. Over the past several years, the city has invested in the Boston About Results (BAR) program to capture performance data and measure results by department. Through the BAR program, the City collects program and service data and reports it publicly by quarter.

The objectives of BAR align closely with Mayor Menino’s priorities around improving performance, customer satisfaction, and responsiveness to citizen needs.1 The City’s experience with BAR has enforced the varied benefi ts of measuring performance. Table 1 outlines the reasons managers may choose to measure performance, whether these measurements are reported through a scorecard or a system like BAR.

The BAR program is distinct from sub-cabinet scorecards in that the BAR measures are department-specifi c while the sub-cabinet scorecard is cross-departmental. Additionally, BAR is a repository for data on a large number of performance metrics, while a sub-cabinet scorecard focuses on only 10 high priority indicators.

The existing BAR data management platform can be leveraged to capture and report the performance metrics developed for the sub-cabinet scorecards. The functionality exists within the BAR platform to tag measures as cross-departmental initiatives. Therefore, incorporating the

Figure 1: Eight Purposes that Public Mangers have for Measuring Performance

The Purpose: The question that performance measurement can help answer:

Evaluate How well is my public agency performing?Control How can I ensure that my subordinates are doing the

right thing?Budget On what programs, people, or projects should my agency

spend the public’s money?

Motivate How can I motivate line staff, middle managers, nonprofi t and for-profi t collaborators, stakeholders, and citizens to do the things necessary to improve performance?

Promote How can I convince political superiors, legislators, stakeholders, journalists, and citizens that my agency is doing a good job?

Celebrate What accomplishments are worthy of the important organizational ritual of celebrating success?

Learn Why is what working or not working?Improve What exactly should who do differently to improve

performance?Adapted from Why Measure Performance: Different Purposes Require Different Measures, Bob Behn, 2003

Table 1:This table outlines the reasons managers may choose to deploy a performance management tool like a performance scorecard

The BAR program is distinct from sub-cabinet scorecards in that the BAR measures are department-specifi c while the sub-cabinet scorecard is cross-departmental.

Page 17: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

3

cross-departmental scorecards into the BAR system is just a matter of ensuring that metrics are added to the system and then tagged as a sub-cabinet scorecard initiative. This addition will not only make BAR more robust, but it will also serve as an example of using performance data to support strategic management decision-making.

The Use of Performance ScorecardsDeveloped in the private-sector, performance scorecards are now generally accepted public management tools. They lay at the core of many public sector management innovations for providing greater government effi ciency, transparency, accountability and ultimately, better service for citizens. More the just a government “report card,” a successful performance scorecard allows sub-cabinet members to:

• Collaborate with one another on shared objectives,• Spot trends in service delivery which may lead to insights on strategies

to solve problems and enhance performance,• Assign roles and responsibilities and improve overall government

accountability,• Strategize to maximize service delivery and prevent duplication of

efforts,• Redirect resources to have the greatest impact on service

objectives,• Improve the level of service delivered to the Citizens of Boston.

In the following report, we examine in detail the process of creating sub-cabinet scorecards that meet the objectives above. In Chapter 3, we discuss the position of each sub-cabinet and defi ne what a successful scorecard may look like for each. In Chapter 4, we present a detailed framework for selecting metrics for a sub-cabinet scorecard. In Chapter 5, we apply this framework to developing a pilot scorecard for the Education, Health and Human Services sub-cabinet. Also in Chapter 5 we closely examine each of the 10 selected EHHS scorecard metrics and what the currently available data says about performance. Following this analysis, we turn to implementation, discussing timelines for creating scorecards on a month-to-month basis as well as a long-term timeline for continuing the work done to date and persistently improving the EHHS scorecard. Finally, we conclude with some general policy recommendations to support successful scorecard development.

Developed in the private-sector, performance scorecards are now generally accepted public management tools.

Page 18: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

4

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 19: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

5

The process for developing the scorecard and recommendations presented in this report evolved in three distinct stages: (1) background research and understanding of sub-cabinet challenges and opportunities, (2) selection of potential EHHS scorecard metrics, (3) data analysis and policy recommendation formulation. We began this project in Early September and the work presented here was completed in March. Across all three stages from beginning to end we utilized a number of research techniques:

• Observation and Participation in Sub-Cabinet Meetings: we attended every EHHS sub-cabinet meeting since November as well as many Public Safety and Development sub-cabinets. We also attended other meetings relevant to performance management such as the Boston Police CompStat meeting.

• Interviews with City of Boston Offi cials: With several dozen interviews serving as our primary source of information, we were lucky to fi nd city government offi cials who were engaged by the project and exceptionally generous with their time. We met with representatives from the EHHS sub-cabinet departments, staff from the Mayor’s Offi ce, and various members of the other sub-cabinets.

• Academic Literature Review: Our respondents gave useful anecdotal evidence in their interviews, but we also consulted broader academic research on the proposed performance measures. An example of this is with the school attendance metric: we sought academic articles discussing the extent to which attendance is predictive of student outcomes, and we looked at research on Massachusetts and Boston Public Schools attendance rates.

• Data Analysis: Upon identifying a preliminary set of performance measures, we requested access to relevant data from appropriate departments. We then analyzed this data looking to identify trends, performance variances, and other information relevant to sub-cabinet members.

• Geographic Analysis: For data that could be captured by location, we used Geographic Information System tools to compare metrics by facility, neighborhood, or district. We also identifi ed areas where geographic analysis would be relevant should the data become available.

• Working Groups with the EHHS Sub-Cabinet: During the course of our project we scheduled three working groups with the sub-cabinet. We conducted a short introductory discussion in December. At the February sub-cabinet, we presented our preliminary fi ndings and facilitated a lengthy discussion to gather information and generate support for the project. Finally, at the upcoming April sub-cabinet we will present the group with our fi nal recommendations.

Research MethodologyChapter 2:

Page 20: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

6

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 21: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

7

The purpose of the sub-cabinet meetings is different than that of the overarching Mayor’s Cabinet. The Mayor chairs the Cabinet, and only department heads attend, while the Mayor’s senior advisors chair the sub-cabinet meetings, and both department heads and senior project managers attend. This distinction between cabinet and sub-cabinet allows the sub-cabinet meetings to be less formal, but more importantly, it permits sub-cabinet members to address both strategic and operational issues within a specifi c policy purview.

In the sections below, we discuss each sub-cabinet’s specifi c needs. While the depth of our analysis was primarily focused on developing a performance scorecard for the Education, Health and Human Services (EHHS) sub-cabinet, our client requested that we provide a preliminary analysis for the Development and Public Safety sub-cabinets. The following analysis can be used to expedite the creation of performance scorecards for these sub-cabinets in the near future.

The Public Safety Sub-CabinetThe Mayor’s Offi ce chairs the Public Safety Sub-Cabinet, and it includes representatives from the Boston Police Department, Boston Emergency Medical Services, the Boston Fire Department, the Offi ce of Emergency Preparedness, the Chief Information Offi cer, and members of the Administration and Finance staff (see Figure 2).

Our observations and discussions indicate that, compared to the other sub-cabinets, the Public Safety sub-cabinet excels in its ability to address operational capacity issues and build new public safety capabilities. However, a heavy near-term focus prevents Public Safety from discussing long-term, cross-departmental strategies for innovation in public safety service delivery.

Chapter 3:

Defi ning Sub-Cabinet Success

The Mayor’s sub-cabinets were established as cross-departmental working groups bringing senior leaders together around each of three major policy areas: (1) Public Safety; (2) Development, and (3) Education, Health and Human Services.

The Public Safety Sub-Cabinet excels in its ability to address operational capacity issues and build new public safety capabilities.

Page 22: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

8

A successful performance scorecard for this sub-cabinet must be truly cross-departmental. While each Public Safety department clearly understands its own operational performance, every department needs to convey this information to the others. For example, the Fire Department and EMS both know their own monthly performance statistics on medical emergency response. However, the two departments do not have a clear understanding of the statistics for their joint response.

A sub-cabinet scorecard can clarify how departmental efforts work in concert, and it can help the departments identify opportunities to learn from each other’s operations. Furthermore, the scorecard should include metrics that address long-term public safety priorities and opportunities in order to foster strategy discussions on service delivery. Therefore, we recommend that efforts to create a Public Safety scorecard include investigating the following performance metrics:

• Response times from the citizen perspective: While each department has a fi rm knowledge of its own response time performance, sharing this information with the sub-cabinet and linking Police, Fire and EMS response times to types of emergency calls will illuminate citizens’ perceptions of response performance for the sub-cabinet departments.

• Percentage of overtime hours per month broken down by overtime classifi cation: Comparing this information will allow public safety agencies to identify and implement best practices for managing overtime, which is a budget priority for each service.

Public Safety Education, Health & Human Services

Economic Development

Public Schools• Public Health Commission• Boston Center for Youth and • FamiliesPublic Libraries• Boston Housing Authority• Thrive in Five• Boston After School and • BeyondBoston Youth Fund• Jobs and Community Services • Dept.Elderly Commission• Intergovernmental Relations• Mayor’s Offi ce• Chief Information Offi cer• Administration and Finance•

Boston Redevelopment • AuthorityBoston Housing Authority• Neighborhood Development• Small and Local Business • EnterpriseJobs and Community Services• Inspectional Services Dept.• Public Works and • Transportation DepartmentEnvironment Department• Mayor’s Offi ce• Chief Information Offi cer• Administration and Finance•

Police Department• Fire Department• Emergency Medical Services• Homeland Security• Emergency Preparedness• Mayor’s Offi ce• Chief Information Offi cer• Administration and Finance•

Figure 2: The Mayor’s three policy sub-cabinets and participating departments

The Development sub-cabinet’s strength is its focus on long-term goal setting and strategic decision-making.

Page 23: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

9

• Fleet maintenance costs per vehicle, percentage of preventative maintenance performed on schedule, and total vehicle out-of-service hours: While the missions of each public safety agency vary, fl eet maintenance procedures across departments should be relatively standard. Sharing this information will help Public Safety departments identify problem areas, share best practices, and investigate opportunities to bring down costs by consolidating maintenance efforts.

• Average monthly training hours per employee and training facility utilization: The performance of police, fi re, EMS and emergency management personnel is directly linked to their training and experience. It is critical for these agencies to coordinate their training efforts in order to enhance the City’s integrated response capacity. Analyzing these fi gures will focus the sub-cabinet on the importance of training. Furthermore, facility and staff utilization rate analysis will identify opportunities for Public Safety departments to share facilities and/or possibly contract their unused training capacity to other municipalities.

• Service calls to and from other municipalities and private entities: City of Boston public safety personnel routinely respond to emergencies outside the city limits. Similarly, other cities and private/non-profi t service providers also respond to emergencies within Boston. Tracking this cross-municipal information at the sub-cabinet level will foster a regionalized approach to public safety issues that could identify possible service improvements and cost savings.

The Development Sub-CabinetThe Development sub-cabinet focuses on economic development issues within the City of Boston and is attended by the Boston Redevelopment Authority (BRA), the Department of Neighborhood Development (DND), the Boston Housing Authority, (BHA) the Small and Local Business Enterprise, the Department of Jobs and Community Services (JCS), the Inspectional Services Department (ISD), the Public Works and Transportation Department, the Environment Department, the Chief Information Offi cer, Administration and Finance, and the Mayor’s Offi ce (see Figure 2).

Compared with the other sub-cabinets, the Development sub-cabinet’s strength is its focus on long-term goal setting and strategic decision making. At the meetings we attended, we witnessed most discussions gravitating toward identifying goals and setting priorities. However, this focus on long-term goals can lead to a lack of discussion around important operational issues.

A successful Development sub-cabinet scorecard should provide information on the city’s performance toward the long-term strategic objectives developed by the group. However, in order to be useful, the scorecard must include operational information, using short-term feedback mechanisms to predict outcomes toward these long-term goals.

While the potential performance management avenues for the Development sub-cabinet warrant much more study, some initial areas to investigate include:

• Aligning metrics to the three part framework for development: This includes the Economic Environment, the Physical Environment and the Human/Information Environment.

• Measuring sustainability and support of the green economy: Possible metrics include weatherization installations, energy consumption rates and green job creation.

Page 24: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

10

• Economic and Industry Indicators: Conversations with experts at the BRA could yield potential leading indicators of job growth, industry trends, tourism rates, and other factors critical to Boston’s economy.

• Permitting Process Metrics: Metrics such as average time between permit application and approval, average time of environmental review process, or other project and process benchmarks can provide insights on the City’s effi ciency in handling the development process.

• Development Milestones: Working with project and department leaders to develop milestones for long-range projects and then tracking performance toward reaching those milestones over time.

The EHHS Sub-CabinetThe Education Health and Human Services (EHHS) sub-cabinet is the largest Mayoral sub-cabinet. The Mayor’s Offi ce chairs the group, and it includes representatives from across Boston’s human service departments, the Chief Information Offi cer, Administration and Finance, and Intergovernmental Affairs. For a complete list of attendees, see Figure 2.

The EHHS sub-cabinet’s strength lies in its members’ eagerness for cross-departmental collaboration and its general agreement on long-term strategy. While the EHHS sub-cabinet’s broader mandate is to serve the

Mayor Menino’s FY10 Budget Priorities

Investing In Youth Invest in Early Childhood Education

Accelerate K-12 Student Achievement Support Student Achievement After School and Year Round

College Readiness and Completion

Improving Our Neighborhoods Increasing Public Safety

Modernizing Basic City Services Greening Our Environment

Improving Public Health

Strengthening the Economy Supporting Private Sector Growth

Stabilizing the Housing Sector Increasing Access to Benefi ts & Training

Investing in Critical Citywide Infrastructure

Figure 3: Mayor Menino’s Fiscal Year 2010 budget priorities are grouped into three categories. The priorities relevant to the Education, Health and Human Service sub-cabinet are highlighted in orange italics.

A strength of the EHHS sub-cabinet is the relative consensus among its members on long-term goals.

Page 25: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

11

entire spectrum of Boston’s population, it has targeted its efforts toward improving outcomes for the City’s youth. A February, 2009 draft of the EHHS sub-cabinet strategy states that the group’s overarching goal is to “Provide an environment where all children and families are supported in reaching their fullest potential.” This strategic focus drives the sub-cabinet’s priorities and actions.

While Boston benefi ts from the EHHS sub-cabinet’s strategic focus on youth, the group’s currently stated focus is not measurable. It is impossible to measure each child’s fullest potential; therefore, it is impossible to gauge whether the EHHS sub-cabinet is getting better or worse at reaching its goal.

The Mayor’s priorities (shown in Figure 3) provide a more detailed picture of where the sub-cabinet’s efforts should be focused. In fact, a strength of the EHHS sub-cabinet is the relative consensus among its members on long-term goals. Looking at the Mayor’s priorities, a citizen can infer that long-term metrics such as dropout rates, MCAS test scores, and college completion rates could be used to track sub-cabinet success. While these metrics are important, they are captured on long time horizons and are therefore less useful for informing operational decision making around education, health and human service policy.

Our recommendation, based on exhaustive interviews with sub-cabinet members, a detailed literature search, and a number of working groups, is for EHHS to develop a sub-cabinet scorecard that includes metrics bridging the gap between operations and strategy. The rest of this report is dedicated to developing a framework for selecting these metrics, identifying a list of ten truly consequential EHHS scorecard metrics, and providing policy recommendations for implementation of the scorecard.

EHHS Sub-Cabinet Overarching Goal:

“To provide an environment where all

children and families are supported in reaching their fullest potential.”

(March 2009)

Areas of Sub-Cabinet Focus:Education and Youth • DevelopmentHealth and Health Equity• Family and Community • Prosperity

Page 26: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

12

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 27: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

13

• Focus Efforts – by ensuring the scorecard does not exceed ten metrics

• Make the Analysis Count – by identifying measures that are meaningful and consequential

• Generate Momentum – by ensuring that metrics can be measured and infl uenced on a monthly basis

• Facilitate Collaboration – by only selecting metrics that are aligned to cross-departmental strategy and whose outcomes can be infl uenced by all sub-cabinet departments

• Achieve Balance – by seeking a set of metrics that analyze performance through multiple perspectives including the customer’s and that of the internal process

“Not all performance measures are created equally,” writes Paul Niven, author of The Balanced Scorecard for Government and Nonprofi t Agencies, “Effective metrics provide direction, align employees, improve decision-making, and serve as a basis for resource allocation decisions.”2 We analyzed all potential metrics against the fi ve criteria listed above in order to provide the EHHS sub-cabinet with a performance scorecard that can help them achieve these goals. In Chapter 5, we describe how EHHS scorecard metrics align to these principles. First, we discuss the importance of the fi ve criteria in greater detail.

Chapter 4:

Selecting Scorecard Metrics

A sub-cabinet scorecard must be built around a limited set of strategically selected performance metrics in order for it to be successful. This section outlines the theory and process used to evaluate potential scorecard metrics which led to the selection of 10 priority metrics for the EHHS sub-cabinet. Our criteria for selecting metrics boils down to fi ve strategic recommendations:

“There is a lot of ‘noise’ in modern organizations, and a good balanced scorecard should rise above the crowd, providing you with a view of the real drives of success in your organization.”

Page 28: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

14

Focus Efforts: Limit the Number of Scorecard Metrics to TenSuccessful scorecard metrics are focused on quality not quantity. In fact, even if all the metrics are of high quality, a scorecard loaded with too many metrics will misdirect efforts, confuse priorities, and ultimately limit results. A performance scorecard must refl ect the priorities of the sub-cabinet. Just as with any management strategy, identifying too many priorities decreases the likelihood that any one priority will be addressed, and makes the process of successful implementation diffi cult.

Paul Niven writes, “There is a lot of ‘noise’ in modern organizations, and a good balanced scorecard should rise above the crowd, providing you with a view of the real drives of success in your organization.”3 By limiting the number of indicators, the sub-cabinet will be able to focus on those measures that truly matter and that the City is able to infl uence.

The sub-cabinet meetings are held monthly and only last ninety minutes. Given this time constraint, a sub-cabinet scorecard should not contain more than ten metrics, which signify the group’s priorities. It is certainly reasonable for the City to track other measures and to call upon them to inform strategy and operations when necessary, but including any more than ten scorecard measures will dilute the sub-cabinet’s ability to focus on its priorities and hold itself accountable to shortcomings.4 As time passes, if better metrics become available for sub-cabinet scorecards, we recommend that they replace rather than add to the existing scorecard metrics.

Make the Analysis Count: Identify Consequential MeasuresAn effective scorecard is built on effective measures. We use the term “consequential” to defi ne those metrics that are (1) based on reliable quantitative information (2) have diagnostic and/or predictive relevance and (3) can be compared against relevant benchmarks in order to create goals.

First and foremost, consequential measures must in fact be measurable. The scorecard goal is to measure performance; therefore, the metrics included in the scorecard must be quantitative. This allows managers to analyze trends, quantify the impact of initiatives, and make decisions based on the best information available.

Measures that are not quantitative preclude any detailed analysis. For example, if the EHHS sub-cabinet listed “improving youth development” as a scorecard metric, it would be diffi cult to measure progress on this metric because it is not be immediately clear when youth development hits the critical value of “improved.” Instead, the EHHS sub-cabinet could track measures such as school attendance, community center attendance, and youth involvement with police. All of these metrics are easily quantifi able and can be analyzed for trends that will allow the sub-cabinet to know

Focus Efforts:by limiting the scorecard to 10 metrics

Make the Analysis Count:by identifying measures that are meaningful and consequential

Page 29: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

15

when service is improving and when it is deteriorating. Most importantly, managing upward performance on these quantitative metrics will contribute toward the ultimate goal of improving youth development.

In order to provide meaningful performance information, the scorecard measures must either diagnose past performance or provide predictive information about future outcomes. The very best consequential metrics have both these properties – diagnostic and predictive – and thereby provide for an analysis of past performance while allowing managers to forecast future results. These properties are often referred to as lagging and leading indicators in operations management literature (see Table 4).

Finally, a consequential measure must have meaningful benchmarks available in order for managers to understand what it means and to set appropriate goals. For example, we may know that Boston Public Library system served 6,000 youths last month. However, this information is meaningless unless we have something to compare it against. Benchmarks may come from historical data sources (e.g., the number of youth that used the library in the same month last year), from internal comparisons (e.g., the number of youths using the libraries in South Boston versus the libraries in Roxbury) or from external sources (e.g., the average number of youths per capita using the library services in Boston compared to Baltimore or Chicago).

Generate Momentum: Ensure Measures are Relevant on a Monthly BasisDeploying a monthly scorecard offers several advantages. First, it ensures that the scorecard is discussed each month thereby consistently keeps the scorecard measures and goals in the minds of sub-cabinet members.

Figure 4: Lagging vs. Leading Measures

Lagging Measures Leading MeasuresDefi nition Backward-looking performance indicators

that represent the results of previous actions. Normally measuring historical performance at the end of a time period.

Forward-looking measures that “drive” or lead future performance on lag measures. Normally measuring intermediate processes and activities.

Examples MCAS Test ScoresRevenueMortality RateEmployee Satisfaction

Student/Teacher RatioGrants WrittenImmunization RateEmployee Absenteeism

Advantages Normally easily to identify, capture and measure. Often bottom-line “success” measures for the organization.

Predictive in nature, therefore allowing the organization to make mid-course adjustments based on results.

Weaknesses Historical in nature and therefore do not refl ect current activities or predict future success.

May prove diffi cult to identify and capture. Are only relevant to the exent they accurately predict performance on key lagging measures.

Adapted from Balanced Scorecard Step-by-Step for Government and Nonprofi t Agencies, Paul Niven, 2002

Figure 4: While leading measures are often more useful as they allow managers to predict results and change course before outcomes happen, this type of predictive indicator is more diffi cult to identify and capture.

Generate Momentum:by ensuring that measures can be measured and infl uenced on a month to month timeframe

Page 30: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

16

Secondly, it builds the sub-cabinet participants’ expectations that they must take steps to positively impact the scorecard measures. Third, it provides sub-cabinet members with rapid feedback on how their previous month’s efforts led to results.

However, none of these advantages are possible if measures are not selected based on their appropriateness for monthly reporting. Most importantly, there must be a reasonable possibility that the selected measures can change over a short time period. Continually focusing on metrics which city departments have little ability to infl uence in the short term will sabotage the scorecard effort. At the very least, departments will fi nd little use in referring to the scorecard, or worse, the scorecard may lead them to feel defeated. Therefore, a useful scorecard should measure those indicators through which focused cross-department strategies can likely lead to success.

Additionally, the data must be regularly available to be useful. While Massachusetts Comprehensive Assessment System (MCAS) test scores may measure the impact of the EHHS community’s daily efforts to improve student performance, the MCAS test is only administered and reported annually. Therefore, if MCAS test scores were included on a monthly scorecard, the metric would only change once per year, providing no feedback on month-to-month subcabinet efforts. Similarly, if test scores were collected monthly, but only at the highest performing schools, the quality of the information would be low and mislead scorecard users. To inform strategy, data must be collected consistently and accurately.

Facilitate Collaboration: Use Truly Cross-Departmental MetricsA sub-cabinet scorecard must be relevant to all departments represented in the meeting and be targeted toward the group’s shared agenda. Therefore, the scorecard measures should not exclusively target a single department’s performance. Instead, the scorecard metrics should measure collaborative, cross-departmental performance.

For example, the student-teacher ratio in BPS classrooms may be a strong indicator of student learning outcomes; however, few departments other than BPS can infl uence this metric. Metrics such as this do not belong on a cross-departmental scorecard. An example of a cross-departmental metric is school attendance. While school attendance is tracked by BPS, nearly all departments on the EHHS sub-cabinet play a role in delivering services necessary to keep students in school: BPHC provides health interventions, BCYF engages students in after-school actives, Jobs and Community Services supplies social services to students and families, the Boston Housing Authority delivers stable housing, etc. The sub-cabinets were created to address the cross-departmental, multi-faceted needs of citizens; therefore, sub-cabinet scorecards must refl ect these complexities.

Facilitate Collaboration:by focusing on cross-departmental metrics

Continually focusing on metrics which city departments have little ability to infl uence in the short term will sabotage the scorecard effort.

Page 31: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

17

Achieve Balance: Select a Set of Metrics to Measure Performance from Multiple AnglesPerformance information comes from a variety of sources including operational statistics, fi nancial records, employee inputs, and customer experiences. The best set of metrics for the sub-cabinet scorecard will balance a diversity of performance perspectives.

Robert Kaplan and David Norton write that “that no single measure can provide a clear performance target or focus attention on the critical areas of the business. Managers want a balanced presentation of both fi nancial and operational measures.” They compare running an organization to fl ying an airplane, and the various performance metrics of the organization to the dials and indicators of the airplane’s cockpit.5 Relying on a single instrument or set of instruments can be fatal – the plane is likely to crash. In the same way, an organization that is led by managers using performance measures focused on only one aspect of the business is also doomed. Therefore, in constructing a scorecard there needs to be a balance of measures that are drawn from a variety of sources and perspectives.

Similarly, a central tension in the course of this project has been to balance strategic versus operational objectives in the fi nal scorecard. Many City Offi cials we spoke with recognize a need for a strategic scorecard metrics which track long term outcomes (for a description of outcome measures see Figure 5 below). However, such a scorecard would change little from month to month and therefore have little impact on the operations of sub-cabinet departments.

While a strategic scorecard would conveniently compile performance outcome information in one place, it would not likely present any new information. Most City of Boston departments have a good idea of where the City stands when it comes to long-term outcomes such as poverty rates, graduation rates, or adult mortality fi gures.

Similarly, the Boston about Results (BAR) program effectively captures operational measures for many departments. The counts and percentages that make up most BAR reports are extremely informative to the budget process and certainly assist department leaders in managing their organizations. However, this heavily operational approach is too fi ne-grained for monthly use at sub-cabinet meetings. As discussed above, the sub-cabinets’ responsibility is to develop cross-departmental strategy, not to manage the operations of individual departments.

In selecting metrics that are both monthly and aligned to cross-departmental objectives, we have sought to achieve a scorecard that balances both operational and strategic focus. The measures must speak to all departments and must also allow for short-term action. To some

Achieve Balance:by analyzing performance through multiple lenses

“No single measure can provide a clear performance target or focus attention on the critical areas of the business. Managers want a balanced presentation of both fi nancial and operational measures.”

Page 32: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

18

extent, we have emphasized output data over outcomes; however, all the metrics selected are highly informative. While the City of Boston may be responsible for the long-term outcomes for citizens, we believe that the sub-cabinet can best manage toward producing these outcomes through an attention to the short and medium-term outputs it delivers to citizens. Inputs, processes and outputs can be rapidly changed, analyzed, and innovated toward the goal of improving services. A Note about Potential PitfallsPerformance management efforts are not conducted without downside risks. While the specifi c risks vary from metric to metric, a few general issues warrant attention. Potential pitfalls of this effort include:

• Selecting the wrong metrics: A central tenet of management is that “what gets measured gets done.” In selecting the metrics for a scorecard, managers must ensure that what they have chosen to measure is indeed what is most important to get done. Priorities will be set around metrics. Thus, if low-priority items fi nd their way onto cross-departmental scorecards (which may happen if the metric is easy to measure) then these low-priority initiatives will rapidly take attention and resources away from the high-priority initiatives not on the scorecard. Therefore, scorecard metrics must be selected with care.

• Cheating on metrics: There are two ways to cheat on a scorecard metric. One is to lie about performance. The other is to change your performance in a way that results in a better score on the metric but does not actually improve the services that the metric measures. The latter is often called the “teaching to the test” problem. Teachers can improve test scores by “teaching to the test,” but this does not mean that students will be better equipped to understand real- world issues and become more capable adults. It simply means the students scored better on the test.6 To correct for this pitfall, managers must be attentive to how performance results are achieved and question non-productive or dishonest practices.

• Employee micromanagement concerns: When employees hear about performance management systems, many immediately think that the system will be use to police their work and punish them when they do not reach a certain performance level. While accountability is an important part of a sub-cabinet scorecard, the sub-cabinet leaders must ensure that their line managers and employees know that scorecard also has other uses. It is a tool to create cross-departmental collaboration, to support employees to make improvements that will make their jobs easier and more effective, and to identify and celebrate the successes of the sub-cabinet departments.

Watch for Pitfalls:beware the “teaching to the test” problem and other common performance management issues

Page 33: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

19

• Failure to connect with citizens: The citizens of Boston do not care about performance management. They care about improved city services. Therefore, the sub-cabinet members should know how to communicate the value of a scorecard to the public. It is not the presence of management scorecards that matter, but rather how these scorecards relate to improved services, experiences and outcomes for citizens.

• Focus on defi cits instead of assets: The job of city government is not just to prevent bad things from happening; it is also to help good things occur. Therefore, scorecard performance measures should be crafted whenever possible to focus on the assets created for citizens rather than the defi cits prevented. This will also impact the way the scorecard informs management decisions. To take an extreme example: if a metric supports the goal of “reducing traffi c fatalities,” then one way of reducing fatalities would be to outlaw driving. This would limit the defi cit (traffi c fatalities), but it wouldn’t serve the citizens’ best interests. Instead an asset-based goal of supporting “safe and effi cient highways” might align with a metric such as “the average number of highway miles traveled without accidents.” Efforts would be taken to drive this number up, and this would limit deaths while also improving the positive aspects of the highway experience.

Figure 5: The Chain of Performance

Inputs Activities Outputs OutcomesDefi nition Components,

information and materials needed to produce desired outputs

The process by which inputs are converted to outputs

The goods and services produced

The benefi ts received by stakeholders

Examples Number of Teachers

Flu Clinic Hours of Operation

Reading Curriculum

Method of Distribution

# of Students Receiving Instruction

Number of Inoculations

Students Reading at or above grade level

Reduction in Flu Infections

Advantages Easiest to Measure Infl uenced by Strategy and Policy

Often more meaningful than inputs and easier to capture than outcomes

Demonstrate results and focuses accountability or true aims

Weaknesses Limited information for related to performance of actual results

Usually measured by outputs, and can only be evaluated in comparison to outputs of other activities

Often fail to disclose whether or not stakeholders are better off

Diffi cult to measure and diffi cult to infl uence in short term

References: Niven and Behn

Page 34: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

20

0

50

100

150

200

250

300

350

Sept Oct Nov Dec Jan Feb

10

20

30

40

50

Jan Mar M t Nov DecFeb Apr M

10

100%89%

83%74% 7700%%

%

0%

20%

40%

60%

80%

100%

120%

Age < s 1-2 Ages 3-5 A 10- AAAges 114s 1A Ages 5-

18Ages 19-

20AAg s 6-9A<1 Ages< es

0%%0%%

g S0gg%gg Sep Octg0gg%gg t%MMMMMM

Ages 6 9 AAg s 6 9 A6-9 Ag1

Aes 6 A58s 15

14Age

8g

0

Page 35: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

21

Chapter 5:

Recommended Indicators & Analysis

A child’s school attendance – or lack thereof – can tell an important story about the resources and support networks available to the child and his or her family. To keep up school attendance, students must be healthy, be engaged by school, feel safe at school and in their community, possess a stable housing environment, and otherwise have the right factors in place that will enable them to learn. Furthermore, if a child does not attend school, the chances of him or her staying or track for graduation and future success are greatly diminished. As such, school attendance is an exceptionally powerful EHHS scorecard metric both in its ability to evaluate cross-departmental service delivery and its potential for predicting outcomes for the city’s youth in the future.

Our recommendation is to track the number of students below 80% attendance for the month (i.e., those students who missed more than one day per week). Through conversations with BPS administrators, it became clear that tracking the gross system-wide attendance would not prove useful, as system-wide attendance is consistently near 95%. However, this high overall attendance rate masks the needs of chronically absent students. It is these students who require cross-departmental interventions in order to get them back on track.

Numerous research studies point to the importance of the attendance as an early warning indicator for school performance, particularly the 80% attendance mark. A study of students in Philadelphia public schools

This section discusses in detail the 10 Education Health and Human Services (EHHS) scorecard metrics we recommend for the sub-cabinet based on the framework explained in earlier chapters. The section for each metric includes discussion of its alignment to the overall scorecard strategy, an analysis of the currently available data, some recommendations on setting goals and ensuring appropriate lines of responsibility and concludes with proposed next steps for future action.

Metric #1: School Attendance - Number of Students below 80% attendance for the Month

Page 36: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

22

demonstrated that students below 80% attendance in 8th grade had a 75% chance of dropping out of school.7 A similar study in Boston had even more powerful results. Using BPS data, researchers demonstrated that among sixth grade students, those who had less than 90% attendance were less than half as likely to graduate on time compared with their well-attending peers, and those who had less than 80% attendance had only a 5% chance of graduating on time.8

The negative outcomes associated with poor attendance do not stop with drop out risk. According to Boston Police Department (BPD) strategy to improve student attendance (START), truancy has been identifi ed by the BPD and Boston Public Schools (BPS) as a signifi cant indicator that a youth is at risk to fail in school and become involved in criminal activity.9

Alignment to Scorecard StrategyThe number of students below 80% monthly attendance exemplifi es the ideal metric for a cross-departmental monthly scorecard. Poor attendance is highly predictive of drop out and negative outcomes for students. It is also diagnostic of the sub-cabinet’s cross-departmental ability to provide the services necessary to keep students in school: healthcare interventions, safe learning environments, after school engagement, social services, housing, transportation, etc.

0

1000

2000

3000

4000

5000

6000

7000

Sept Oct Nov Dec Jan Feb

System-wide Number of Students below 80% Monthly Attendance

Figure 6 (above): Student attendance is diagnostic of the levels of stable housing, community safety and other items to the left. It predicts future test scores, drop out rates, behavior problems and the other outcomes to the right.

Figure 7 (above): The number of students below 80% attendance for the month has steadily risen since September 2009.

Page 37: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

23

Furthermore, the number of students below 80% attendance changes monthly and is reported in a reliable and readily available format. By comparing the current month’s attendance rates to the previous month’s, EHHS sub-cabinet members can immediately obtain feedback on the effi cacy of their efforts and interventions toward improving attendance. The availability of rapid feedback presents the potential to build cross-departmental momentum around attendance, bringing this issue into strategic focus and rallying resources to support efforts to improve outcomes for chronically absent students.

Analysis of Existing DataThrough the Boston Public Schools Offi ce of Research, Assessment and Evaluation, we were able to obtain school-year-to-date information on the number of students below the 80% attendance mark by month. This information was further broken down by school, and also included the number of students between 80-89% attendance, between 70-79% attendance, and below 70% attendance.

Figure 7 shows the trend for the 2009-2010 school year in the number of students system-wide below 80% attendance by month. The data shows that the number of students with poor attendance has been rising every month since September.

Figure 8 (above left): The number of students below 80% attendance by school

Figure 9 (above right): The change in number of students below 80% attendance between January 2010 and February 2010 by school. Red circles represent schools that increase absence rates while green circles are those who managed to decrease the absence rate.

Page 38: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

24

Figure 8 maps the number of students below 80% attendance by school for the Month of February 2010 (the most recent data available). This map illustrates that there is a large number of schools across the district that have more than 200 students below the 80% mark. However, this map does not take into account the fact that some schools have higher student enrollment than others. Using total enrollment fi gures by school it would be possible to normalize this data so that the map would show the percentage of students by school who are below 80% attendance, which is more actionable. Full data by school is available in Appendix A.

Figure 9 shows the change in number of students below 80% attendance between January 2010 and February 2010. It is immediately obvious from the map that most schools had signifi cantly more students below 80% attendance in February as compared with January. This negative trend is only countered by the Galvin Middle School in South Boston which had 12 less students below 80% attendance in February.

Figures 10, 11 and 12 show the number of students by grade level below 80% monthly attendance. In the elementary grades attendance tends to improve with grade level. However, after 5th grade, the number of students with poor attendance rises precipitously, peaking in 9th grade, but remaining high through the high school years. This information can be useful to the EHHS sub-cabinet in deciding which age groups to target with attendance intervention programs.

Goals and ResponsibilitiesSchool attendance is not just a BPS issue. In order to truly impact the number of chronically absents students, the sub-cabinet must engage all EHHS departments to take action to infl uence the causes of attendance that relate to their mandates. For this reason we are proposing a second metric (discussed in the next section) which will delimit absences by type so that more targeted interventions can be formulated.

Tracking and reporting the data on attendance is the responsibility of the Boston Public Schools Offi ce of Research, Assessment and Evaluation.

Next Steps and Future EvolutionsThe analysis presented above could be improved through the use of data that is normalized by enrollment at each school. While total absence fi gures are important to understand the scale of the attendance problem, normalizing the number of absences by the school size will allow for more accurate comparisons between schools.

The benefi ts of comparative data cannot be underestimated. Publicly illustrating with obvious maps which schools have improved attendance rates and which schools have deteriorated can serve as a serious motivating force toward improvement. As Bob Behn notes, no individual, team or

Figure 10 (above): Among elementary schools, the number of students below 80% monthly attendance decreases with age. In other words, attendance improves with age.

Figure 11 (above): Among middle school students, the number of students below 80% attendance increases with age.

Figure 12 (above): Ninth grade has by far the highest number of students below 80% monthly attendance but absence rates remain high through high school.

0

200

400

600

800

1000

1200

1400

Sept Oct Nov Dec Jan Feb

Nu

mb

er

of

Stu

de

nts

be

low

80

% A

tte

nd

an

ce

Number of High School Students below 80% Monthly Attendance

9th Grade

10th Grade

11th Grade

12th Grade

0

200

400

600

800

1000

1200

1400

Sept Oct Nov Dec Jan Feb

Nu

mb

er

of

Stu

de

nts

be

low

80

% A

tte

nd

an

ce

Number of Middle School Students below 80% Monthly Attendance

6th Grade

7th Grade

8th Grade

0

200

400

600

800

1000

1200

1400

Sept Oct Nov Dec Jan Feb

Nu

mb

er

of

Stu

de

nts

be

low

80

% A

tte

nd

an

ce

Number of Elementary Students below 80% Monthly Attendance

1st Grade

2nd Grade

3rd Grade

4th Grade

5th Grade

Page 39: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

25

group likes to be on the losing side of an analysis, but by providing this analysis it creates a form of “friendly competition” whereby all schools can work to ensure in future months they fall into the “improving” category. 10

Given this, comparative mapping should be used to the greatest extent possible.

Future evolutions of the scorecard might also track cohorts of students falling below 80% attendance, and over time seek to determine if the interventions being delivered to them help them improve their attendance. We have included a suggested metric (Metric #3) that does just this.

Chronic school absence can be caused by a number of factors. Some students may regularly miss school because they have a chronically ill parent to care for or they must watch they pre-school aged siblings when their parent is ill. Other students miss school because they have an unstable housing situation which prevents them from establishing a school attendance routine. Still others may simply lack the money to pay MBTA fares to get to school. Meanwhile many students gradually become disengaged with school due to poor academic performance or behavior problems in the classroom which lead them to stop attending school.

Children who are frequently missing school and who are also wresting with these enormous challenges are at high risk for not reaching the academic, social and economic outcomes of their peers. While Metric #1 tracks the number of chronically absent students, this metric provides the sub-cabinet with information on why students fail to attend school. This metric will highlight the potential interventions the EHHS departments may be able to make to get the cohort of kids missing the most school back into the classroom. Furthermore, it allows the EHHS sub-cabinet to hone in on the issues facing some of the most underserved families in the city.

Alignment to Scorecard StrategyThe absence classifi cation metric will help the EHHS sub-cabinet target its interventions to where they are needed most. This is a highly cross-departmental metric since many EHHS departments can infl uence factors relating to health, transportation, and employment. Knowing the categories of absence will also assist the sub-cabinet in assigning responsibilities to appropriate departments.

This metric also holds the same diagnostic and predictive relevance as the number of chronically absent students (reported in Metric #1). By tracking the reasons for chronic absence the EHHS sub-cabinet can evaluate past intervention efforts as well as predict what services may be needed in the future (see Figure 6).

Analysis of Existing DataIn order to truly understand why kids are attending fewer than 80% of school days, the sub-cabinet must deploy a more comprehensive system to track the causes of chronic absence. Currently, there is no single authoritative absence classifi cation database. Information on the causes of chronic absence is separately investigated based on a variety of inputs. Absences are tracked at the BPS school level, by the BPS Offi ce of Alternative Education (OAE), and by the Boston Public Health Commission.

The Offi ce of Alternative Education (OAE) frequently handles casework on truancy. One of the department’s fi ve Supervisors of Attendance works with students and their families to try to resolve school attendance issues through phone calls, home visits, and even court action if necessary. Through these interventions, the case worker is often able to uncover certain factors leading to school absence.

Metric #2: School Attendance – Number of Monthly Absences by Absence Classifi cation

Page 40: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

26

Through their supervisors of attendance, OAE is perhaps in the best position to gather the necessary information on the cohort most frequently missing school. If provided with a systematic way of tracking and classifying absences, the dedicated team at OAE would be able to provide the EHHS sub-cabinet with actionable information on the causes of absences.

However, not all absence data makes its way to OAE. Absences are fi rst reported at the school level, and therefore efforts to improve data collection must target schools as well. Anecdotal evidence suggests that each school tracks absence in a different way – what may be coded as an excused absence at once school may be unexcused at another. Therefore, in order to accurately track absences system wide, BPS must work to establish a standardized classifi cation system.

Important information on absences also fl ows from the Boston Public Health Commission. Through the Connecting Families to Schools program, BPHC provides home visits by social workers and advocates to families whose elementary school children are experiencing excessive school absences. Home visitors conduct a needs assessment, develop a referral plan to help families get appropriate services, and incorporate parent training curricula into their visit activities to help build and strengthen family resiliency.11 These social workers are in an excellent position to track and codify the reasons behind excessive absence.

Should comprehensive absence classifi cation data become available, it will have powerful implications for shaping EHHS department policy and interventions around absences. Trending analysis will allow the sub-cabinet to predict patterns of absences and intervene before they occur. Tracking absences by category will also allow EHHS departments to develop needs assessments and deploy resources to where they are likely to have the greatest impact. This analysis may also refl ect geographic trends, identifying neighborhood-based centers of need around certain absence causing issues.

Goals and ResponsibilitiesOnce data on the number of absences by category are available from a central source, the EHHS sub-cabinet should assess which absence classifi cations have the most glaring needs and can be most directly impacted by EHHS departments. After several months of discussion on this metric, the sub-cabinet should be able to set target numbers of interventions by classifi cation and begin to measure their progress. Because there are a number of different classifi cations for chronic absence, it is likely that all of the departments represented on the sub-cabinet will be responsible for delivering services to meet the targets.

Figure 13: Absence Categories

Category Reason

Health Health – chronic (asthma, diabetes, sickle cell, mental health, etc.)

Health – Short term illness or injury

Health – Student pregnancy

Health – Family Member

Behavioral / social

Trouble waking up in time / chronic tardiness

Friends don’t go / didn’t feel like it

Substance abuse

Discipline Assigned to Counseling or Intervention Center

Suspension

Court appearance

Detained or incarcerated

Distress Death / Morning / Family Crisis

Fear for safety, bullying, crime

Domestic violence / abuse / neglect

Academic Didn’t do homework / not ready for test

Disengagement due to Failing Grades

Don’t like the school or the teacher / teacher doesn’t like me / school “fi t”

Housing Family homelessness / unstable housing

Child not in supervised living situation

Responsi-bility

Home care – sibling, parent, elder

Own child

Work

Transporta-tion

Failure of school-based transportation (bus did not stop)

Non-school-based transportation problem (parent could not take student to school / MBTA breakdown)

Student cannot afford / does not know how to get to school

Other Family vacation / travel

Religious

Other

Page 41: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

27

Next Steps and Future EvolutionsIn order to be able to track absences by category in the future, the EHHS sub-cabinet must take on two concurrent tasks: (1) developing a standardized set of absence classifi cation categories, and (2) designating and/or deploying a system for collecting this data.

Figure 13 presents a possible set of absence classifi cation categories that have been developed in consultation with OAE and the Mayor’s Offi ce. The left hand column suggests broad categories of absences, while the right hand column includes more detail about the cause of the absence. Both columns should be tracked when possible, but it often may be the case that only information on the broad category will be available. This is just a starting point for absence classifi cation – more work needs to be done to investigate the relevance of these categories before the classifi cation system is deployed.

Establishing the best system for tracking absence classifi cation should be an effort led by the BPS Chief Information Offi cer in consultation with OAE and the BPHC. This effort will involve moving some paper-based tracking systems to electronic records as well as coordinating system access for data collection.

Making a meaningful impact on school truancy rates not only requires EHHS departments to understand the root causes of chronic school absence, but it is also necessary for city departments to aggressively track whether and why intervention efforts are working. This measure tracks the monthly percentage of chronically absent youth receiving intervention and case management services that improve their school attendance. As mentioned previously, once classifi cations are tracked, the EHHS sub-cabinet can begin to look at the progress they have made over the prior month. The group can then ask, “If the number of chronic absence cases have decreased – particularly by classifi cation – why? How can these successes be replicated?” Conversely, if few changes have occurred, the discussion may be centered on how the EHHS sub-cabinet can change their policies and direct new efforts toward absenteeism interventions.

There are two ways this information can be tracked. One way is to collect the number of court cases fi led against truant students – this is a defi cit-based approach. The other is tracking information on whether an intervention is being staged – and possibly what kind of intervention – through a case-management database. EHHS can begin to focus on this measure once the database for chronic absences is up and running. Since the database will be managing information on the universe of truant students, the Supervisors of Attendance can then input information in the student’s case on what interventions the student is receiving to meet his or her specifi c needs.

Alignment to Scorecard StrategyThis metric is the mirror image of the chronic absence classifi cation metric because many of the EHHS sub-cabinet interventions that might lead to higher school attendance will be based on information gather on the monthly causes of chronic absence. As such, this measure is a tool for matching interventions to needs, and progressively improving the effi cacy of various interventions. This metric functions as an effectiveness measure regarding the EHHS sub-cabinet’s efforts to promote stronger educational outcomes and to give families access to resources they need to be healthy and economically secure.

Metric #3: School Attendance – Percentage of Successful Absence Interventions

Page 42: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

28

Analysis of Existing DataAt present, data on the percent of successful interventions in chronic absence is diffi cult to collect. However, the EHHS sub-cabinet can measure its intervention performance by looking at number of court cases fi led against truant youth as a percentage of the total number of truant youth. This court case measure is a proxy for unsuccessful interventions, since court cases are fi led as a last resort when a student or his family have neglected to remediate the students absenteeism. This metric only measures one aspect of the absence interventions, so it is far from comprehensive.

The sub-cabinet may be able to analyze this metric more accurately once the chronic absence classifi cation metric is established. This is so because the monthly intervention percentage is determined against the number of chronic absences. Authorized city offi cials interacting with these students – such as Supervisors of Attendance and BPHC workers – can use case management software to update and track progress for this cohort. The data from these cases can then be refl ected in monthly measures.

Goals and ResponsibilitiesEHHS’s fi rst goal should be to systematically collect the numbers on court cases fi led, and then to establish this data management system. In the initial stages of this metric, the EHHS sub-cabinet’s objective should be to stage as many interventions as possible. This means that the sub-cabinet will be looking to drive the number of court fi lings down. Once BPS has developed a case management system to track chronic absenteeism, the EHHS sub-cabinet’s will be looking to drive the number of interventions up. When enough data has accumulated in BPS’s truancy case management system and the sub-cabinet can understand the trends in chronic absence, then the group may establish more specifi c goals.

Next Steps and Future EvolutionsEHHS’s fi rst step should be to acquire data on court case fi lings on a monthly basis until BPS can create its own case management system for chronically absent students. The sub-cabinet can work with the OAE to gather this data from the district court system in Boston. Concurrently, BPS should have OAE, BPHC (through Connecting Families to Schools), and dedicated staff from individual schools working with BPS’s Management and Information Services (MIS) and Boston’s MIS department to craft a useful case management program.

The Boston Centers for Youth and Families (BCYF) metric tells us how many Boston residents are accessing the network of city-sponsored community centers on a monthly basis. This measure is a proxy for after-school participation among Boston’s school-aged population, and it may also be able to explain the range of generations – young children, school-aged children, and adults – who are using these facilities. BCYF center attendance is an important metric because education research indicates that after-school participation is a predictor of outcomes such as higher rates of school attendance and improved academic performance.12 Extracurricular activity is also strongly associated with lower early school dropout rates, particularly among high-risk students. 13

BCYF Collects this information on a daily basis, as people sign-in to use one of the 46 community centers across the city. The sign-in sheets can include information on the raw numbers of people entering the centers, the identities of folks using the centers, and the programs they are attending. However, this data is not currently analyzed beyond raw center attendance because of inconsistent sign-in policy and identity protection.

Metric #4: After School Program Attendance - Monthly BCYF Attendance by Facility

Page 43: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

29

Alignment to Scorecard StrategyEHHS sub-cabinet departments strive to provide a continuum of services to Boston’s youth and their families. An after-school program participation metric aligns with one of the main principles of Mayor Menino’s Youth and Family Opportunity Agenda, which is to support achievement both in and out of the classroom. The metric demonstrates how well EHHS departments have performed over the past month at providing out-of-school time opportunities to Boston’s youth. Attending and participating in after school programs is also predictive of school attendance, dropout risk, pro-social skill development, and exercise related health.

While BCYF is responsible for obtaining and reporting attendance data, the Boston Public Schools, Boston Public Libraries and Boston After School and Beyond all share a role in increasing BCYF attendance and narrowing the citywide achievement gap. The school department can partner with BCYF to engage students in after-school programming sponsored by the community centers and also arrange transportation. Other after-school programming providers can work with BPS and BCYF on providing relevant programming and conducting outreach efforts in communities that need the most engagement. Similarly, at-risk youth involved with Boston Public Health Commission programs may be referred to BCYF programs because of their pro-social impacts and support of healthy living.

Analysis of DataThe current BCYF center attendance data suggests the degree to which the public has interacted with the network of community centers. It also indicates which neighborhood facilities receive the most traffi c. However, this information does not highlight the quality or the type of interaction that individuals are having with programs.

0

50000

100000

150000

200000

250000

300000

350000

Jul 09 Aug 09 Sep 09 Oct 09 Nov 09 Dec 09 Jan 10

BCYF Monthly Visits Jul 09 to Jan 10

Figure 14 (above): BCYF center attendance is diagnostic of outreach and collaboration between partners. It predicts future attendance rates, skill development and health outcomes.

Figure 15: BCYF center attendance declined at the end of the summer (corresponding with pool closure) but has remained stable since October.

Page 44: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

30

Since July, 2009 a monthly average of 213,000 Bostonians have passed through the doors of the community centers. This means the system serves an average of 7,000 individuals daily. The data are collected by each community center; therefore, geographic analysis shows which community centers are being used more frequently than others. Figure 15 shows that attendance has remained relatively stable throughout the winter months.

Figure 16 provides a map of BCYF attendance in by facility in January 2010 (the most recent data available). This information should be compared against the data presented in Figure 17, which shows the change in attendance between December and January. Centers with green circles saw greater attendance in January compared with December, while red circles indicate BCYF centers that saw a decline in attendance. The size of the circle indicates the magnitude of the growth or decline in attendance. The EHHS sub-cabinet can use this information to decide where deploy resources for sub-cabinet initiatives as well as where to deploy greater advertising for programs.

Goals and ResponsibilitiesIn order to accurately align community center attendance with predicted after-school related outcomes, the EHHS sub-cabinet should aim to uncover the rate of program attendance for individual children. Once BCYF can

Figure 16 (below, left): January 2010 attendance by facility. Large circles denote larger attendance fi gures

Figure 17 (below, right): December 2009 to January 2010 change in BCYF center attendance. Green circles denote attendance growth

Page 45: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

31

track program attendance rates, the sub-cabinet should track the number of children who have a greater than 79% program attendance rate. According to research, a 79% attendance rate describes the level of after-school program participation which is associated a greatly improved probability of positive youth outcomes.14

In the meantime, BCYF and EHHS should be striving to increase daily contact with Boston’s youth. Perhaps even more importantly, the sub-cabinet should pay close attention to which neighborhoods have higher contact rates with their BCYF centers and aim to narrow gaps across neighborhoods (taking scale and community center amenities into consideration).

Next Steps and Future EvolutionThere are a number of challenges associated with this data that prevent it from truly describing the link between BCYF attendance and EHHS sub-cabinet efforts. At present, BCYF center sign-in policy is not standardized across facilities; therefore, we don’t know exactly who these numbers represent. Through interviews we have learned that attendance rates include children attending after school programs, adults attending evening programs, parents picking their children up from the center, or staff members signing in. It is currently impossible to disaggregate this information. In the future, a more meaningful BCYF measure would describe the rate of BCYF center attendance by age group.

Boston’s proposed KidTrax data collection program has the potential to change BCYF attendance tracking data. Kid Trax will enhance the overall accuracy in measuring youth programming attendance by streamlining the collection process and documenting individual program attendance. This data may become even more useful if it is connected with Boston Public Library and Boston Public School attendance.

A good fi rst start is for BCYF to adopt a more standardized sign-in policy across all community centers. This will at least provide for enhance consistency in data quality in the short term, while in the long term the Kid Trax program will provide for greatly expanded analytical capacity.

The libraries metric tracks the raw number of Bostonians who enter the libraries on a monthly basis and participate in library programing. There is a strong overlap between both the Library attendance metric and the BCYF attendance metric in the sense that both types of facilities foster after-school time programming, which is predictive of attendance, academic achievement, and lower drop-out rates. 15 Studies of library programming indicate that immigrants and youth from low-income communities benefi t from English language and technology classes, respectively.16 Additionally, library participants benefi t from access to health and employment information.17

BPL attendance data are collected by an automated gate count at every library branch. Additionally, the BPL collects program attendance data per branch through program sign-in sheets and which are reported monthly data to the central branch. Taken together, these data paint a picture of relative attendance across Boston neighborhoods and a relative assessment of who is accessing library programming.

Metric #5: Library Program Attendance – Monthly BPL Attendance by Facility

Page 46: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

32

Alignment to Scorecard StrategyThe BPL attendance metric is an indicator of how well the EHHS sub-cabinet is supporting out-of-classroom achievement and establishing community-based learning environments.18 Additionally, since the indicator also applies to informational programs for employment and health, it provides information on how well-informed communities are informed on creating their own healthy environment.

In the context of the EHHS sub-cabinet, this BPL information can be used to strategically direct enrichment and information resources to where they are most needed. Geographic-based data can relate how participation and attendance differ across neighborhoods. Additionally, program-based data can demonstrate which age cohorts are taking most advantage of library resources, indicating whether EHHS should plan programming interventions based on age groups and/or types of programs.

Analysis of Existing DataThe Boston Public Library currently collects gate counts and program attendance data branch by branch. However, due to the way the data are reported, it is currently not possible to accurately analyze monthly attendance by branch. This problem can be rapidly corrected through improved data reporting, which can be informed by the ways in which other EHHS departments report data.

Figure 19 shows the trend in the number of BPL programs offered system wide by month, while Figure 20 shows the number of attendees to

Figure 18: Library attendance is diagnostic of programming relevance and outreach, it is also predictive of literacy rates and future youth outcomes.

Page 47: Managing Cross-Departmental Collaboration: A Performance ... · Managing Cross-Departmental Collaboration: A Performance Scorecard ... Metric #1: School Attendance ... Managing Cross-Departmental

33

programs by month. It is interesting to note from this data that the number of children’s programs offered and the number of children attending are inversely proportional – more programs are offered in the fall but more children attend in the summer. The attendance for young adult and adult programs seems to move in sync with the number of programs offered.

Figure 20 is a map showing total library visits by branch in 2009. This map provides some information on which libraries are more regularly utilized then others. However, some libraries, like the Copley Square Main Library, receive vastly greater traffi c due to their larger size and are therefore diffi cult to compare with smaller branches.

Once gate counts are accessible on a month to month basis it will be useful to map this information as well as the comparative month to month information to see which libraries are seeing increasing attendance and which libraries have declining attendance.

Goals and ResponsibilitiesThe EHHS sub-cabinet should strive to narrow the BPL attendance gaps in chronically underserved communities and in communities with low program participation rates. The group should also identify trends in age group program attendance to see if programming interventions are necessary based on demographics.

0

100

200

300

400

500

600

July Aug Sept Oct Nov Dec Jan

Prog

ram

s

Number of BPL Programs

Adult Programs Young Adult Programs Children Programs

0

5000

10000

15000

20000

July Aug Sept Oct Nov Dec Jan

Prog

ram

Att

ende

es

Number of Persons Attending BPL Programs

Adult Attended Young Adult Attended Children Attended

Figure 19 (upper left): Trend in the number of BPL programs offered, July 2009 to January 2010.

Figure 20 (lower left): Number of attendees to BPL programs by type of program.

Figure 21 (upper right): Total visits to BPL branches in FY09 by branch. Larger circles indicate more attendance.