KPI Management and Policy Consultants Conference on Performance Evaluation of the Judiciary Ankara,...

1KPI Management and Policy Consultants

Conference on Performance Evaluation of the JudiciaryAnkara, 25 March 2011

1

CoE Conference on Performance Evaluation of the Judiciary

Court Statistics, Judge Workload Analysis, Quality and

Performance Management

Dr Axel G KoetzAnkara 25 March 2011

This Document is complete only together with the oral presentation;use of isolated pages might lead to misunderstandings.

Questions: Dr Axel G. Koetz, Managing Partner, KPI Management and Policy ConsultantsUnicenter 2920, D-50939 Cologne, axel.koetz @ koetz-ag.com, Tel/Fax +49 (0)221-9411801 / 05

Dr Axel G Koetz, Managing Partner, Koetz Partner International, Unicenter 2920, D-50939 Cologne Germany, e-mail axel.koetz @ koetz-ag.comCourt Statistics, Judge Workload Analysis, Quality and Performance Management – Ankara (CoE) 25 March 2011


KPI Management and Policy Consultants


This presentation refers in part to results of work generated by the author during

the Projects

Dr Axel G Koetz, KPI International Management and Policy Consultants, Unicenter 2920, D-50539 Cologne Germany, [email protected] Management and

Policy Consultants

ACCOUNTABILITY AND EFFECTIVENESS OF UKRAINIAN JUDICIARY FUNCIONING: Civil Service Component EuropeAid/125611/C/SER/UA

STRUKTURANALYSE DER RECHTSPFLEGEOrganisation der Amtsgerichte

Organisation der Kollegial- und InstanzgerichteOrganisation der Staatsanwaltschaften

Im Auftrag des Bundesministeriums der Justiz, Bonn(published in German by Bundesanzeiger Printing House)

and other related studies on behalf of German State Governments




Use of Statisticsin Court Management

Conference MaterialPart 1 of 4




Today Problems

• Data are collected for many use(r)s – including academic exercises – but not for management

• Nobody cares for data quality as those who create do not profit from results

• Case data, HR data and financial data are collected by different departments and for different users and never integrated for management use

• An immense quantity of data is collected, stored and forgotten

• Collection of date itself creates an inappropriate resource consumption




Necessary Integration of Statistics

Case Statistics

HR andResourcesStatistics

FinancialStatistics

ExternalQuality

Statistics

CourtManagement

Data Base




What means Court Management ?

Top Down View:Optimization of the overall Judicial System from the

viewpoint of Effectiveness and Efficiency

Bottom-Up View:Ensure a proper functioning of the court according to central goals and making best use of court resources

Effectiveness, Quality and Efficiency of the Judiciary

Inst

rum

ents

Legal p

oss

ibili

ties

to „

man

ag

e“




Orient Statistics to Management Needs 1:What the Head of Court Might Like to Know

• Number of cases (incoming / completed / unfinished)• Case workload (per relevant case type and overall)• Time needed for case types• Backlogs• Percentage of cases sent back from higher court due to successful

appeals• Satisfaction of Court users with speed, friendlyness, accessibility• Differences in performance between judges – in quantity, speed

and correctness of output• De facto available staff off all types• Position of the courts performance compared with others• Changes of numbers compared with the last year/sMay be more ?

May be other ?May be none at all ?




Orient Statistics to Management Needs 2:

What the Central Level Might Like to Know

• Number of cases nationwide, regional, per court• Case workload: Overall indicator for key case types• Time needed for case types overall / per court• Backlogs per case type / per court• Percentage of cases sent back from higher court due to

successful appeals – overall and per court• Satisfaction of Court users with speed, friendlyness,

accessibility, overall and per court• Differences in performance between courts and judges – in

quantity, speed and correctness of output• De facto available staff off all types versus plan (overall /

per court)• Comparative data describing court performance• Changes of numbers compared with the last year/s




Are Big Courts More Productive Than Small Courts ?

0

50

100

150

200

250

300

350

400

450

0 5 10 15 20 25 30 350

50

100

150

200

250

300

350

400

450

0 5 10 15 20 25 30 35

666 District Courts, Unweighthed Cases, Planned Judges




Do Judges in Small Courts Need More Support Staff Than in Big Courts ?

FT

PE

Sup

port

per

FT

PE

Judg

e

FTPE Judge0

2

4

6

8

10

12

14

16

18

20

0,00 5,00 10,00 15,00 20,00 25,00 30,00

KH

OD

LV

KO

CY

DZ

Potenziell (DZ)

0

2

4

6

8

10

12

14

16

18

20

0,00 5,00 10,00 15,00 20,00 25,00 30,00




How much time does a Judge need for for an average Case ?

HoursjudgeTime per case

Jud

ge

FT

PE

pe

r C

ou

rt

9

0

5

10

15

20

25

30

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5

OD

KH

LV

KO

CY

DZ

9

0

5

10

15

20

25

30

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5




Why Does One Court Solve 50% of the Cases in

3,5 Months Whilst Another Takes 6 Months ?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9

10

11

12

13

14

15

16

17

18 24

60

12

0

mo

re

1-CUM

2-CUM




Why Differ Backlogs by the Factor 10 Between Courts of Different Regions ?




... And External Quality Evaluation ?

• Problem: External data need extra processing, this means work

• Chance: See the unbiased view of the „users“ instead of what the system produces internally

• Customer cards might provide multiple choice answers on– Accessibility of the court– Evaluation of court staff behaviour– Evaluation of felt work quality– Evaluation of processes, timing etc.

• Interesting is the time series analysis and the internal / external comparison




Questions Lead to Other Questions...

• Should we change the court network and eliminate courts with less than xx Judges ?

• In how far can we exchange court support staff to technology (and uphold small courts)

• How can we deal with non performing Judges who solve less cases / are systematically slower than the average

• Do we have under-resourced courts and are lacks in resources or regional „styles“ responsible for backlogs ?




... Other Questions Lead to Decisions

• Change the „court network“• Increase training for nonperforming judges and

judges with a high proportion of successful appeals

• Introduce better workflow software and optimize regulations to save support staff capacity

• Redistribute resources according to real workloads

• Have discussions with Court managers who fail to bring their numbers in order.




Necessary Elements for a System of Collecting and Analyzing Data





From Data Cemeteries to Useful MIS

Relevant

Automatic

Motivating

„Drilling Thru“

Start from Existing Systems

Leads to the right decisions (e.g.: necessity of proper case weighting)

Data generated during the normalwork; no additional data collection

Short and easy to understand by thedecision makers and motivate them

Trace nationwide information throughall levels down do the individual

Base as far as possible on existing data and data collection structures




Relevance(Aspect 1 of 5)

• We do not need to count what we can not influence

• For all other things we need the data• In any case, a value analysis of all data collection

is necessary

• For example: Proper case weighting is extremely relevant for all management decisions and a „must have“ for the system.




Automatic Collection(Aspect 2 of 5)

• As far as possible, all data should be generated automatically within existing workflows– Case data– Human resources dData– Other resources procurement / register data– Financial data

• For all data we need integrated collection and storage systems

• For all data we need clear definitions and clearly defined interfaces

• Existing workflow software has to be modified or exchanged to software which is able to do it.




Integration of Workflow Software and Statistical Software becomes a Key

ElementWorkflow System collects inter alia

- Case Type- Case Generation Date- Judge Name- Numerous case properties

(like n of hearings, n of witnesses, lawyers use, experts use etc.)

- Last hearing date- Verdict- Appeal

Automatic transfer to the statistical system at any time

Automatic forwarding to a national Court Data Base

Other Workflows in the Court

- HR- Equipment / Maintenance- Finance

Regular reports to central / regional / court level / public

Reports on demand to all levels, according to needs

DB availability for own research to the academic world and the public




Motivation(Aspect 3 of 5)

• Data have to be understandable to decision makers

• Information has to be publicly available

• No overcomplicated indicators• Decision makers have to have the power to act




Drilling Thru(Aspect 4 of 5)

• All levels have to have the technical opportunity to compare and track the reasons of problems as well as origins of good practice

• Top-Down and cross-cutting analyses have to be possible on every level

• „Drilling thru“ capacities: Analyze individual performance on all levels ...

• This means that ALL individual case data have to be available, retrievable, connectable and analyzable

• Never „aggregate“ data and give up potential information – nothing is as cheap as data storage




Pyramid Model to Identify the Souces of Problems...

National

Regional level

Court Level

Judge level

Which informations comefrom the national Average ?

Where come Differencesin regional performance from ?

Which courts have problems,Which are fine ?

Are there problems of Judges ?




„Drilling Thru“ and Extended Analytical Capacities Needed

JUDGES

COURTS

REGIONS

„Which judges are the most (un)productive nationwide ?“„Is there a systematic performance difference between City and rural area courts ?“„Are big courts more / less productive than small courts“ ?




Start from Existing Systems(Aspect 5 of 5)

• State of the Art Management Information Systems (MIS) are challenging, costly and the development might take years

• Instead of waiting for funds for optimal solutions, in many cases small changes in existing Software might bridge the time.

• Sometimes the work with pilot systems on court and region level can give important insight before the big project is started




Analysis of Judges Workload





Caseload versus Workload

Caseload:

Number of casesto be completed

- By a Judge- By a Court- By the

judges/courts in a region

- Nationwide

Workload:

The work capacity neededto complete

- a case- all cases on the judges

table- all cases in the court- all cases in the courts of

a region- all cases nationwide




The „Caseload“ concept leads to mismanagement

Caseload computes the numbers of cases irrespective of complexity.

Adding all cases and basing policies on this leads to severe problems as we are „adding pumpkins and cherries“

Many „order cases“Many „administrative offenses“

Lead to

High completion numbers„productive“ judges

Many complex criminal cases

Lead to

Low completion numbersUnproductive judges




Differences Unweighted/Weighted Cases Demonstrates Importance of Complexity

Analysis

0

0,5

1

1,5

2

2,5

3

3,5

4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

NW

W

Comparison of deviation from average (1) in caseload and workload per Judgein 33 Courts of a region, weighted (w) and not weighted (cases)




Determinants of Workload

CASE WORKLOAD PER CASETYPECASE WORKLOAD PER CASETYPE

Court Work Structures Workflow and IT

Case Complexity

QuantitativeAspects

QualitativeAspects




Case Complexity Determinators

Quality:Case TypeProcedural LawExisting Standards

Quality:Case TypeProcedural LawExisting Standards

Size:N of involved parties

/ defendantsN of witnessesN of needed

hearingsQuantity of

Documents

Size:N of involved parties

/ defendantsN of witnessesN of needed

hearingsQuantity of

Documents

Simplified:

Case Type plusSize Indicator

Simplified:

Case Type plusSize Indicator




Case Type Modeling

ModelAlternatives

Minimalistic Model/sConcentrates on a small number of consolidated case types

100% Model/sTries to identify (almost) all potential cases, Following the §§ of the law(s)

Key Indicator Model/sBased on a substantial number of relevant case types




Think in A-B-C Categories

Instead of killing people with megabytes of Data, follow the ABC model

A = Vital informationB = Important informationC = Unimportant information

Make sure that „A“ level information is not buried under „C“ level information

B CA

80%

90%

100%

20% 40% 100%

Information

Exp

lana

tion




Sample Key Indicator Model: PEBB§Y

• Identification of key case types, inter alia based on an ABC Analysis

• Identification of case complexity and related workload

• Detailed analysis of „A“ case types and selected others

• Correction factors and other instruments to cover the non-key case types (C, partly B)

• Please note: The following sheets show an extremely simplified picture of the methodology and the results ! PEBB§Y in reality is much more complicated when you come to the details.




PEBB§Y Goals

• Key Goal: Identify the need for Judges / prosecutors posts on state level and define the budget accordingly

• Second Goal: Ensure a just distribution of posts across the courts

• Third Goal: Create transparency and acceptance amongst stakeholders

• No Goal: Establish a legal right of judges not to work more than given by the set indicators




-

„Courts have to be equipped adequately with personnel and other resources;

details are specified by law“ (Constitution, Hamburg, Art 62 – similar in other constitutions)




Case type Selection (P1)

• 48 case types in local courts• 20 case types in lower appellate courts• 22 case types in higher appellate courts

• Other case types and administrative work taken into consideration via correction factors

• Also „Training“, „Administrative Functions“ and „other Tasks“ included, also numerous registers (German speciality) at local courts




Sample Judge Workload per Case Types

- for general civil cases 150 minutes- for civil claims from car accidents 170 minutes- for divorce cases 200 minutes- for small criminal cases 170 minutes- for major criminal cases 510 minutes- for punishment orders (comparable

to administrative offenses) 22 minutes- for economic and environmental crime 970 minutes

- Appeals to Lower Appellate Court 430/910 minutes- Cassation Cases, higher Appellate Court 660 minutes




Computing the Need for Judge Capacity

(Simplified)

• Judges work capacity per year 102.240 minutes

• Sample „car accident case“ 170 minutes• Cases per Judge per year 601 cases

N of cases x case weights

Judge work capacitycorrection factors

N of postsfor Judges

+/- =




Development of the 1st PEBB§Y 1 Model

2001 - 2002

• About 40 courts and Prosecutors Offices• 7 German States• About 1.900 Judges and Prosecutors• About 900.000 case cards analyzed• External project executed by a consulting / accounting firm

• In addition, a PEBB§Y 2 model was developed to get data for the non-judicial staff.




PEBB§Y StepsA Process of a Decade (by now)

• 2001: PEBB§Y 1 (General Courts –Judges, Prosecutors)

• 2001/2 PEBB§Y 2 (General Courts - Secretaries + Support Staff)

• 2005 PEBB§Y Fach (Judges in Specialized Courts for Labor, Tax, Social, Administrative

Cases)• 2008 PEBB$Y Update Analyses: 1,2,Fach

00 01 02 03 04 05 06 07 08 09

P 2

P 1

P Fach

P Update




Need for Participation...... and permanent updating

• States, Courts, Judges, Prosecutors intensively involved

• Numerous workshops, meetings and so on• Intensive cooperation and involvement of the

judges associations• Consideration of specialities in the different states• Check for realism and determination of the final

results in workgroups• Update of results according to chages in laws,

jurisdiction, procedures, technology after some years




Don‘t forget the civil service Structures

• PEBB§Y 1 was accompanied by PEBB§Y 2• PEBB§Y 2 covers the workforce needs in the field

of civil service– Secretaries– Typists / Note takers (if still existing)– Court guards– Other professions in the court

– Capacity needs for some professions still determined outside the PEBB§Y system




From Cases to Staff to BudgetFrom Budget to Staff

• Workload from weighted cases defines Staff need– Per single court– Per region

• Accumulated staff needs determine overall number of judges

• Overall number of Judges determine Financial Budget for Judges

• New judges have to be hired accordingly (or posts have to be made free)

• Capacity has to be distributed justly amongst courts to ensure equal workload (Courts with overcapacity lose first the posts, secondly the staff when pensioned or replaced)




Use of PEBB§Y Date in Budgeting

Weighted Case Data per Region

Staff ListHigh Courts

Min of Justice

Min of Finance

ParliamentGovernment

Does it work this way...or not ?

Most hopefullyBut not all the time

Weighted Case DataLower Courts

Staff Lists Low Courts




Performance and Quality Management





What do we want ?

• Performance Management

Ensure best possible Service delivery by the couer

Multi-Dimensional Approach (finance is one dimension)

• Performance Budgeting (Funding)

Assignment of State Resources according to the performance of the courts

Orientation to the financial dimension




What do we need

1. A working system to collect / retrieve the necessary information in the necessary disaggregated form

2. A system to present the data in an understandable form with complete „drilling thru“ and analysis capacities

3. An evaluation and reaction system (legal / organizational framework)

4. At the end a complete Management Information System (MIS) for the courts




Dimensions of Performance

Dimension Key Customer

Quantitative Output (in Workload Categories)

Ministry of Finance

Quality Court User

Speed Court User

Systems (HR, IT, Structures, Workflows)

Internal use




The Basic Approach – Balanced Scorecard

Multi-Dimensionality of „Performance“ in the Court system

Interaction ofperformance dimensions

Strategic long term controlling and optimizationBased on customer / stakeholder goals




A BSC Approach takes Multi-Dimensionality and Interaction into consideration

• Take Multi-Dimensionality into consideration

• There is not one single goal, there are more than one, usually– Our financial situation today– Satisfaction of the customers to ensure our

profit of tomorrow– Staff capacity to keep satisfaction and income– Investments in the future to ensure the profit

of next year• Take Interaction into consideration

– Investment in Staff, Systems, Innovation today improves performance in the future




BSC – From Private Sector Needs to Court Needs

Company• Profit• Customer Satisfaction• Trained + Motivated Staff• New Products / Markets

Court• Court productivity• Speed / Minimized Backlogs• Trained and motivated staff• Minimized (successful) appeals

Elaborate Goals for all DimensionsSupervise goals continuosly by IndicatersDont allow the priority of one dimensionCare for a „balanced“ score




Productivity Indicator: Hours per weighted standard case in the District courts SAMPLE

HoursjudgeTime per case

Jud

ge

FT

PE

pe

r C

ou

rt

9

0

5

10

15

20

25

30

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5

OD

KH

LV

KO

CY

DZ

9

0

5

10

15

20

25

30

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5




Case Time Indicator:Cumulated Civil Cases Duration in

Months

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 24 36 60 120

mor

e

1-Cum

2-Cum




Case Time Indicator (2):Cumulated Admin Cases Duration in Months

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9

10

11

12

13

14

15

16

17

18 24

60

12

0

mo

re

1-CUM

2-CUM




Quality Indicator: Appeals and District Court Quality

988,824(100%)

97,275(9.8%)

LocalCourt

Decisions

Appeals(ProceduralMistakes)

SuccessfulAppeals

max

min

Wrong Decision ?Missing Knowledge of new Jurisdiction ?Bad Knowledge of Procedures ?Bad Communication ?Greedy Lawyers ? Wrong Case

InvestigationWrong Law ApplicationFailure in ProceduresDifferent Legal Opinion

25,587(26.3%)




Systems Indicators in Human Resources and IT

Human Resources:

Staff Turnover (Retention ofqualified staff is a success indicator, so the value should be low.

Training days (More training improves staff quality, leads to more output, higher speed, less failures

ICT Systems

Different development stages exist and can be measured by a IT quality indicator

In general, the more staff has access to a server based intranet with common databases and functions and central internet access,• the higher the staff productivity,• the higher the work speed,• the lower the share of failures




BSC (Standard Form) For a Sample Court

Productivity

872

Weighted (standard) cases per judge/year

Speed

144,85 DAYS to decisive meeting

Staff Turnover

9,96 %(new + leaving/2) staff

Of total staff p.a.

Quality

3,17%Successful Appeals of

Considered cases

Goal ???City average 847

Region average 977Region best 25%

1.0724 Pilots Average 819




BSC Diagram - Model




BSC Table - Model

Introduce government goalsfor this




A View to Reality – Denmark (1)




A View to Reality – Denmark (2)

Інформація, яку можна отримати завдяки показникам. Вживання відповідних заходів.




Performance and Management

• Put instruments in place to identify the real reasons for performance problems in whatever BSC area

• Give HoC the Instruments to identify and solve problems in their own courts

• Try to create Incentives for good performance• Re-arrage the staff distribution between courts in

a region if needed• Develop plans to improve the „System“ situation

(HR and IT)




Performance and Funding

• Assign Budget and Posts according to the weighted caseload of the courts (we need the case complexity based workload determination systems)

• Have an instrument in place to give relief to courts with problems (Judge Pool – retain a percentage of open posts for this)

• Consequently withdraw posts from „inefficient“ courts and put judges to courts with work overload.

• Put a bonus system in place for very efficient courts, giving extra funding for example for IT, materials and so on.




Feedback Loop and „Continuous Improvement“

• Performance management should not end with isolated reactions on individual indicator problems.

• Follow up of results over the time and the analysis of the effects of changes is necessary

• Continous improvement has to be a key element:– You never reach an „Optimal“ result and it is in no case in one

step– If you implemented one improvement, you see what can be

done next– Actively promote re-thinking all achievements– Use BSC Indicators to measure the effect of each step

• Knowledge exchange Groups should be established to learn from each other and to improve the indicators continuously




End of the PresentationThank you for your Attention

Questions:Dr Axel G Koetz

Managing Partner, KPI CologneE-mail: [email protected]

KPI Management and Policy Consultants Conference on Performance Evaluation of the Judiciary Ankara,...

Documents

Transcript of KPI Management and Policy Consultants Conference on Performance Evaluation of the Judiciary Ankara,...