Quality Assurance and Peer Review -...
Transcript of Quality Assurance and Peer Review -...
Quality Assurance and Peer Review
Professor Alan Paterson OBE
CPLS, University of Strathclyde
The ILAG Family 2019
Concepts and levels of Quality
➢Historically, synonymous with Excellence
➢Japanese manufacturers – quality as:
a) conforming to international standards, or
b) “fitness for purpose”, or
c) “value for money”.
➢Binary notion of quality replaced by a continuum.
The Spectrum of Quality
EXCELLENCE
COMPETENCE PLUS
THRESHOLD COMPETENCE
INADEQUATE PROFESSIONAL SERVICES
NON PERFORMANCE
Figure 1: The Performance Continuum
© Paterson & Sherr
Setting the Pass Mark
➢ Best practice
➢Good practice
➢ To pass a peer review assessment
➢ Fitness to practice – threshold competence. The discipline standard. Does this change with time?
➢ The negligence standard for specialists
➢ The negligence standard
➢Minimum requirement of Art 6 Jurisprudence ECHR
➢ To permit an “ineffective assistance of counsel” appeal
INPUTS OUTCOMES
STRUCTURE
PROCESS
Education
Qualification
Licensure
Experience
Knowledge
Skill sets
Property
Equipment
Library
Training
IT
Contacts
Management Allocation
Supervision
TrainingSystems
Performance
Advice
Correspondence
Decisions AdvocacyStrategy
Fact Gathering
Legal Analysis
Client Handling
Practice Management
Financial
Social
Public
Private Inaction
Action
Sentence
Damages
International Developments
UN Principles and Guidelines on Legal Aid (2012)
Principle 13
“All States to put in place mechanisms to ensure that all legal aid providers possess the education, training, skills and experience that are commensurate with the nature of their work “
• Criteria for accreditation of the legal aid providers (Guidelines 15);
• Quality standards for legal aid providers and paralegal (Guidelines 16 and 14);
• Adequate training, and supervision by qualified lawyers (Guideline 14 –mentioned in the context of paralegals only, though this would probably be appropriate for lawyers as well);
• Monitoring and evaluation mechanisms to assess the quality of the services provided by lawyers and paralegals (Guidelines 16 and 14)
International Developments II
EU Commission
Recommendation of 27 November 2013 on the right to legal aid for suspects or accused persons in criminal proceedings
Section 3 on effectiveness and quality of legal aid establishes that “Legal assistance provided under legal aid schemes should be of high quality in order to ensure the fairness of proceedings. To this end, systems to ensure the quality of legal aid lawyers should be in place in all Member States.”
This includes accreditation of legal aid providers and appropriate training; appointment of lawyers while taking into account the wishes of the accused and allowing accused to choose a provider from a list; and replacement of lawyers who fail to provide adequate legal assistance.
[ Input & Structure ]
EU Directive 2016/1919 of the European Parliament and of the Council of 26 October 2016 on legal aid for suspects and accused persons in criminal proceedings and for requested persons in European arrest warrant proceedings, establishes in Article 7 ‘Quality of legal aid services and training’:
1. Member States shall take necessary measures, including with regard to funding, to ensure that: (a) there is an effective legal aid system that is of an adequate quality; and (b) legal aid services are of a quality adequate to safeguard the fairness of the proceedings, with due respect for the independence of the legal profession.
[ Input & Structure ]
International Developments III
UN Global Study findingsImproving quality of legal aid services was identified as the number one priority for Member States in the Global Study on Legal Aid (UNODC,2016)Monitoring/ Evaluation mechanism of Quality of legal aid?Bar Association 35%Legal Aid Board 20%Ministry of Justice 20%None 11%However, 57% of responding states indicated that the mechanism used to assess quality was client complaints.Only 26% reported taking any proactive steps to assess quality – the most frequent measure being client satisfaction surveys.
Publication of the UNODC Handbook on Ensuring Quality of Legal Aid Services in Criminal Justice Processes (2019 )
➢ Global Study: Most pressing technical assistance need cited was “development of quality criteria for legal aid providers”
➢ To respond to this need, as a first step UNODC aided by Miri Sharon and Alan Paterson with 50 other experts from all regions developed a new Handbook to provide an overview and guide on measures to constantly improve and monitor the quality of criminal legal aid services
What we know about what is out there ?
Nikartas and Limante: Tools and Criteria for Measuring Legal Aid Quality: Guidelines for EU Member States (2018)
Accreditation and Training
Complaints
Client satisfaction
Peer Review
“The evaluation of the serviceprovided against specified criteria and levels of performanceby an independent lawyer with significant current or recent practical experience in the areas of law being reviewed”
Peer Review
The Spread of Peer Review England and Wales, Scotland, Canada (Ontario), Finland, Georgia, Moldova, South Africa, Chile, China,
New Zealand, Ukraine, The Netherlands and Australia
14
MECHANISMS OF CONTROL AND MEASUREMENT
• Peer review,
• External Audits,
• Claims
• Semiannual and Final Reports
In addition, the Public Criminal Defense has other measurements and instruments
that help to control the service, evaluate its performance and account for it.
INDICATORS OF CRIMINAL DEFENSE DELIVERY
EXTERNAL AUDITS OF USER SATISFACTION
GLOBAL QUALITY INDEX
INTEGRATED DATA ANALYSIS
BUSINESS INTELLIGENCE TOOLS
Lessons from Chile
Peer Review SystemGeneral flow chart
Annual plan
Peer reviewsschedule
Preliminaryreview
Fieldwork, sample and
analysis
ReportObjections
Districtsupervision
Follow up
Evaluation
The aim of Peer Review in the UK
• To enhance efficiency and effectiveness?• To assess the quality of performance and
outcomes achieved by legal aid providers?• To reassure the public and the state as to quality
and value for money?• To enhance choice for the public?• To exclude poor performers from the list of legal
aid providers?• To attain continuous improvement by providers?• To change the culture of legal aid providers?
Competing Perspectives
• Appointed by panel of laypersons and lawyers after an open recruitment process.
• Peer reviewers are experienced solicitors.
• Peer reviewers receive training for 3 to 4 days on reviewing and marking files according to the criteria.
• Training fosters consistent application and interpretation of the criteria.
• Once accepted as peer reviewers, they have to go through a peer review themselves.
• Cannot review a competitor firm or one they have links to.
Scottish Peer Reviewers
File selection and Firm selection
• 5 random files (OR 10% of a lawyer's files where the client is likely to be vulnerable, adult incapacity; mental welfare or immigration cases ) for each lawyer in the firm are assessed against 14 published peer review criteria. These include:
• Effectiveness of lawyer’s initial fact and information gathering skills.
• Effectiveness of communications.
• Whether the lawyer identified the need for appropriate experts or counsel.
• Evidence of adequate preparation for each stage of the case.
• Evidence of the lawyer taking steps agreed with the client with a reasonable time.
• Whether the case was concluded effectively.
Criteria are Client Centred
Peer Review Cases and Criteria - CIVIL
• 8 random files for each lawyer are assessed against published peer review criteria for less serious criminal cases (e.g. assault), serious criminal cases (robbery) and appeals. Criteria include:
• Whether the lawyer made an effective application for bail,
• Whether special defences were considered,
• Adequate preparation for trial,
• Advice given to the client on how to plea,
• Advice given to client about sentence,
• Advice give to client about appealing,
Criteria are client centred
Peer Review Cases and Criteria - CRIMINAL
Peer Review in less important Cr cases
Client contact at initial interview stage whether in custody or in an office 1. How effective were the solicitor’s initial fact and information gathering skills,
including identifying any additional information required and taking steps necessary to obtain it?
1 2 3 C N/A
This should also include consideration of any advice given to the client during a police interview at a police station, where appropriate.
2. Did the solicitor give the client correct and appropriate advice about the defence case including whether it is stateable and whether an early plea should be considered? Has a discount for an early plea been canvassed with the client?
1 2 3 C N/A
File and Lawyer Scoring
54321
EXCELLENCECOMPETENCE PLUS
THRESHOLD COMPETENCEPOOR
NON-PERFORMANCE
“That’s it? That’s Peer Review?”
Points of Good Practice
Good Terms of Engagement letters
Use of easy to understand language used in letters
Keeping clients well informed of the development of their cases
Sensitivity shown when dealing with child witnesses
Clear and easy to follow correspondence
Excellent initial letter of instructions - supplied master copy for
circulationFile keeping
Good court minutes endorsed on the front cover of the file
Detailed notes with a careful attitude to preparation
Use of a checking system
The solicitor carries out file reviews and records these
Where appropriate the solicitor will record comprehensively the
narration given by the Crown: this could be important in the event of
an Appeal
Communications
Points of Good Practice II
Good and detailed gathering of information
Early resolutions in cases benefitting both clients and the wider justice system
Application of S71 procedure to identify and resolve matters prior to trial in
Solemn cases
Copy legislation on file, copy case law on file
Solicitor negotiating the discontinuation of proceedings with Crown
Full and detailed notes taken after Full Committal where solicitor discussed
bail with client followed by careful notes taken upon sentence in case of
appeal
Legal work
The application for legal aid was very detailed and demonstrated that the solicitor had a detailed knowledge of the client’s background and relevant issues.
Legal aid
Areas for Improvement
Client meetings being held too infrequently
Style and contents of letters poor
Sparse instructions to agents
Letters to clients containing wrong dates and incorrect information
Lack of letters to clients updating them that disclosure was available to discuss
Communications
Conflict of interest when acting for accused and co-accused
Failure to attend to conduct a trial
Absence of S196 early plea discount
Limited information obtained regarding the defence.
Defence witnesses not cited when they should have been
Legal work
Areas for Improvement II
Conflicting information about client's financial circumstances (eg working, not working)
Lack of vouching of client’s financial circumstances for the legal aid application
Legal aid
Files not being kept chronologically
Papers being kept loose leaf
Lack of printed LA applications on file to see what had been submitted to the Board
File keeping
Lessons from Peer Review
Quality is not binary
The pursuit of Quality can have different objectives
Quality, Peer Review and the independence of the profession
Quality, Peer Review and client confidentiality
Peer Review and expense
The Quality floor is a dynamic rather than a static target
Lessons from Peer Review
Quality is not binary
The pursuit of Quality can have different objectives
Quality, Peer Review and the independence of the profession
Quality, Peer Review and client confidentiality
Peer Review and expense
The Quality floor is a dynamic rather than a static target
Lessons from Peer Review
Quality is not binary
The pursuit of Quality can have different objectives
Quality, Peer Review and the independence of the profession
Quality, Peer Review and client confidentiality
Peer Review and expense
The Quality floor is a dynamic rather than a static target
Lessons from Peer Review
Quality is not binary
The pursuit of Quality can have different objectives
Quality, Peer Review and the independence of the profession
Quality, Peer Review and client confidentiality
Peer Review and expense
The Quality floor is a dynamic rather than a static target
• Scots cost of QA in 2013/14 was 220, 000 Euros (2.3% of Scots expenditure on administration) ( 123 Euros per file)
• (Complaints 2016/17 2,582 Euros per case) .
Costs and Benefits
Lessons from Peer Review
Quality is not binary
The pursuit of Quality can have different objectives
Quality, Peer Review and the independence of the profession
Quality, Peer Review and client confidentiality
Peer Review and expense
The Quality floor is a dynamic rather than a static target
Quality Targets are Dynamic
1) Technological advances and innovation mean that the requirements of good practice change e.g. the digital discovery of evidence, the probabilistic basis of DNA profiling and the hidden biases of AI algorithms
2) Twenty years ago the model was “the professional knows best”. Patient care and client communication were not person centred but professional centred. Today, case law and professional ethics requires professionals to be client centred – the bar has clearly been raised.
2012-19 File Stats All ReviewersMarker FM F % F D % D
T Allan 550 29 5.3 - 92 16.7
S Brown 698 16 2.3 124 17.8
I Cruikshank 273 20 7.3 22 8.0
K Douglas 638 47 7.4 60 9.4
C Dunipace 701 23 3.3 82 11.7
T Gallanagh 571 17 3.0 46 8.1
D Henderson 407 67 16.5 29 7.1
M Hutchinson 408 10 2.5 14 3.4
R Lindsay 225 26 11.6 35 15.6
F Mccready 220 13 5.9 22 10
I MccLelland 314 9 2.9 12 3.8
J McKay 608 86 14.1 99 16.3
L McPhie 129 4 3.1 9 7.0
P Ralph 353 46 13 66 18.7
J Rafferty 138 3 2.2 30 21.7
G Robertson 666 8 1.2 109 16.4
A Ross 568 40 7 97 17.1
S Walker 417 18 4.3 57 13.7
G Wiggins 777 89 11.5 78 10.1
R Yuill 146 1 0.7 18 12.3
Total 8,661 (11,307) 571 1,084
Overall Av. 433 28.6 6.6% 54.2 11.7%
Civil 3rd Cycle review scoresReviewer PM FM %F %D
1 69 338 13.3 16.3
2 70 365 10.7 22.5
3 32 172 2.9 23.8
4 69 356 8.4 13.8
5 51 235 10.2 17.0
6 118 626 12.0 19.5
7 126 656 8.5 16.5
8 124 607 14.3 12.7
9 65 316 16.5 6.0
10 89 458 10.5 8.5
11 175 1009 8.4 11.9
12 129 642 17.1 8.9
13 98 565 10.3 11.2
14 48 259 6.6 24.3
15 81 369 18.2 29.0
16 53 264 12.9 9.9
17 73 334 11.7 13.5
18 39 251 7.6 12.0
19 103 596 6.9 15.9
20 44 189 16.9 9.5
21 101 567 19.6 5.1
22 108 555 15.5 15.7
TOTALS 1865 9729
Overall Av 84.8 442.2 11.9 % 14.1%
• Evidence that peer review is driving up standards.
• Fail rate has reduced on criminal.
• Numbers of Final Reviews and Special Reviews has fallen on civil.
• Numbers of solicitors achieving the highest marking has increased.
• Survey of solicitors in 2013 showed that 84% of respondents had a positive or neutral opinion on whether the QA scheme was an effective way of ensuring quality.
• A few solicitors who were doing very small amounts of legal aid, or had failed reviews, withdrew from our register of suppliers. The great majority of lawyers/firms that fail their initial review pass the next time round
Driving up Standards?
Practice Unit Identified
8 Files/ Practitioner Selected
Reviewer(s) Selected
Practice Unit Informed
Routine File Review Pass Fail
Extended Review
Or
Deferred Extended
Review
Pass Fail
Final Review
(3/6 Months) Pass
Fail
SLAB Notified
Consider Withdrawing
Compliance Certificate
Compliance
Certificate
Issued
Monitoring of
Practice Unit
Special Review
Compliance
Certificate
Issued
Compliance
Certificate
Issued
Continued for
Comment
from Practice
Unit
The Role of the Quality Assurance Committee
1) All Scots assessments are reported to a Quality Assurance Committee (QAC ) or Criminal Quality Assurance Committee (CQAC). Each comprises of:
3 solicitors appointed by the Law Society
3 non-legal members (representing the interests of the consumer)
3 members appointed by SLAB
2) Meets monthly to consider all summary reports andfailed file reports from peer reviewers. It is the QAC , not the reviewer to decides whether a practitioner or firm passes or not.
The Role of the CQAC a) Pass ( with feedback ) provider with a good or average grade.
Result: Not reviewed again for several years;b) Award a marginal pass with detailed feedback
Result: Reviewed again in 12-18 months c) Continue consideration of the lawyer for comment from the lawyer as to the report of the reviewer including flaws in earlier reviews;
Result : Lawyers expected to address past flaws even if passed last review – continuous improvement
e) Award a marginal fail with feedback and order a “deferred extended review” in nine months –
Result: Lawyers given early chance to improvef) Award an immediate fail and order an immediate “extended review”
Result: Public protected from really poor providersg) Order an immediate special review;
Result: Public purse protected
CRIMINAL CIVIL
2012-2019 2012-2018
Registered firms 613 ( 454 ) 622
Registered practitioners 1,655 ( 687 ) [Some do no legal aid or have no files ]
Reviews undertaken 1149 602
Reviews continued re comments 122 (11%) 299 (48%)
Extended Review / deferred ER 6/69 ( 7%) 21/35 (9%)
Practitioners to final 8 (1%) 12 (2%)
Special reviews 4 4
Marginal passes 111 (10%) ??
Quality Assurance Scheme Peer Reviews
Solicitor 1 2(a) 2(b) 2(C) 3(a) 3(b) 3(C) 3(d) 4 5 6 7(a) 7(b) 8 9(a) 9(b) 10 11 12 13(a) 13(b) 14 Overall
I MCCulloch (KD) 2 2 2 1 1 2 2 2 2 2 N/A 2 2 2 2 2 1 1 C 2 2 2 2
2 C 2 2 1 2 2 N/A 1 2 N/A N/A N/A 2 2 2 1 1 N/A 2 2 2 2
3 3 2 2 1 2 2 2 3 3 N/A N/A N/A 3 2 2 3 3 N/A 2 2 2 4
2 C 2 1 1 2 2 2 2 2 N/A 2 N/A N/A 2 C 2 C N/A N/A N/A N/A 2
2 2 1 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 2 N/A N/A N/A N/A 3
G McIntyre (KD) 2 2 2 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 2 2 2 N/A 2 3
2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 2 N/A N/A N/A 3
2 2 2 2 2 2 N/A N/A 2 2 N/A N/A N/A N/A N/A N/A 2 2 N/A N/A N/A N/A 3
3 2 2 2 2 2 2 2 3 3 N/A N/A N/A N/A 2 3 3 3 3 N/A N/A N/A 4
2 2 2 2 2 2 2 2 2 2 N/A N/A N/A 2 2 2 3 3 N/A 2 2 2 4
A Mackie (KD) 2 2 2 2 2 2 2 3 3 3 N/A N/A N/A 2 2 2 2 3 2 2 2 1 3
2 2 2 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 1 N/A 2 N/A 2 3
2 2 2 2 2 2 2 2 2 2 N/A N/A N/A N/A 2 2 2 2 2 N/A N/A N/A 3
2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 N/A 2 N/A C 3
2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 N/A 2 N/A 2 3
Case
Report
Analysis
Total Noper criteria 1 2(a) 2(b) 2(c) 3(a) 3(b) 3(c) 3(d) 4 5 6
7(a)
7(b) 8 9(a) 9(b) 10 11 12 13(a) 13(b) 14
1 43 58 290 237 638 25 10 21 196 53 32 6 66 25 199 98 227 167 13 83 14 134
2 5920 6004 5646 4547 4680 6385 5647 4215 5359 5174 3431 1649 3697 2326 4490 5068 5514 5301 2153 4092 1433 3702
3 548 298 14 33 122 53 68 156 415 515 73 63 22 196 45 369 275 415 96 8 22 10
C 170 308 494 759 1107 133 85 72 105 57 139 44 221 41 226 188 94 201 79 445 85 583
N/A 19 31 256 1125 147 99 890 2237 626 901 3023 4939 2695 4113 1741 973 591 616 4360 2070 5146 2266
Overall CR file scores 2012-19
1 43 (0.4%)
2 653 5.8%)
3 9,281 (82.1%)
4 1,271 (1.2%)
5 60 ( 0.5%)
Criminal Peer Review E & WApril 2017 – March 2018 April 2018 – March 2019
•The most common rating awarded to Crime providers in 18/19 was ‘Threshold Competence’ (3). This was consistent
with provider performance reported in 17/18.
Rating
Number
of
providers
%% Below
Competence
5 16 3.48%
4 159 34.57%
3 228 49.56%
2 57 12.39%12.39%
1 0 0%
Total460
Ratings
Number
of
provider
% Below
Competence
5 11 2.18%
4 189 37.43%
3 256 50.69%
2 48 9.5%9.7%
1 1 0.20%
Total 505
• Typical causes of fails:• Delays in taking action or applying for legal aid
• Poor communication with clients relating to the operation of the costs rules for legally aided persons
• Poor file notes of phone call or interviews
• No terms of engagements letters on files.
• Occasional examples of abuse:• Private charges to legally aided clients or padding of
accounts.
Results from peer reviews
Running a pilot programme
1) Appoint a body to oversee the pilot programme partnership
2) Identify the standards of criminal practice 3) From these derive the peer review criteria for
assessing process and outcome4) Identify the assessment system5) Run a selection competition for peer reviewers6) Decide on what should be assessed – performance in
court, files7) Decide on whether additional factors should be taken
into account e.g. client satisfaction; complaints