Post on 02-Jan-2016
Tom MarshallDepartment of Public Health & Epidemiology,
University of BirminghamT.P.Marshall@bham.ac.uk
Understanding VariationUsing Registry data to drive improvement - what makes
clinicians take note of statistics
Acknowledgements
Mohammed A. Mohammed,
Department of Public Health & Epidemiology, University of Birmingham
Quality & Variation• Conventional tools
– Standard setting• Clinical Audit
– Ranking or League tables
– Hypothesis testing
• Effect– Pass or fail
– Action on those that fail
• Another way ..
Shewhart’s Concepts• Letter a
COMMON CAUSEACTION: PROCESS
SPECIAL CAUSEACTION:
FIND & ELIMINATE
PROCESS OF WRITING
Basis of Control Limits
• Tchebycheff’s theorem
X mean +/- t SD
P > 1 - 1/t2
• t=3
• Economic– common cause vs special cause
Special cause variation - action
1. Data • Accuracy• Definition of errors
2. Raw materials• Difficulty of tasks
3. Equipment, facilities, staffing• Typewriters, workload, lighting
4. Processes, procedures• How is the work organised?
5. People• Skill levels and techniques
Application to Health Care
• Case studies
Surgeon Variability• Surgeon Survived Died %• A 82 16 16• B 58 8 12• C 49 9 16• D 45 7 13• E 37 15 29• F 41 5 11• G 35 3 8• H 26 11 30• I 31 5 14• J 27 7 21• K 28 4 13• L 19 2 10• M 18 3 14
McArdle & Hole BMJ 1991;302:1501-5
Conclusions: “There were significant variations in patient outcome among surgeons after surgery forcolorectal cancer; such differences compromisesurvival. A considerable improvement in overallsurvival might be achieved if such surgery wereundertaken by surgeons with a special interest incolorectal surgery or surgical oncology.”
Surgeon Variability
McArdle & Hole BMJ 1991;302:1501-5
A
BC
D
E
F
G
H
IJ
K
LM
X Number alive
Y N
um
nber
die
d
..
Common cause variation
Fractured Hips
• 90 day mortality (N=580; 104 deaths 18%)– Hospital Mortality
• 1 19/79 24%
• 2 5/24 21%
• 3 16/79 20%
• 4 19/80 24%
• 5 12/80 15%
• 6 4/81 5%
• 7 14/79 18%
• 8 15/63 19%
Todd et al BMJ 1995;301:904-8
Conclusions: “Uncritical acceptance of the "advantages" of hospital 6 should, however, be avoided as random variationalmost certainly plays some part in these findings.”
Fractured Hips
X Number alive
Y N
umbe
r di
ed
Todd et al BMJ 1995;301:904-8
Special cause variation
Western Electric Company Rules
• Additional rules for detecting special causes– 1 data point >3 sigma from mean– 2 out of 3 data points >2 sigma from mean– 4 out of 5 data points >1 sigma from mean– 9 successive data points on one side of mean– Trend of 6 successive data points
Special cause variation: nine successive data points below the mean
Renal Registry Data
• Average haemoglobin per quarter
Run chart of quarterly mean Hb
9
10
11
12
13
14
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
Quarter
Qu
arte
rly
mea
n H
b
UK
Average
Run Chart – sequential data points + mean
Trend of 6 data points
Interpretation
• Rising trend in mean Hb nationally
• Difficult to interpret changing Hb in a single centre except in relation to rising trend nationally
Run chart of quarterly mean Hb
9
10
11
12
13
14
Jan-
97
May
-97
Sep-9
7
Jan-
98
May
-98
Sep-9
8
Jan-
99
May
-99
Sep-9
9
Jan-
00
May
-00
Sep-0
0
Jan-
01
May
-01
Sep-0
1
Jan-
02
May
-02
Sep-0
2
Jan-
03
May
-03
Sep-0
3
Jan-
04
May
-04
Sep-0
4
Jan-
05
May
-05
Sep-0
5
Jan-
06
May
-06
Sep-0
6
Quarter
Qu
arte
rly
mea
n H
b
Middlb
Average
XMR chart of difference between quarterly mean Hb and national average
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0Ja
n-97
Jul-9
7
Jan-
98
Jul-9
8
Jan-
99
Jul-9
9
Jan-
00
Jul-0
0
Jan-
01
Jul-0
1
Jan-
02
Jul-0
2
Jan-
03
Jul-0
3
Jan-
04
Jul-0
4
Jan-
05
Jul-0
5
Jan-
06
Jul-0
6
Qu
arte
rly
mea
n H
b
Middlb
Average
+3 sig
-3 sig
Consistent with national average
9 data points above mean i.e. own long term average
Below national average
10 data points below mean
XMR chart of difference between quarterly mean Hb and national average
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0Ja
n-97
Jul-9
7
Jan-
98
Jul-9
8
Jan-
99
Jul-9
9
Jan-
00
Jul-0
0
Jan-
01
Jul-0
1
Jan-
02
Jul-0
2
Jan-
03
Jul-0
3
Jan-
04
Jul-0
4
Jan-
05
Jul-0
5
Jan-
06
Jul-0
6
Qu
arte
rly
mea
n H
b
Middlb
Average
+3 sig
-3 sig
XMR chart of quarterly mean Hb
9
10
11
12
13
14
Jan-
97
May
-97
Sep-9
7
Jan-
98
May
-98
Sep-9
8
Jan-
99
May
-99
Sep-9
9
Jan-
00
May
-00
Sep-0
0
Jan-
01
May
-01
Sep-0
1
Jan-
02
May
-02
Sep-0
2
Jan-
03
May
-03
Sep-0
3
Jan-
04
May
-04
Sep-0
4
Jan-
05
May
-05
Sep-0
5
Jan-
06
May
-06
Sep-0
6
Quarter
Qu
arte
rly
mea
n H
b
Truro
Average
Run Chart – difference between this centre + UK Average
9 data points above mean
Average determined from first 8 data points
Control Chart – Run Chart + 3 sigma limits
XMR chart of quarterly mean Hb
9
10
11
12
13
14
Jan-
97
May
-97
Sep-9
7
Jan-
98
May
-98
Sep-9
8
Jan-
99
May
-99
Sep-9
9
Jan-
00
May
-00
Sep-0
0
Jan-
01
May
-01
Sep-0
1
Jan-
02
May
-02
Sep-0
2
Jan-
03
May
-03
Sep-0
3
Jan-
04
May
-04
Sep-0
4
Jan-
05
May
-05
Sep-0
5
Jan-
06
May
-06
Sep-0
6
Quarter
Qu
arte
rly
mea
n H
b
Truro
Average
+3 sig
-3 sig
XMR chart of quarterly mean Hb
9
10
11
12
13
14
Jan-
97
May
-97
Sep-9
7
Jan-
98
May
-98
Sep-9
8
Jan-
99
May
-99
Sep-9
9
Jan-
00
May
-00
Sep-0
0
Jan-
01
May
-01
Sep-0
1
Jan-
02
May
-02
Sep-0
2
Jan-
03
May
-03
Sep-0
3
Jan-
04
May
-04
Sep-0
4
Jan-
05
May
-05
Sep-0
5
Jan-
06
May
-06
Sep-0
6
Quarter
Qu
arte
rly
mea
n H
b
Truro
Average
+3 sig
-3 sigConsistent with two stable processes:
before Sept 04& after Sept 04
Special cause variation - action
1. Data (including definitions)
2. Raw materials (case-mix)
3. Equipment, facilities, staffing
4. Processes, procedures
5. People
Monitoring Many Centres
Walter A Shewhart 1931
“The central problem in management and leadership …is failure to understand the
information in variation”
William E Deming 1986 Out of the Crisis MIT pg 309
Summary• Shewhart’s concepts
– Understand variation– Simple & powerful– Guide action– Wide application
• Continual improvement• Clinical governance• Other implications ..
Implications• Prediction
– Limits of common cause variation– Statistical control
• Action– Common cause variation
• League tables, ranking, hypothesis testing all misleading• Improve process/system as a whole
– Special cause variation• Investigate and eliminate (or learn lessons)
• Data order important
How It Works In Industry
• Balanced set of measures
Balanced Set Of Measures
Customer
Financial
Internal ExternalAim
Four or five measures for each box
Balanced Set Of Measures
Patient Experience
Financial / Resources
Clinical effectiveness
Strategic Effectiveness
Aim
Special Causes
• Identify special causes in each domain
• Collate & prioritise for action– Low hanging fruit first