1

Skills Competency Education for

New PI Directors & Coordinators

Session Three February 14, 2007

Data Aggregation and Assessment

Sponsored by: The MT Rural Healthcare PI Network

Co-Sponsored by: Mountain Pacific Quality Health

2

Today’s Session

Recap Session 2: Data Collection

Turning Data into Useful Information Step one: aggregate Step two: assess Step three: data quality issues Tools

Questions

3

Why Aggregate & Assess Data?

To increase the usefulness of data

To help make it ‘actionable’

To identify areas where other or more data needs to be collected

To identify mistakes, poor quality data

4

Why Aggregate & Assess Data?

To provide objective information as the foundation of objective decision-making

Always end with a decision about how to go forward

Ultimately, supports the organization in achieving its mission, vision

5

A Little Background… Statistics

The science of probability Can become very complex We are not statisticians We don’t need to be; someone else has

done that work for us

PI uses basic statistical methods and tools to scientifically, objectively support improvement efforts It is a scientifically sound approach It is improvement, not research

6

Step One: Aggregate the data

7

Step One: Aggregate the data

Group like-kinds of data together Called a data set Can start this process during collection

Aggregation tools Log sheets Table (matrix) Dot plot

8

Simple Log Sheet:Group one kind of data together

Data set name

Data label

Data bit label

Data bit

MR Review H + P in 24 hrs

Record 1 √

Record 2 √

Record 3 √

Record 4 √

etc

9

Med Rec Review

H & P Care Plan

Immunizations

DC Sum

Record 1 √ √ √

Record 2 √

Record 3 √ √ √ √

Record 4 √ √ √

Table, MatrixGroup Several Kinds of Data

10

Dot Plot (Scattergram)

Medication Room Refrigerature Temperatures

34

34.5

35

35.5

36

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Day

11

Data Aggregation Limits

What do we know so far about the value or importance of the data we’ve collected?

Can we determine if the variation present is “significant”?

Can we draw overall conclusions from it? Can we take constructive action based

on it?

12

Data Aggregation Limits

If our data represents a sample, what can we say, or infer, about the rest of the group (“population”) based on our aggregated data?

Making valid statements of this kind is the work of ‘inferential statistics’

Example: reviewing 10% of closed records

13

Step Two: Assess the data

14

Assessment Techniques

Calculate measures and/or rates

Construct charts and graphs Look for trends and relationships

Evaluate the variation: is it… Normal or an outlier? Common cause or special cause?

15

Assessment: Calculations

Frequency Relative Frequency Percent, percentage Range Average (mean)

Median (middle) Quartile Decile

16

Med Rec Review

H & P Care Plan

Immunizatio

ns

DC Sum

Total (Freq)

Record 1 √ √ √ 3

Record 2 √ 1

Record 3 √ √ √ √ 4

Record 4 √ √ √ 3

Total (Frequency)

4 3 1 3

Calculations: Frequency“Count” data: how often something happened or was observed

17

Med Rec Review (n=4)

H & P Care Plan

Immu DC Sum

Total

RF

Record 1 √ √ √ 3 0.75

Record 2 √ 1 0.25

Record 3 √ √ √ √ 4 1.0

Record 4 √ √ √ 3 0.75

Total (x) 4 3 1 3

RF (Rel Freq)

4/4 =1.0

¾ =0.75

¼ =0.25

¾ =0.75

Calculations: Relative Frequency

Relative Frequency (RF) = x / n

18

Med Rec Review (n=4)

H & PCare Plan

ImmuDC

Sum

RF Percent

Record 1 √ √ √ 0.75 75%

Record 2 √ 0.25 25%

Record 3 √ √ √ √ 1.0 100%

Record 4 √ √ √ 0.75 75%

RF (Rel Freq)

4/4 =1.0

¾ =0.75

¼ =0.25

¾ =0.75

Percent 100% 75% 25% 75%

Calculations: PercentPercent (%) = RF x 100 or ((x/n) * 100)

19

Med Rec Review (n=4)

H & PCare Plan

ImmuDC

Sum

Record 1 √ √ √

Record 2 √

Record 3 √ √ √ √

Record 4 √ √ √

Percent 100% 75% 25% 75%

Range = 100-25

= 75%

Calculations: RangeSubtract the lowest value from the highest value

20

Med Rec Review (n=4)

H & PCare Plan

ImmuDC

Sum

Record 1 √ √ √

Record 2 √

Record 3 √ √ √ √

Record 4 √ √ √

Percent 100% 75% 25% 75%

Average= 275/4

= 68.7%

Calculations: Average, meanSum of all values / n

21

Assessment: Charts, Graphs

Construct charts and graphs

Add limits for evaluation Control limits: upper, lower Threshold: point we will intervene Benchmark: internal or external

Look for trends and relationships

22

Charts and Graphs

Help us understand and identify normal variation in systems and processes and leave it alone

Range of normal body temperatures, pulse rates and blood pressures

Record but leave it alone

23

Charts and Graphs

Help us understand and identify variation that is not normal within a system or process and take corrective action that will reduce or remove it.

Ice pack to reduce extreme fever Medication to reduce elevated blood

pressure or heart rate

24

Charts and Graphs: What’s Normal? The Standard Normal Curve

34.1 %

13.6 %

2.2 %

0.1 %

Mean + 1 SD + 2 SD + 3 SD- 1 SD- 2 SD- 3 SD

34.1 %

13.6 %

2.2 %0.1 %

+/- 1 SD = 68.2 % area

+/- 2 SD = 95.4 %

+/- 3 SD = 99.8 %; upper and lower control limits

25

Charts & Graphs:Add Control Limits

Glucose Control Values

86

88

90

92

94

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Day

Upper Control Limit, + 3SD

Lower Control Limit, - 3SDMean, average

Source: mathematical calculations, internal or external

26

Charts & Graphs:Add Threshold for Intervention

Mortality Rate % Inpatients

0.0

0.5

1.0

1.5

2.0

2.5

Q4'04

Q1'05

Q2'05

Q3'05

Q4'05

Q1'06

Q2'06

Q3'06

Q4'06

12 M

onth

Rollin

g A

v in % Threshold: a

predetermined point at which action will be taken

Source: internal discussions

27

Charts & Graphs:Add Benchmarks

Heart Failure Clinical Care Guidelines

0

20

40

60

80

100

120

Q4 '06

Succ

ess

Rate

in %

LVS AssessACEI/ ARBDischarge InsSmoke Cess

Benchmark: a pre-determined level of desired performance

Source: internal or external

28

Charts & Graphs:Look for Trends, Relationships

Percent All Falls By Day of Week

0

5

10

15

20

25

Mon Tue Wed Thur Fri Sat Sun

Per

cent

More falls: why?

Fewer falls: why?

29

Charts & Graphs:Look for Trends, Relationships

CAH Admission Sources, 2004

59%16%

12%

8%

3%

2%

Emergency room

Physician clinic

Scheduled surgery

OB

SNF

Unexpected post op

30

Evaluate Variation If know what ‘normal’ looks like, you are

Able to identify outliers: unusual, unexpected process/system events

Able to evaluate relative severity or importance when multiple factors contribute

Able to identify improvement and work to maintain gains

31

Evaluate Variation

Common Cause Variation The expected variation inherent in

any process due to the normal interaction of the process variables.

Special Cause Variation Unexpected variation in the process

due to a specific reason or cause.

32

Evaluate Variation: The Standard Normal Curve

34.1 %

13.6 %

2.2 %

0.1 %

Mean + 1 SD + 2 SD + 3 SD- 1 SD- 2 SD- 3 SD

34.1 %

13.6 %

2.2 %0.1 %

+/- 1 SD = 68.2 % area

+/- 2 SD = 95.4 %

+/- 3 SD = 99.8 %; upper and lower control limits

33

Normal Distribution

Normal Distribution

456789

10111213

1 2 3 4 5 6 7 8 9 10 11 12 13

Series1

+3 SD

- 3 SD

mean

Even and varied distribution of points on both sides of the mean, all within control limits; common cause variation; the process is said to be ‘in control’ and/or ‘stable’.

34

Evaluate Variation:Westgard Rules for Control Charts

1 Point Outside Control Limits

2-2SD Rule

4SD Rule

1-3SD Warning

6 point trend

7 + point trend

Sawtooth

Source: http://www.westgard.com/mltirule.htm

35

1 Point Outside Control Limits

1 Point Outside Control Limits

456789

1011121314

1 2 3 4 5 6 7 8 9 10

Series1

+3 SD

- 3 SD

mean

1 point exceeding the upper or lower control limit is special cause variation

36

2-2SD Rule

2:2 SD Rule

456789

10111213

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Series1

UCL

LCL

mean

2 consecutive points greater than or less than 2 SD; special cause variation

37

1:4SD Rule

1:4SD Rule

4

5

6

7

8

9

10

11

12

13

1 2 3 4 5 6 7 8 9 10 11 12

Series1+3 SD- 3 SDmean

Change of 4SD up or down is special cause variation

38

3 SD WarningChange of 3SD; special cause variation may be present; investigate

1: 3SD Warning, Cross Center Line

456789

10111213

1 2 3 4 5 6 7 8 9 10 11

Series1

+3 SD

- 3 SD

mean

39

6 Points on One Side of Mean

6 Points on One Side of Mean

4

5

6

7

8

9

10

11

12

13

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Series1UCLLCLmean

6 consecutive points on one side of the mean is special cause variation

40

7 Ascending, Descending Points

7 Point Trend, Ascending or Descending

456789

10111213

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Series1

UCL

LCL

mean

7 consecutive ascending or descending points is special cause variation

41

Sawtooth

Sawtooth

4

5

6

7

8

9

10

11

12

13

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Series1

+3 SD

- 3 SD

mean

A sawtooth pattern is not normal, it is special cause variation

42

Practice Assessment

Reduce Immunization Failure Rate

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10

Week of Improvement

Num

ber

of Pts

Standing orders

24 hour review

Discharge order sheet

Are the reductions in immunization failure rate below ‘significant’?

Hint: think about how the Westgard 3SD and 4SD rules looked

43

Step Three:Resolve Data Quality Issues

44

Data Quality: Validity

Is the data itself valid?

Are your conclusions valid?

Is the data accurate

Is the data reliable

45

Valid Data Accuracy

Precision: how close is the measured value to the true value?

Confidence intervals: how confident can you be that they are the true value?

46

Valid Data Reliability:

do repeated measurements produce the same results?

Sample size Confidence

intervals

47

Valid Data: Sample Size

30 data points approximates the normal curve

no less than 10 data points unless it is 100%

10% of a large population

100% of a small population

For PI, the data just needs to be valid and actionable!

48

Questions?

Next Time:

Performance Reports

Wed, March 14 1pm

49

PI Ed Session 3 References Handbook For Improvement, 3rd Edition;

Healthcare Management Directions, Inc.; 2002.

Norman, G. and Streiner, D; Biostatistics The Bare Essentials; Mosby-Year Book Inc; 1994.

http://www.westgard.com/mltirule.htm

www.mtpin.org