Washington State’s Approach: From CLABSI to SSICSTE Annual Conference, 2013

David Birnbaum, PhD, MPH

Overview

• Now in our 4th year for CLABSI validation  • Two components in our annual protocol  

– Both draw samples that span at least one entire year of surveillance in each hospital. 

– Annually, the first component reaches all (62) hospitals in our reporting network

– The second reaches a number that we do not disclose

• Confidential collaboration in each hospital– “quality improvement” confidentiality laws

Overview (cont.)

• “Internal” self-assessment component– Every hospital in our reporting network follows

our instructions once annually to conduct its own “internal” assessment of CLABSI surveillance program sensitivity & specificity 

• Takes each hospital ≤6 hours per year• Takes us approx. 30-60 minutes to review all the results each year

Overview (cont.)

• “External” validation verification visit component– Hospitals selected on basis of questionable

“internal” results or a spot-check basis – ISO-2859 double-sampling technique used

• Gives sample size for defined statistical power to detect any performance levels below those we specified as acceptable

– β-error (consumer ‘s risk) has more emphasis than α-error (producer‘s risk) in acceptance sampling

• Takes 1 qualified person ≤1 day per hospital to conduct the review   • That equates to less than 0.5 FTE to run our CLABSI validation work • Anticipate when we begin SSI validation it will be a similar staffing workload 

Overview (cont.)

• During our visit, we check to see that:– various practices are in place – sensitivity & specificity for detection and review of bloodstream infection cases

meets our standards – coding of CLABSI cases (if any) conforms to NHSN criteria and meets our

standards – any discordance between clinical (ICD-9 billing proxy) versus surveillance

(NHSN) coding can be reasonably explained

• We also estimate total CLABSI burden – in-scope vs. out of scope hospital-acquired infections – infections present on admission– total range spanning a minimum from the restrictive NHSN criteria to a maximum

from the more permissive ICD-9/clinical criteria using capture-recapture formulae

Technical aspects consistent with other industries

• We validate accuracy of the surveillance & reporting supply chain process, not “data”

• We view this as a quality control type problem, not an estimation type problem– More like HACCP than one-time studies…– Actively manages process quality, introducing

quality improvement when appropriate– Ensures hospitals maintain a set standard for

sensitivity & specificity that we defined

Device day check in data cleaning (not in validation visit)Brushed scatterplot matrix, highlighting highest number of

central line days

Definition of Technical Terms• Definition from A Dictionary of Epidemiology, 5th Edition

(International Epidemiological Association)– Validation – “The process of establishing that a method is sound.”– Data Cleaning – “The process of excluding the information in incomplete,

inconsistent records or irrelevant information…before analysis begins…”• Definitions from Quality Glossary, American Society for Quality,

available at http://asq.org/glossary/ – Validation – “The act of confirming a product or service meets the requirements

for which it was intended.”– Verification – “The act of determining whether products or services conform to

specific requirements.”– Acceptance sampling – “Inspection of a sample from a lot to decide whether to

accept that lot. There are two types: attributes sampling and variables sampling. In attributes sampling, the presence or absence of a characteristic is noted in each of the units inspected. In variables sampling, the numerical magnitude of a characteristic is measured and recorded for each inspected unit; this involves reference to a continuous scale of some kind.”

Definitions (cont.)– Acceptance sampling plan – “A specific plan that indicates the sampling sizes

and associated acceptance or nonacceptance criteria to be used. In attributes sampling, for example, there are single, double, multiple, sequential, chain and skip-lot sampling plans. In variables sampling, there are single, double and sequential sampling plans. For detailed descriptions of these plans, see the standard ANSI/ISO/ASQ A3534-2-1993: Statistics—Vocabulary and Symbols—Statistical Quality Control.”

– Audit – “The on-site verification activity, such as inspection or examination, of a process or quality system, to ensure compliance to requirements. An audit can apply to an entire organization or might be specific to a function, process or production step.” 

– Process – “A set of interrelated work activities characterized by a set of specific inputs and value added tasks that make up a procedure for a set of specific outputs.” 

Definitions (cont.)• Definitions from HF-2 Business Operating Systems (BOS) for

Health Care Organizations: Requirements for Process Improvements to Achieve Excellence. 2007, AIAG, Michigan– Validation – “objective confirmation that the specified requirements for a specific

use or application have been satisfied in the end-user environment”– Verification – “objective confirmation that the specified requirements have been

satisfied”

Immediate Priority Issue!

• Validate the Accuracy of What is Reported– Our industry too often uses a sad disclaimer

• “Resources for this audit were limited to a review of 200 patient records. Determination of an appropriate sample size is difficult and the sample size (202) is too small to draw statistically significant conclusions about the validity of CLABSI data reported to NHSN.” 

– Other industries use international standards• (ISO 2859, ANSI/ASQ Z1.4, MIL STD 105E)

Validation in Other Industries• Acceptance sampling uses statistical sampling to

determine whether to accept or reject a production lot of material. It has been a common quality control technique used in industry and particularly the military for contracts and procurement.– 1930s

• Dodge-Romig acceptance sampling tables– 1940s

• MIL STD 105A– 1990s

• MIL STD 105E cancelled, ANSI/ASQ Z1.4 in U.S.• ISO 2859 as international equivalent

Our Operating Characteristics Curves

• See our technical reference manual– Examples on the next two slides

• First slide is from “internal” component• Second slide is from “external” component’s

double-sampling plan (at “normal” inspection level)– 40 positive blood culture charts

» Read 20; pass or fail or read remaining 20– Also review all charts tagged with ICD9 code 999.31

– Reference manual available upon request

Sensitivity 85% Sensitivity 70% Sensitivity 60% Sensitivity 50% Sensitivity 30%

CasesIndividual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

0 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00039 1.00000

1 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00001 1.00000 0.00369 0.99961

2 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00006 0.99999 0.01659 0.99592

3 0.00000 0.00000 0.00000 1.00000 0.00001 1.00000 0.00037 0.99994 0.04740 0.979334 0.00000 0.00000 0.00000 1.00000 0.00007 0.99999 0.00174 0.99957 0.09649 0.93194

5 0.00000 0.00000 0.00001 1.00000 0.00035 0.99992 0.00628 0.99783 0.14886 0.83545

6 0.00000 0.00000 0.00004 0.99999 0.00150 0.99957 0.01779 0.99155 0.18076 0.68659

7 0.00000 0.00000 0.00020 0.99996 0.00513 0.99808 0.04066 0.97376 0.17707 0.50582

8 0.00000 0.00000 0.00088 0.99975 0.01442 0.99295 0.07624 0.93310 0.14229 0.328759 0.00000 0.00000 0.00320 0.99887 0.03364 0.97853 0.11859 0.85686 0.09486 0.18646

10 0.00002 0.00002 0.00971 0.99567 0.06560 0.94489 0.15417 0.73827 0.05285 0.09160

11 0.00010 0.00012 0.02471 0.98596 0.10734 0.87929 0.16819 0.58409 0.02471 0.03874

12 0.00053 0.00065 0.05285 0.96126 0.14760 0.77195 0.15417 0.41591 0.00971 0.0140413 0.00231 0.00296 0.09486 0.90840 0.17031 0.62435 0.11859 0.26173 0.00320 0.00433

14 0.00842 0.01139 0.14229 0.81354 0.16422 0.45405 0.07624 0.14314 0.00088 0.00113

15 0.02545 0.03684 0.17707 0.67125 0.13138 0.28982 0.04066 0.06690 0.00020 0.0002516 0.06311 0.09995 0.18076 0.49418 0.08622 0.15844 0.01779 0.02624 0.00004 0.00004

17 0.12621 0.22616 0.14886 0.31341 0.04564 0.07223 0.00628 0.00845 0.00001 0.00001

18 0.19866 0.42482 0.09649 0.16455 0.01902 0.02658 0.00174 0.00217 0.00000 0.00000

19 0.23700 0.66182 0.04740 0.06806 0.00601 0.00756 0.00037 0.00043 0.00000 0.00000

20 0.20145 0.86328 0.01659 0.02067 0.00135 0.00156 0.00006 0.00006 0.00000 0.0000021 0.10872 0.97200 0.00369 0.00408 0.00019 0.00021 0.00001 0.00001 0.00000 0.00000

22 0.02800 1.00000 0.00039 0.00039 0.00001 0.00001 0.00000 0.00000 0.00000 0.00000

0

0.05

0.1

0.15

0.2

0.25

0 2 4 6 8 10 12 14 16 18 20 22

Sensitivity 85%

Sensitivity 70%

Sensitivity 60%

Sensitivity 50%

Sensitivity 30%

Indi

vidu

alPr

obab

ility

Number correct out of 22

Future Plans

• Introduce SSI reporting (August 2012)– CMS requires abdominal hysterectomy & colon resection– State law requires abdominal & vaginal hysterectomy, total hip,

total knee & open chest cardiac– We hope state law can be amended in the next session to match

federal requirements

• Pilot test SSI validation (2012-2013)• Publish SSI 2013 annual rates (2014)

– Publish 2012 (partial year?) rates (2013)• Implement SSI annual validation (2014)

Future Plans (cont.)

• What we plan to report on public website– Questions patients can ask and practices to watch for– SSI rates for specified surgical procedures

• How we plan to display this information– Incidence rate by procedure (map, table & box plot formats)

• Rates shown separately for complex vs. superficial SSI– Whether SIR value is <1, =1 or >1

• But not display the actual SIR number

• Thoughts thus far on validation protocol– 100% compliance with surveillance practices– 95% sensitivity (+5/-10 tolerance) & 98% specificity– Same two component & ISO 2859 approach as used for CLABSI

Expect all SSI surveillance programs to include the following activities:

Reviewing in-patients with positive microbiology results from surgical site wound, abscess, aspirate or tissue, or blood cultures, post-operatively – at least once a week if not daily

Reviewing cases with an ICD-9 code suggestive of surgical site infection upon any readmissions – at least weekly to monthly (a post-discharge surveillance activity already present in 95% of facilities according to our survey*)

Communicating with ICP of other hospitals or with clinical lead at an ambulatory surgery facility when patients with surgical site infection coded as present on admission are admitted following surgery at another facility – at least weekly to monthly

Communicating with surgeons, primary care physicians, surgical patients, or a combination thereof to confirm post-operative status several weeks after discharge – at least monthly to quarterly (63% of facilities already exchange lists with surgeons, 18% telephone patients, 8% have patients return a post-card, according to our survey*)

*Zarate R, Birnbaum D. Postdischarge Surgical Site Infection Surveillance Practices in Washington Acute Care Hospitals. INFECT CONTROL HOSP EPIDEMIOL 2012;33(1):87-89.

Expect all SSI surveillance programs to include the following activities:

New thoughts in November 2012

Reviewing in-patients with positive microbiology results from surgical site wound, abscess, aspirate or tissue, or blood cultures, post-operatively – at least once a week if not daily Maybe also imaging re: abscess after colon surgery

Reviewing cases with an ICD-9 code suggestive of surgical site infection upon any readmissions – at least weekly to monthly (a post-discharge surveillance activity already present in 95% of facilities according to our survey*)

Review either ICD-9 & CPT codes (Harvard list), or NSQIP SSI cases instead

Communicating with ICP of other hospitals or with clinical lead at an ambulatory surgery facility when patients with surgical site infection coded as present on admission are admitted following surgery at another facility – at least weekly to monthly

Communicating with surgeons, primary care physicians, surgical patients, or a combination thereof to confirm post-operative status several weeks after discharge – at least monthly to quarterly (63% of facilities already exchange lists with surgeons, 18% telephone patients, 8% have patients return a post-card, according to our survey*) *Zarate R, Birnbaum D. Postdischarge Surgical Site Infection Surveillance Practices in Washington Acute Care Hospitals.

INFECT CONTROL HOSP EPIDEMIOL 2012;33(1):87-89.

The OC curve describes the probability of accepting a lot, OC(p), as a function of the proportion non-conforming (p), for the single and double sampling plans described above.

The ASN curve describes the average sample size when employing double sampling, as a function of the proportion of non-conforming items (p).

Sensitivity 95% Sensitivity 90% Sensitivity 85% Sensitivity 80% Sensitivity 70% Sensitivity 60%

CasesIndividual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

Individual Probability

Cumulative Probability

0 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000

1 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000

2 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000

3 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00001 1.00000

4 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00007 0.99999

5 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00001 1.00000 0.00035 0.99992

6 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00004 0.99999 0.00150 0.99957

7 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00000 1.00000 0.00020 0.99996 0.00513 0.99808

8 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00001 1.00000 0.00088 0.99975 0.01442 0.99295

9 0.00000 0.00000 0.00000 1.00000 0.00000 1.00000 0.00005 0.99999 0.00320 0.99887 0.03364 0.97853

10 0.00000 0.00000 0.00000 1.00000 0.00002 1.00000 0.00028 0.99994 0.00971 0.99567 0.06560 0.94489

11 0.00000 0.00000 0.00000 1.00000 0.00010 0.99998 0.00124 0.99965 0.02471 0.98596 0.10734 0.87929

12 0.00000 0.00000 0.00002 1.00000 0.00053 0.99988 0.00455 0.99841 0.05285 0.96126 0.14760 0.77195

13 0.00000 0.00000 0.00013 0.99998 0.00231 0.99935 0.01400 0.99386 0.09486 0.90840 0.17031 0.62435

14 0.00001 0.00001 0.00073 0.99985 0.00842 0.99704 0.03600 0.97986 0.14229 0.81354 0.16422 0.45405

15 0.00006 0.00007 0.00351 0.99912 0.02545 0.98861 0.07681 0.94386 0.17707 0.67125 0.13138 0.28982

16 0.00051 0.00058 0.01383 0.99561 0.06311 0.96316 0.13441 0.86705 0.18076 0.49418 0.08622 0.15844

17 0.00344 0.00402 0.04392 0.98178 0.12621 0.90005 0.18976 0.73264 0.14886 0.31341 0.04564 0.07223

18 0.01816 0.02218 0.10979 0.93787 0.19866 0.77384 0.21084 0.54288 0.09649 0.16455 0.01902 0.02658

19 0.07264 0.09482 0.20803 0.82807 0.23700 0.57518 0.17755 0.33204 0.04740 0.06806 0.00601 0.00756

20 0.20703 0.30185 0.28084 0.62004 0.20145 0.33818 0.10653 0.15449 0.01659 0.02067 0.00135 0.00156

21 0.37462 0.67647 0.24072 0.33920 0.10872 0.13672 0.04058 0.04796 0.00369 0.00408 0.00019 0.00021

22 0.32353 1.00000 0.09848 0.09848 0.02800 0.02800 0.00738 0.00738 0.00039 0.00039 0.00001 0.00001

Remaining Questions

1. What microbiology triggers to specify1. Specimen types & organisms to trigger case review

by ICP for possible SSI

2. What ICD-9 triggers to specify1. How many and which codes to trigger case review

by ICP for possible SSI

3. If/When to add post-op non-prophylactic antimicrobial usage as a trigger

4. Whether to assess process measures in validation (e.g. SCIP measure data)

Remaining Questions New thoughts in November 2012

1. What microbiology triggers to specify1. Specimen types & organisms to trigger case review by ICP for

possible SSIConsider Laboratory Messaging Community of Practice (LabM CoP) group’s documents to identify synonyms & electronic coding conventions

2. What ICD-9 triggers to specify1. How many and which codes to trigger case review by ICP for

possible SSIObtain list of ICD-9 & CPT codes from Harvard group

3. If/When to add post-op non-prophylactic antimicrobial usage as a trigger

4. Whether to assess process measures in validation (e.g. SCIP measure data)