Results from the LA rapid testing study: What can they tell us about proposed point of care

strategies?

Kevin Delaney, MPHDivision of HIV/AIDS Prevention

Behavioral and Clinical Surveillance Branch

Diagnostics Applications Team

Thank You (So Far…)LA County STD

Peter KerndtApurva UniyalStaff at LAGLC/Altamed

DHAP/ODBernie Branson

BCSBSteve EtheridgeDawn GnesdaDuncan MackellarPatrick Sullivan

QSDM Debra Hansen

Serum BankDollene Hemmerlein

Lab BranchDebra CandalTrudy DobbsTom FolksTim GranadeSusan KennedySteve McDougalMichelle Owen

Outline

• Original study methods

• Problematic specimens

• Comparing strategies

• Why prevalence matters

Original Study Objectives• Describe the performance of individual rapid tests

– Sensitivity, specificity and predictive value

• Evaluate the performance of combinations of rapid tests– Particular focus on tests designed for use at point of

care

• Foster discussion of a point of care diagnostic algorithm

• Testing – Specimens from participants were tested with all

6 FDA approved rapid tests– Serum also tested with EIA and Western blot:

• Vironostika HIV-1 Microelisa system • Genetic Systems HIV-1 Western Blot

– Samples of both serum and plasma retained for future testing

• Tested at CDC with both Genprobe Aptima and Biorad HIV1/2 + O EIA

Original Study Methods

• Analysis– Sensitivity, specificity and predictive value for

individual tests and combinations used in POC strategies:

• Performed in Sequence• First two tests performed in Parallel• Second and third must agree to be considered

positive• Proposed strategies 1, 2, 3 and 4

Original Study Methods

Line List of “Multiply Discordant” Specimens (n=8) – Including CDC results

ID LA res

Aptima Biorad OQ-o OQ-b Sp-b Sp-pl UG-b Ug-pl Co-b MS-pl Re-pl

1 N P P X 3 N N N N P N P N

4 N N N P P N N N N P N N

6 N N N N N N P N N N P N

9 N N N N N N N N N N P P

10 P N P X 3 N N N N N P P N

11 P P P X 3 N P P P N N P P

13 P P P X 3 P P P N P N P N

15 P P P X 3 P P P N P N P P

Line List of Western blot IND Specimens (n=6) – Including CDC results

ID Aptima Biorad OQ-o OQ-b Sp-b Sp-pl UG-b Ug-pl Co-b MS-pl Re-pl

16 N N N N N N N N P N

17 N N P P N N P P P P P

18 P P X 3 N N N N N N N N

19 P P X 3 N N N N N N N N

20 N P X 3 P P P P P P P P

21 P P X 3 P P P P P P P P

Comparing Strategies• Bootstrap resampling

– Random selection of individuals with replacement– Random selection of a test combination– 1100 theoretical combinations of test and specimen type– Excluded:

• Combinations of the same test used on the same (n=290) • Or different (n=200) specimen types • Combinations that used the two Clearview tests together (n=152) • Combinations with a CLIA moderate complexity test followed by a

CLIA-waived test (n=232) • Combinations in which Oraquick performed on oral fluid was used

as the second or third test (n=21)

– 51 two-test and 154 three-test potential combinations

Comparing Strategies• Bootstrap resampling

– Includes variability introduced by individual test combinations

• e.g. starting with OMT vs. starting with blood • “bad batch” of tests with low specificity

– and problematic specimens • e.g. likely specimen mix• “specific non-specific” reactions with antigens

– Medians and bootstrap confidence intervals reported

Comparing Strategies

• Diagnostic likelihood– The “relative risk” of being infected given that

you test:• Positive (PLHR) = sensitivity/(1-specificity)• Negative (NLHR) = (1-sensitivity)/specificity • Good test performance PLHR > 10 and NLHR < .1• Combining with prevalence will give Positive

(PLHR) or Negative (NLHR) predictive value

Compare algorithms – Diagnostic Likelihoods = Remember these tests are GOOD!scenario

Nlhr2.5%

Nlhr50%

Nlhr97.5%

Plhr2.5%

Plhr50%

Plhr97.5%

AUC2.5%

AUC50%

AUC97.5%

one test (Strategy 1)

0 0.013288 0.030633 390 777 3870 0.98360 0.99320 0.99949

Two different Rapids (Strategy 2)

0 0.014778 0.036810 3774 Inf. Inf. 0.98193 0.99268 1.00000

Two Oraquicks (Strategy 3)

.004149864 0.017471 0.037450 1385 4226 Inf. 0.98121 0.99119 0.99784

Strategy 4+ (allow +,-,- = not infected)

0 0.014634 0.034216 3341 Inf. Inf. 0.98312 0.99272 1.00000

Parallel screening

0 0.006536 0.024004 3074 Inf. Inf. 0.99018 0.99733 1.00000

3 Tests must agree

0 0.016393 0.038674 Inf. Inf. Inf. 0.98054 0.99171 1.00000

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Tw oRapid

one test

paraalg

seqalg

tw o oq

w hoseq

0.97

0.985

1

0 0.015 0.03

Tw oRapid

one test

paraalg

seqalg

tw o oq

w hoseq

Sensitivity

1-Specificity

ROC Curves for various POC Algorithms

Compare algorithms – Predictive Value (prevalence ≈ 5%)

scenarioPpv2.5%

Ppv50%

Ppv97.5%

Npv2.5%

Npv50%

Npv97.5%

one test (Strategy 1)

0.95181 0.97566 0.99490 0.99851 0.99925 1.00000

Two different Rapids (Strategy 2)

0.99338 1.00000 1.00000 0.99850 0.99926 1.00000

Two Oraquicks (Strategy 3)

0.98502 0.99548 1.00000 0.99819 0.99909 0.99977

Strategy 4+ (allow +,-,- = not infected)

0.99402 1.00000 1.00000 0.99808 0.99915 1.00000

Parallel screening 0.99260 1.00000 1.00000 0.99876 0.99968 1.00000

3 Tests must agree

1.00000 1.00000 1.00000 0.99809 0.99916 1.00000

Compare algorithms – Further testing; Incorrect results

scenarioInd

2.5%Ind50%

Ind97.5%

Fn2.5%

Fn50%

Fn97.5%

Fp2.5%

Fp50%

Fp97.5%

one test (Strategy 1)

0 0 0 0 3 6 1 5 10

Two different Rapids (Strategy 2)

0 2 10 0 3 6 0 0 1

Two Oraquicks (Strategy 3)

0 1 3 1 4 8 0 1 3

Strategy 4+ (allow +,-,- = not infected)

0 0 0 0 3 7 0 0 1

Parallel screening

0 0 0 0 1 4 0 0 1

3 Tests must agree

1 4 11 0 3 7 0 0 0

Remember, these numbers won’t change for a given number of tests

0.4

0.5

0.6

0.7

0.8

0.9

1.0

p

5% 4% 3% 2% 1% 0%

Prevalence affects predictive value, if you have false positives…

Predictive value positive – Single test

Prevalence

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

p

5% 4% 3% 2% 1% 0%

The percentage of specimens going to the lab (red line) will increase as prevalence goes down

Predictive value positive – Single test

Percentage of tests that will require lab follow-up in strategies 2 and 4

Prevalence

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

p

5% 4% 3% 2% 1% 0%

In the worst case (Positive diagnostic likelihood ~ 400) @ .1% prevalence 70% of specimens would still require laboratory follow-up

Predictive value positive – Single test

Percentage of tests that will require lab follow-up in strategies 2 and 4

Prevalence

Compare algorithms – Further testing; Incorrect results

scenarioInd

2.5%Ind50%

Ind97.5%

Fn2.5%

Fn50%

Fn97.5%

Fp2.5%

Fp50%

Fp97.5%

one test (Strategy 1)

0 0 0 0 3 6 1 5 10

Two different Rapids (Strategy 2)

0 2 10 0 3 6 0 0 1

Two Oraquicks (Strategy 3)

0 1 3 1 4 8 0 1 3

Strategy 4+ (allow +,-,- = not infected)

0 0 0 0 3 7 0 0 1

Parallel screening

0 0 0 0 1 4 0 0 1

3 Tests must agree

1 4 11 0 3 7 0 0 0

But in that case that’s still less than 20 specimens per 4000 tested…

• Specificity of a single rapid test >99%– At low prevalence, the proportion of positive tests that

are false positive increases.– Adding a second test resolved nearly all false positives

correctly– There will likely always be some problematic specimens

no testing strategy can fix

Summary

• All testing strategies perform exceptionally well– In low prevalence settings a POC strategy that resolves

all false positives without lab intervention may be necessary

– In high prevalence settings the sensitivity of the first test and the option to test for acute infection should be considered

Summary

Results – Sensitivity of a single testTest Specimen type True

PosFalse neg

Sensitivity (95% CI)

Oraquick Oral fluid 280 5 98.2 (95.9-99.4)

Whole blood 277 1 99.6 (98.0-100)

Unigold Whole blood 173 5 97.2 (93.6-99.1)Plasma 247 8 96.8 (93.7-98.6)

Complete Whole blood 213 0 100 (98.6-100)

Stat-pak Whole blood 246 2 99.2 (97.1-99.9)

Plasma 246 4 98.4 (95.9-99.6)

Multispot Plasma 252 1 99.6 (97.8-100)

Reveal Plasma 278 3 98.9 (96.8-99.8)

Results – Specificity of a single testTest Specimen type True

NegFalse Pos

Specificity (95% CI)

Oraquick Oral fluid 5473 2 100 (99.9-100 )

Whole blood 5466 1 100 (99.9-100 )

Unigold Whole blood 3591 2 99.9 (99.8-100 )Plasma 4747 2 100 (99.9-100 )

Complete Whole blood 4202 2 99.9 (99.8-100 )

Stat-pak Whole blood 4771 1 100 (99.9-100 )

Plasma 4763 2 100 (99.9-100 )

Multispot Plasma 4835 33 99.3 (99.0-99.5)

Reveal Plasma 5478 6 99.9 (99.8-100 )

• Location – Two clinics in Los Angeles, CA

• Participants– Persons of unknown serostatus seeking HIV

testing– Persons known to be HIV positive

Original Study Methods