False Discovery Rates for Discrete Data

Joseph F. Heyse Merck Research Laboratories

Graybill Conference

June 13, 2008

Graybill.ppt.

Introduction Almost all multiplicity considerations in clinical trial applications are

designed to control the Family Wise Error Rate (FWER).

Benjamini and Hochberg (1995) argued that in certain settings, requiring control of the FWER is often too conservative.

They suggested controlling the “False Discovery Rate” (FDR) is a more powerful alternative.

Accounting for the discrete endpoints can further improve the power of FDR (and FWER) methods.

Graybill.ppt.

Outline

1. Definition and properties of FDR

2. FDR for discrete data

3. Application: Genetic variants of HIV

4. Summary of simulation results

5. Application: Rodent carcinogenicity study

6. Concluding remarks

Graybill.ppt.

Familywise Error Rate (FWER)

Let F = {H1,H2 … HK} denote a family of K hypotheses.

FWER = Pr(any true Hi F is rejected).

The procedures currently used for clinical studies are intended to control the FWER .

Benjamini & Hochberg (1995) proposed controlling the “False Discovery Rate” (FDR) as a more powerful alternative to FWER.

Graybill.ppt.

Declared

Insignificant

Declared Significant

Total

# of true Hi U V K0

# of false Hi T S K K0

Total K R R K

False Discovery Rate (FDR)(Benjamini & Hochberg, 1995)

When R=0, FDR is defined to be 0.

rejected.y incorrectl are whichhypotheses

null rejected of proportion expected RV

EFDR

Graybill.ppt.

False Discovery Rate (FDR) (cont’d)

(Benjamini & Hochberg)

Kj

P if H,,H,H jectRe jj21

1Kj,PjK

,P~

minP~

PP~ :values-P Adjusted

j1jj

KK

Unadjusted p-values .0193 .0280 .2038 .4941

FDR-adjusted p-values .0560 .0560 .2718 .4941

Example (K=4)

Graybill.ppt.

Properties of FDR Control The B&H sequential procedure controls the FDR at

FDR < FWER and equality holds if K=K0.

The Hochberg (1988) stepwise procedure compares

while the FDR procedure compares

FDR is potentially more powerful than FWER controlling

procedures.

KK0

j-1K to P j

.Kj to P j

for independent hypotheses.

Graybill.ppt.

Comparing FDR and FWER

FDR adjusted P-values < FWER adjusted P-values

Example (K=4)

Unadjusted P-values

FDR-adjusted P-values

FWER-adjusted P-values

.0193

.0560

.0772

.0280

.0560

.0840

.2038

.2718

.4076

.4941

.4941

.4941

Graybill.ppt.

Modified FDR for Discrete Data

Adjusted P-values for FDR

For discrete data

Where is largest P-value achievable for hypothesis i that is less than or equal to P(j).

jj Pj

KP ][~

*iP

jPPK

iij

1

*][

~

Graybill.ppt.

FDR for Discrete Data

Gain in power for the discrete data FDR comes from the difference .

If endpoint i is not able to achieve a P-value ≤ P(j) then and the dimensionality is reduced.

If endpoint i is able to achieve a P-value ≤ P(j) then and a smaller quantity adds to .

*)( iPP j

0* iP

)(*

ji PP ][

~jP

Graybill.ppt.

Other Approaches for Discrete Data

Tarone (1990) proposed a modified Bonferroni procedure for discrete data by removing those endpoints unable to reach that level of statistical significance.

Gilbert (2005) proposed a 2 step FDR method for discrete data.

1. Apply Tarone’s method to identify endpoints suitable for adjustment.

2. Apply B-H FDR to those endpoints.

Calculating the FDR adjusted P-value is expected to improve upon these approaches by using the complete exact distribution.

][

~jP

Graybill.ppt.

Example: Genetic Variants of HIV

• Gilbert (2005) compared the mutation rates at 118 positions in HIV amino-acid sequences of 73 patients with subtype C to 73 patients with subtype B.

• The B-H FDR procedure identified 12 significant positions.

• The Tarone modified FDR procedure reduced the dimensionality to 25 and identified 15 significant positions.

• The fully discrete FDR identified 20 significant positions.

Graybill.ppt.

Simulation Study for Independent Hypotheses

A simulation study was conducted to evaluate the statistical properties of the FDR controlling methods for discrete data using Fisher’s Exact Test.

Simulation parameters– Number of Hypotheses: K = 5, 10, 15, 20

– Varying numbers of false hypotheses (K-K0)

– Background rates chosen randomly from U(.01, .5)– Odds Ratios for Effect Size: OR = 1.5, 2, 2.5, 3– Sample sizes: N = 10, 25, 50, 100 = 0.05 1-Tailed

Graybill.ppt.

Rate of Rejecting True Hypotheses When All Hypotheses are True

(K0=K)

5 10 15 20

0.01

0.02

0.03

0.04

0.05(N=25)

Discrete FDRT-G FDRB-H FDR

5 10 15 20

0.01

0.02

0.03

0.04

0.05(N=100)

No. of Hypotheses (K)

Graybill.ppt.

Rate of Rejecting True Hypotheses When Some Hypotheses are False

(K0<K)

1 2 3 4 5 6 7 8 9

0.01

0.02

0.03

0.04

0.05(K=10, N=25, OR=2.5)

Discrete FDRT-G FDRB-H FDR Reference Level

No. of True Hypotheses (K0)

1 2 3 4 5 6 7 8 9

0.01

0.02

0.03

0.04

0.05(K=10, N=100, OR=2.5)

Graybill.ppt.

Rate of Rejecting False Hypotheses

1.5 2.0 2.5 3.0

0.2

0.4

0.6

0.8

1.0(K=10, K-K0=4, N=25)

Discrete FDRT-G FDRB-H FDR

1.5 2.0 2.5 3.0

0.2

0.4

0.6

0.8

1.0(K=10, K-K0=4, N=100)

Odds Ratio

Graybill.ppt.

Other Applications

Analysis of Tier II clinical trial adverse experiences

Trend test analysis of rodent carcinogenicity data

Similar modification applied to Bonferonni adjustment for discrete endpoints

Graybill.ppt.

Rodent Carcinogenicity Studies

Long-term carcinogenicity studies typically test candidate drugs in several graded doses and use a vehicle control group.

50 male and 50 female rodents are randomly assigned to each drug treated group with 100 rodents of each sex assigned to control. Male and female studies are considered separately.

Treatment is administered daily and a terminal necropsy is performed at the end of the study.

Graybill.ppt.

Rodent Carcinogenicity Studies (cont’d)

Each individual tumor site encountered is described by a combination of organ or tissue with tumor type.

A statistical analysis of trend is performed for all tumor sites encountered.

An exact test uses the permutation distribution of the trend statistic.

Exact tests can account for age at necropsy and lethality of tumor.

For illustration purposes, this analysis only considered dose and presence of tumor.

Graybill.ppt.

Summary of Statistical Results from a Long-Term Carcinogenicity Study in Male Mice

Tumor Site

Liver, Hepatocellular CarcinomaP.S.U., HemangiosarcomaAdrenal Cortex, AdenomaP.S.U., SarcomaP.S.U., LymphomaLung, AdenomaLiver, Hepatocellular AdenomaLiver, HemangiosarcomaHarderian Gland, AdenomaSkin, FibromaThyroid, Follicular Cell CarcinomaP.S.U., LeukemiaLung, AdenocarcinomaTestes, Interstitial Cell TumorStomach, Papilloma

Number of Mice on Study

Control0

10007

1612

22115012

100

2

010048600002310

50

4

100016700112010

50

8

31116

11611002000

50

TrendP-value

.0342

.20

.20

.20

.24

.24

.49

.50

.50

.50

.50

.44N

.41N

.41N

.16N

Trend P-value is reported 1-tailed using exact permutational distribution. N indicates 1-tailed P-value for negative trend.

Test Agent Dose

Graybill.ppt.

Available Methods

Adjusting P-values to account for multiple tumor types. (Mantel 1980, Brown and Fears 1981, Mantel et al. 1982)

Adapting for interpreting unadjusted P-values (Haseman 1983 and 1990, Lin and Rahman 1998)

Resampling methods to adjust P-values (Heyse and Rom 1988, Westfall and Young 1989, Westfall and Soper 1998)

Bayesian methods using historical control priors (Westfall and Soper 2001)

Graybill.ppt.

Multiplicity of Statistical Tests

Liver, hepatocellular carcinoma was only 1 of K=15 tumor sites encountered.

P(1)=0.0342 was the most extreme individual trend P-

value.

Interest is in the likelihood of observing P(1)=0.0342 as

the most extreme P-value among the K=15 in this study.

Need to consider the discrete nature of the data since several tumor sites may not be able to achieve significance levels of P(1).

Graybill.ppt.

P-value Adjustment Methods

Mantel (1980) attributed to J.W. Tukey

Where k=number of tumor sites that could yield P-values as extreme as P(1).

Mantel et al. (1982)

Where is largest P-value achievable for tumor site i that is less than or equal to P(1).

Discrete FDR adjustment: = 0.264

268.0P11P k1]1[

*iP

235.0P11P *i

K

1i

*]1[

]1[P~

Graybill.ppt.

Conclusions

FDR control provides higher power than FWER control when some hypotheses are false.

Proposed procedure based on exact analysis of binomial data controls FDR at

Discrete nature of data results in slightly conservative FDR control. FDR is less conservative for increasing sample size and increasing numbers of hypotheses.

Accounting for discrete endpoints increases the power of FDR.