Discussion: Sun & Cai, JASA 2007 - Duke University Pura_Rules_for_False...Discussion: Sun & Cai,...

Discussion: Sun & Cai, JASA 2007

John PuraBIOS900

September 30, 2016

John Pura BIOS900 Discussion: Sun & Cai, JASA 2007

Hypothesis Testing

Well developed in single test scenario (e.g. Neyman-PearsonLemma)Goal: maximize power (minimize Type II error) whilecontrolling Type I errorProcedure (frequentist): Compute test statistic (e.g. likelihoodratio) and compare to a single critical value, while maintainingType I error


Multiple Hypothesis Testing

Controlling Type I error more complicated when we have manytestsCommon problem in many big-data applications - e.g. imaging,genomics, Netflix: number of tests on the order of 103-106

Goal: separate the null from the nonnulls while still balancingType I and Type II errorsPower is critical in these applications because the mostinteresting effects are usually at the edge of detection.Rich literature on the topic spanning frequentist and Bayesianapproaches


Data Structures in Large-Scale Studies

m features (e.g. voxels, genes, movies) and k groups

Table 1: Common Data Matrix in Large Scale Studies

X1 · · · Xn1 X2 · · · Xn2 · · · Xk · · · Xnk

X11 · · · Xn11 X21 · · · Xn21 · · · Xk1 · · · Xnk1X12 · · · Xn12 X22 · · · Xn22 · · · Xk2 · · · Xnk2

......

......

X1m · · · Xn1m X2m · · · Xn2m · · · Xkm · · · Xnkm


Multiple Testing Framework

Suppose we perform m simultaneous hypothesis tests of H0,i vs.H1,i , 1 ≤ i ≤ m under a given procedureSummarize the outcomes of such a procedure as follows:

Table 2: Classification of tested hypotheses

Claimed nonsignificant Claimed significant TotalNull N00 N10 m0

Nonnull N01 N11 m1Total S R m


Type I Error Control

Different ways of addressing Type I errorPer comparison error rate (PCER) =E (N10/m) = Pr(N10,i > 0) ≤ α, 1 ≤ i ≤ mFamilywise error rate (FWER) = Pr(N10 > 0) = 1− (1− α)m

False-discovery rate (FDR) = E (N10/R|R > 0)Pr(R > 0) andits variants. . .


Some Notation

α = prespecified level, 0 < α < 1Pi=p-value for testing H0,i vs. H1,i , 1 ≤ i ≤ m


Classical Procedures to Type I Error Control

FWERBonferroni

single step procedureReject H0,i if Pi ≤ α/mguarantees FWER ≤ α

Holm (1979)stepwise procedureOrder p-values: 0 < P(1) ≤ P(2) ≤ ... ≤ P(m)Reject H0,i if P(i) ≤ α/(m − i + 1)more powerful than Bonferroni


Classical Procedures to Type I Error Control

FDRBenjamini-Hochberg (1995)

For m independent tests, of which m0 are true, order p-valuesin decreasing order: P(m) ≥ P(m−1) ≥ ... ≥ P(1) > 0For a fixed α ∈ (0, 1), find k = maxi{P(i) ≤ m0

m α}Reject H0,(i) for all i ≤ kguarantees FDR ≤ αmore powerful than FWER control procedures (greater numberof rejections)equivalent to FWER-control when all m null hypotheses aretrue (m0 = m)works well for sparse situations (m0 ≈ m)


Additional Notation

Data: Xi |θi ∼ (1− θi )F0 + θiF1

Target of inference: θ1, ...θmindep∼ Bern(p)

null: θi = 0, nonnull: θi = 1

Test statistic: T(x)=[Ti(x): 1 ≤ i ≤ m]Z-values, Zi = Φ−1(F (Ti )) and corresponding p-values, Pi ,1 ≤ i ≤ mDecision rule: δ = (δ1, ..., δm) ∈ I = {0, 1}mmFDR = E (N01)/E (R), mFNR = E (N01)/E (S)p = Pr(θi = 1) 1 ≤ i ≤ m


Sun and Cai’s Approach

Compound decision view of multiple testing problem:Find δ that minimizes mFNR (efficient) while controlling mFDRat level α (valid) among all SMLR decision rules

Procedures based on thresholding ordered p-values are notgenerally efficient (e.g.δ(p(·), c) = {I[p(Xi ) < c] : 1 ≤ i ≤ m})Remark 1. Copas (1974): if a symmetric (akapermutation-invariant) classification rule for dichotomies isadmissible in a weighted classification problem, then it must beordered by the likelihood ratios


Sun and Cai’s Approach

Spoiler Alert!Optimal solution to multiple testing problem equivalent tooptimal solution to weighted classification problem amongsymmetric/permutation-invariant decision rulesThe local FDR (Lfdr) is a fundamental quantity which can beused directly for optimal FDR control.An adaptive Lfdr procedure based on z-values unifies the goalsof global error control and individual case interpretation


Oracle Setting - Example 1

Suppose we have m tests of H0,i : µ = 0 vs. H1,i : µ 6= 0,1 ≤ i ≤ m with corresponding z-values z1, ..., zm ∼ f (z),f (z) = p0φ(z) + p1φ(z − µ1) + p2φ(z − µ2)p0 = 0.8, p1 + p2 = 0.2, mFDR = 0.1Compare the oracle procedure (more on this in a bit) based onp-value (Genovese & Wasserman, 2002) to one based onz-value


Oracle Setting - Example 1


Remarks on Example 1

p-value oracle procedure (Disclaimer: for two-sidedalternatives) is dominated by z-value oracle procedure, withgreater difference in performance as p1 > p2.The Lfdr oracle procedure based on z-values may accept moreextreme (i.e. larger absolute z-value or equivalently lowerp-value) observations while rejecting less extreme ones =⇒rejection region is asymmetric

z = −2 (Lfdr = 0.227, p-value = 0.046) is rejected, but z = 3(Lfdr = 0.543, p-value = 0.003) is aceptedThis is due to the inherent properties of the Lfdr statistic


Local FDR (Lfdr)

Inherently Bayesian ideaLfdr = Pr(null |z) = (1− p)f0(z)/f (z)Allows for local inference, in which outcomes such as z = 3 are“judged on their own terms and not with respect tohypothetical possibility of more extreme results” (Efron, 2010)

In other words, our interest in an outlying zi is judged relativeto the local feature f0/f about zi , rather than relative tofeatures of the theoretical null distribution. See Remark 4.


Oracle Procedure

What is the optimal procedure and how does it behave if weknow p, f0, and f1?Assume SMLR property for a given test statistic. SeeProposition 1. Why is this useful?The optimal solution to the classification problem is alsooptimal in the multiple testing problem. See Theorem 1.Oracle rule:δλ(Λ, 1/λ) = [I{Λ(xi ) < 1/λ}, 1 ≤ i ≤ m] = Lfdr(·)

1−Lfdr(·) , whereΛ(·) = (1−p)f0(·)

pf1(·)

Bayes rule that minimizes the classification risk (alwayssymmetric, admissible, and ordered)


Oracle Procedure based on z-values

Test statistic: TOR(Zi ) = (1− p)f0(Zi )/f (Zi ) ∼ GOR , whereGOR = (1− p)G0

OR + pG1OR

mFDR = (1− p)G0OR/GOR

Threshold: λOR = supt∈(0,1){QOR(t) ≤ α}Testing rule: δ(TOR , λOR) = [I{TOR(zi ) < λOR}, 1 ≤ i ≤ m]The Lfdr oracle procedure ranks the relative importance of thetest statistics according to their likelihood ratios

Rankings generally different from p-values, unless alternativedistribution is symmetric about null hypothesis


Another Example


Remarks - Example 2

When |µ1| = |µ2|, mFNR is constant for p-value oracleprocedure; equivalence only under symmetric alternative (a)Lfdr oracle procedure utilizes the distributional information ofthe nonnulls (i.e. adapts to asymmetry), but the p-value oracleprocedure does not (a)Under large degree of asymmetry in alternative distribution,oracle procedures differ the greatest when |µ1| < µ2 andp1 > p2 (b,c)mFNR decreases with mFDR for a given range (d)


Adaptive Procedure

Suppose we have consistent estimators p̂0, f̂0, f̂ (e.g. Jin & Cai,2007)Define T̂OR(zi ) = min([(1− p̂)f̂0(zi )/f̂ (zi )], 1)Order the T̂OR(zi ) as ˆLfdr (1) ≤ ... ≤ ˆLfdr (m)

Let k = max1≤i≤m{1i

∑ij=1 ˆLfdr (j) ≤ α}

Reject H0,(i) for all i ≤ k ⇐⇒δ(T̂OR , λ̂OR) = [I{T̂OR(zi ) < λ̂OR}]Asymptotically efficient: asymptotically controls mFDR at levelα and attains the mFNR level achieved by the Lfdr oracleprocedure. See Theorems 4 & 5.


Yet, More Examples


Conclusion

The adaptive Lfdr procedure based on z-values is asymptoticallyoptimal, particularly under asymmetric alternative distributionThe Lfdr can be viewed as ranking the relative importance ofthe LRs, allowing for local inference and efficient separation ofnulls and nonnulls


Discussion: Sun & Cai, JASA 2007 - Duke University Pura_Rules_for_False...Discussion: Sun & Cai,...

Documents

Transcript of Discussion: Sun & Cai, JASA 2007 - Duke University Pura_Rules_for_False...Discussion: Sun & Cai,...