Shun Watanabe

Tokyo University of Agriculture ante Technology

Neyman-Pearson Test and Hoeffding Test

Beyond IID Workshop@SingaporeJuly, 2017

arXiv:1611.08175

(Binary) Hypothesis Testing

We shall discriminate

Zn = (Z1, . . . , Zn) ⇠ Pn Zn = (Z1, . . . , Zn) ⇠ Qnor

Acceptance region: accept P

type I error probability:

type II error probability:

An ✓ Zn

↵n = Pn(Acn)

�n = Qn(An)

Neyman-Pearson Test

Zn is accepted if

1

n

nX

i=1

⇤(Zi) > ⌧⇤(z) := log

P (z)

Q(z)for log-likelihood ratio (LLR)

[Neyman-Pearson 28]

Neyman-Pearson Test

Zn is accepted if

1

n

nX

i=1

⇤(Zi) > ⌧⇤(z) := log

P (z)

Q(z)for log-likelihood ratio (LLR)

Type I and Type II error trade-off:

[Neyman-Pearson 28]

↵n = P

✓1

n

nX

i=1

⇤(Zi) ⌧

◆�n = Q

✓1

n

nX

i=1

⇤(Zi) > ⌧

◆

Neyman-Pearson Test

Zn is accepted if

1

n

nX

i=1

⇤(Zi) > ⌧⇤(z) := log

P (z)

Q(z)for log-likelihood ratio (LLR)

Type I and Type II error trade-off:

Asymptotically

limn!1

↵n = 0for�n·= exp{�nD(PkQ)}

[Neyman-Pearson 28]

↵n = P

✓1

n

nX

i=1

⇤(Zi) ⌧

◆�n = Q

✓1

n

nX

i=1

⇤(Zi) > ⌧

◆

Neyman-Pearson Test

Zn is accepted if

1

n

nX

i=1

⇤(Zi) > ⌧⇤(z) := log

P (z)

Q(z)for log-likelihood ratio (LLR)

Type I and Type II error trade-off:

Asymptotically

limn!1

↵n = 0for

for ↵n·= exp{�nr}�n

·= exp

⇢� n min

P̃ :D(P̃kP )rD(

˜PkQ)

�

�n·= exp{�nD(PkQ)}

[Neyman-Pearson 28]

↵n = P

✓1

n

nX

i=1

⇤(Zi) ⌧

◆�n = Q

✓1

n

nX

i=1

⇤(Zi) > ⌧

◆

Hoeffding Test

Zn is accepted if type satisfies tZn

D(tZnkP ) < r

[Hoeffding 65]

Hoeffding Test

Zn is accepted if type satisfies

Type I and Type II error trade-off:

tZn

D(tZnkP ) < r

[Hoeffding 65]

↵n = P

✓D(tZnkP ) � r

◆�n = Q

✓D(tZnkP ) < r

◆

Hoeffding Test

Zn is accepted if type satisfies

Type I and Type II error trade-off:

Asymptotically

tZn

D(tZnkP ) < r

[Hoeffding 65]

↵n = P

✓D(tZnkP ) � r

◆�n = Q

✓D(tZnkP ) < r

◆

limn!1

↵n = 0for�n·= exp{�nD(PkQ)}

Hoeffding Test

Zn is accepted if type satisfies

Type I and Type II error trade-off:

Asymptotically

tZn

D(tZnkP ) < r

[Hoeffding 65]

↵n = P

✓D(tZnkP ) � r

◆�n = Q

✓D(tZnkP ) < r

◆

limn!1

↵n = 0for

for ↵n·= exp{�nr}�n

·= exp

⇢� n min

P̃ :D(P̃kP )rD(

˜PkQ)

�

�n·= exp{�nD(PkQ)}

Geometrical Interpretation of Two Tests

Neyman-Pearson test

P × × Q

M(τ)

M(⌧) =

(P̃ :

X

z

P̃ (z)⇤(z) = ⌧

)

E =

⇢˜P :

˜P (z) / exp{s⇤(z)}, s 2 R�

[Csiszár, Amari-Nagaoka, etc.]

Geometrical Interpretation of Two Tests

Neyman-Pearson test

P × × Q

M(τ)

M(⌧) =

(P̃ :

X

z

P̃ (z)⇤(z) = ⌧

)

E =

⇢˜P :

˜P (z) / exp{s⇤(z)}, s 2 R�

P1

P2P3

D(P1kP3) = D(P1kP2) +D(P2kP3)

[Csiszár, Amari-Nagaoka, etc.]

Geometrical Interpretation of Two Tests

Neyman-Pearson test

Hoeffding test

P × × Q

M(τ)

M(⌧) =

(P̃ :

X

z

P̃ (z)⇤(z) = ⌧

)

E =

⇢˜P :

˜P (z) / exp{s⇤(z)}, s 2 R�

P1

P2P3

D(P1kP3) = D(P1kP2) +D(P2kP3)

[Csiszár, Amari-Nagaoka, etc.]

P × × Q

⇢P̃ : D(P̃kP ) < r

�

Finite Length PerformanceNeyman-Pearson test/Hoeffding test:

↵n

�n

Second-Order PerformanceNeyman-Pearson test

[Strassen 62]For non-vanishing type I error

limn!1

↵n "

type II exponent is

� log �n = nD(PkQ)�p

nV (PkQ)Q�1(") +O(log n)

V (PkQ) :=

X

x

P (x)

✓log

P (x)

Q(x)

�D(PkQ)

◆2

Q(a) :=

Z 1

a

1p2⇡

et2

2 dt

Second-Order PerformanceHoeffding test

It is known that asymptotically [Wilks 37],

2nD(tZnkP ) ⇠ �2|Z|�1 chi square distribution of degree |Z|� 1

Second-Order PerformanceHoeffding test

For non-vanishing type I error

limn!1

↵n "

type II exponent is

It is known that asymptotically [Wilks 37],

2nD(tZnkP ) ⇠ �2|Z|�1 chi square distribution of degree |Z|� 1

� log �n = nD(PkQ)�qnV (PkQ)Q�1

�2,|Z|�1(") +O(log n)

Q�2,|Z|�1(a) :=

Z 1

a�2|Z|�1(t)dt

Note thatq

Q�1�2,|Z|�1(") > Q�1(")

Summary of Standard Hypothesis Testing

StandardHypothesis

Testing

Hoeffding Test

• LLR• mixture plane

Neyman-Pearson Test

• divergence• divergence sphere

Multiterminal Hypothesis Testing

PXY or QXY

Xn

Y n

f1

f2

g

[Berger 79]M1

M2

Consider zero-rate case: lim

n!1

1

nlog kMik = 0

eg) send marginal types

Multiterminal Hypothesis Testing

PXY or QXY

Xn

Y n

f1

f2

g

[Berger 79]M1

M2

Consider zero-rate case: lim

n!1

1

nlog kMik = 0

eg) send marginal types

Tests for the standard hypothesis testing cannot be used…

Multiterminal Hypothesis Testing

PXY or QXY

Xn

Y n

f1

f2

g

[Berger 79]M1

M2

Consider zero-rate case: lim

n!1

1

nlog kMik = 0

eg) send marginal types

For example, we can consider the following test:

tXn 2 T nX,� and tY n 2 T n

Y,� =) accept PXY

Tests for the standard hypothesis testing cannot be used…

Multiterminal Hypothesis Testing

PXY or QXY

Xn

Y n

f1

f2

g

[Berger 79]M1

M2

Consider zero-rate case: lim

n!1

1

nlog kMik = 0

eg) send marginal types

Proposition [Han 87, Shalaby-Papamarcou 92]

for limn!1

↵n = 0�n·= exp{�nE(PXY kQXY )}

E(PXY kQXY ) := minP̃XY :

P̃X=PX,P̃Y =PY

D(P̃XY kQXY )

Geometrical Interpretation[Amari-Han 89]

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

M(P ) =

⇢P̃XY : P̃X = PX , P̃Y = PY

�

E(Q) =

⇢˜

QXY : 9 a, b, log

˜

QXY (x, y)

QXY (x, y)= a(x) + b(y)

�

M(Q) =

⇢Q̃XY : Q̃X = QX , Q̃Y = QY

�

E(P ) =

⇢˜

PXY : 9a, b, log

˜

PXY (x, y)

PXY (x, y)= a(x) + b(y)

�

Geometrical Interpretation[Amari-Han 89]

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

M(P ) =

⇢P̃XY : P̃X = PX , P̃Y = PY

�

E(Q) =

⇢˜

QXY : 9 a, b, log

˜

QXY (x, y)

QXY (x, y)= a(x) + b(y)

�

D(PXY kQXY ) = D(PXY kP ⇤XY ) +D(P ⇤

XY kQXY )

M(Q) =

⇢Q̃XY : Q̃X = QX , Q̃Y = QY

�

= E(PXY kQXY )

E(P ) =

⇢˜

PXY : 9a, b, log

˜

PXY (x, y)

PXY (x, y)= a(x) + b(y)

�

Hoeffding-like Test

is accepted if satisfies[Han-Kobayashi 89]

(tXn , tY n)

E(tXn ⇥ tY nkPXY ) < r

PXY

Hoeffding-like Test

is accepted if satisfies

Type I and Type II error trade-off:

[Han-Kobayashi 89](tXn , tY n)

E(tXn ⇥ tY nkPXY ) < r

Note thatE(tXn ⇥ tY nkPXY ) = E(tXnY nkPXY )

�n = Q

✓E(tXnY nkPXY ) < r

◆↵n = P

✓E(tXnY nkPXY ) � r

◆

PXY

Hoeffding-like Test

is accepted if satisfies

Type I and Type II error trade-off:

Asymptotically

for ↵n·= exp{�nr}

[Han-Kobayashi 89](tXn , tY n)

E(tXn ⇥ tY nkPXY ) < r

Note thatE(tXn ⇥ tY nkPXY ) = E(tXnY nkPXY )

�n·= exp

⇢� n min

E(Q̃XY kQXY )rD(

˜QXY kQXY )

�

which is optimal among zero-rate schemes [Han-Amari 98].

�n = Q

✓E(tXnY nkPXY ) < r

◆↵n = P

✓E(tXnY nkPXY ) � r

◆

PXY

Hoeffding-like Test

PXY

×

××

P ∗

XY

QXY

E(P )

E(Q)

M(P )

⇢Q̃XY : E(Q̃XY kPXY ) < r

�

Neyman-Pearson Test for Multiterminal?

Standard Hypothesis

Testing

MultiterminalHypothesis

Testing

Hoeffding Test

• LLR• mixture plane

Neyman-Pearson Test

• divergence• divergence sphere

Hoeffding-like Test

• projected divergence• divergence cylinder

How to define a proxy of LLRTheorem [SW 16]

For ,

there exists a unique satisfying (P�XY , Q

�XY ) 2 E(P )⇥ E(Q)

�E(QXY kPXY ) < � < E(PXY kQXY )

D(Q�XY kQXY )�D(P�

XY kPXY ) = �

X

x

P

�

XY

(x, y) =X

x

Q

�

XY

(x, y)

X

y

P

�XY (x, y) =

X

y

Q

�XY (x, y)

log

P

�XY (x, y)

PXY (x, y)= c1 log

Q

�XY (x, y)

QXY (x, y)+ c2

for some .c1, c2 2 R\{0}

How to define a proxy of LLR

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

P̃XY

Q̃XY

(P̃X , P̃Y ) 6= (Q̃X , Q̃Y )

How to define a proxy of LLR

P̃XY

Q̃XY

(P̃X , P̃Y ) 6= (Q̃X , Q̃Y )

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

How to define a proxy of LLR

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

P�XY

Q�XY

(P�X , P�

Y ) = (Q�X , Q�

Y )

How to define a proxy of LLR

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY

P�XY

Q�XY

(P�X , P�

Y ) = (Q�X , Q�

Y )

log

P

�XY (x, y)

PXY (x, y)= c1 log

Q

�XY (x, y)

QXY (x, y)+ c2

Parallel condition:

How to define a proxy of LLRFor , define�E(QXY kPXY ) < � < E(PXY kQXY )

⇤�(x, y) := log

Q

�XY (x, y)

QXY (x, y)� log

P

�XY (x, y)

PXY (x, y)

For , define� = E(PXY kQXY )

� = �E(QXY kPXY )

⇤�(x, y) := log

P

⇤XY (x, y)

QXY (x, y)

For , define

⇤�(x, y) := log

PXY (x, y)

Q

⇤XY (x, y)

Neyman-Pearson-like Test

X

x,y

tX

n(x)tY

n(y)⇤�

(x, y) > ⌧

is accepted if satisfies(tXn , tY n)PXY

Neyman-Pearson-like Test

Type I and Type II error trade-off:

Note that implies

X

x,y

tX

n(x)tY

n(y)⇤�

(x, y) > ⌧

(P�XY , Q

�XY ) 2 E(P )⇥ E(Q)

⇤�(x, y) = a(x) + b(y)

for some , which implies a(x), b(x)

X

x,y

tX

n(x)tY

n(y)⇤�

(x, y) =X

x,y

tX

nY

n(x, y)⇤�

(x, y)

↵n = P

✓1

n

nX

i=1

⇤�(Xi, Yi) ⌧

◆�n = Q

✓1

n

nX

i=1

⇤�(Xi, Yi) > ⌧

◆

is accepted if satisfies(tXn , tY n)PXY

Numerical ExamplePXY =

1/2 1/81/8 1/4

�, QXY =

1/8 1/41/2 1/8

�For with , n = 100

Neyman-Pearson-like test outperform Hoeffding-like test:

↵n

�n

Second-Order PerformanceNeyman-Pearson-like test

For limn!1

↵n "

type II exponent is

� log �n = nE(PXY kQXY )�pnV Q�1

(") +O(log n)

Hoeffding-like test

type II exponent is

� log �n = nE(PXY kQXY )�qnV Q�1

�2,k(") +O(log n)

k = (|X |� 1) + (|Y|� 1)

V :=

X

x,y

P

XY

(x, y)

✓log

P

⇤XY

(x, y)

Q

XY

(x, y)

� E(P

XY

kQXY

)

◆2

Large Deviation PerformanceTheorem

Neyman-Pearson-like test with achieve the optimal LDP performance: � = ⌧

for ↵n·= exp{�nr}�n

·= exp

⇢� n min

E(Q̃XY kQXY )rD(

˜QXY kQXY )

�

Adjustment of depending on is very important.� ⌧

� = �E(QXY kPXY )

� = E(PXY kQXY )

� = ⌧ (optimal)

Type I exponent

Type II exponent

Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing

Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing

P × × Q

M(τ)

Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing

P × × Q

M(τ)

P × × Q

Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing

P × × Q

M(τ)

P × × Q

PXY

×

××

P ∗

XY

QXY

E(P )

E(Q)

M(P )

Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing

P × × Q

M(τ)

P × × Q

PXY

×

××

P ∗

XY

QXY

E(P )

E(Q)

M(P )

PXY

×

× ×

P ∗

XY

QXY

E(P )

E(Q)

M(P )

×

M(Q)

Q∗

XY