Distinguishability of Hypotheses

S.Bityukov (IHEP,Protvino; INR RAS, Moscow)N.Krasnikov (INR RAS, Moscow)

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Institute for High Energy Physics, Protvino, Russia

Institute for Nuclear Research RAS, Moscow, Russia

Planned experiment will give information about : (a) existence of new phenomenon (yes or no),

(b) magnitude and accuracy of measured value.We consider the case (a) from the frequentist point of view,

namely, we will calculate the total amount of possible cases and, after that, the amount of cases in favor of one of the statements.

This approach allows to estimate the probability of making a correct decision and, correspondingly, the quality of planned experiment by using the hypotheses testing. Also we can introduce the conception of the distinguishability of hypotheses.

Concept Concept

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Distinguishability of hypotheses: What is it ? Let us consider the planned experiment for searching for new phenomenon.

Hypotheses testingHypotheses testing

Hypothesis Hypothesis Hypothesis H0 : new physics is present in Nature : new physics is absentHypothesis H1

= P(reject H0 | H0 is true) -- Type I error = P(accept H0 | H0 is false) -- Type II error

is a significance of the test 1- is a power of the test

Note that and can be considered as random variable. If H0 is true then takes place and is absent. If H1 is true then is absent and takes place.

Many researchers prefer to exchange the places of these hypotheses

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

What to test ? What to test ?

Suppose the probability of the realization of n events in the experiment is described by function f(n; ) with parameter

Expected number of signal events in experiment is s Expected number of background events is b

Hypothesis H0 corresponds to = s + b , i.e. f(n; s + b) H1 corresponds to = b , i.e. f(n; b)

The Type I error and the Type II error allows to estimate the probability of making a correct decision when testing H0 versus H1 with an equal-tailed test and to estimate the distinguishability of these hypotheses H0 and H1 with an equal probability test.

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

The probability of making a correct decision in hypotheses The probability of making a correct decision in hypotheses

testingtesting

where the estimator

Let us consider the random variable = +

the case of applying the equal-tailed test

,^

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

2

^^^ is a constant term and

2

^^

is a stochastic

term. ^^

, are the estimators of Type I () and Type II () errors. In^^

the stochastic term isequal to 0 independently of whether H0 or H1 is true. Hence the estimator

^

can be named the probability of making incorrect choice in favor

of one of the hypotheses. Correspondingly,^

1 is the probability to

make a correct decision in hypotheses testing.

Here

: advantages and disadvantages: advantages and disadvantages Advantages +1° This probability is independent of whether H0 or H1 is true +2° In the case of discrete distributions the error of this estimator can be taken into account +3° This is an estimator of quality of planned experiment … However, the probability of making a correct decision has disadvantages to be the measure of distinguishability of hypotheses.

Disadvantages -1° Non minimal estimation of possible error in hypotheses testing -2° The region of determination is [0, 0.5] (desirable area [0, 1]) -3° Difficulties in applying of equal-tailed test for complex distributions… Goal: disadvantages ---> advantages

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

^

Distinguishability of hypothesesDistinguishability of hypotheses

-1° --> + 4° The applying of the equal probability test gives the minimal half-sum of estimators of Type I error and Type II by the definition of this test. The critical value n0 is chosen by the condition f(n0; b) = f(n0; s + b) .

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

The relative number of incorrect decisions under equal probability The relative number of incorrect decisions under equal probability testtest

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

It is a relative number of incorrect decisions. The region of determination is [0, 1] .

-3° --> + 6° The applicability of equal probability test to the complex distributions is obviously.

-2° --> + 5° The transformation makes a good candidate to be a measure of distinguishability

)(21^^

^^

^

^~

The equal probability test : the critical value for Poisson The equal probability test : the critical value for Poisson distributiondistribution

The applying of the equal probability test for Poisson distributiongives the critical value

,ln)ln(0

bbs

sn

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

where square brackets mean the the integer part of a number. Note that the critical value conserves the liner dependence on the time of measurements

.ln)ln(0

bbs

sttn

The estimation of the hypotheses distinguishabilityThe estimation of the hypotheses distinguishability

The relative number of correct decisions under equal probability testis a measure of distinguishability of hypotheses at given s and b.

The magnitude of this value can be found from equations

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

)(211

);(1

);(

ln)ln(

^^

^^~

0

^

0

^

0

0

0

n

ib

n

ibs

bbs

s

if

if

n

Dependence of relative number correct decisions on the Dependence of relative number correct decisions on the measurement timemeasurement time

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Gamma- and Poisson distributionsGamma- and Poisson distributions

)(),(

1

x

x

exg

Let us consider the Gamma-distributions with probability density

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

If to redefine the parameters and variable 1/, and x via a, n+1, , correspondingly, the probability density will be

)1(),(

1

n

eaa

ann

ng

If a = 1 then the probability density looks like Poisson probabilities

.1,0,!

);(

nn

en

n

g

Conditional probability density Conditional probability density

The probability density g( ;N) of the true value of the parameter of Poisson distribution to be in the case of the single observation N has Gamma distribution , where

!);(

N

eNg

N

It follows from identity

1 0

21 2

1

21

1!!!Nn

N

n

nNn

n

ed

N

e

n

e

We checked the statement about g(;N) by the Monte Carlo and these calculations give the confirmation of given supposition.As a result we can take into account the statistical uncertainties by the simple way (see, NIM A502(2003)795; JHEP09(2002)060).

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Statistical uncertainties in determination of Statistical uncertainties in determination of ss and and bb

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Suppose that the expected number of signal events s and expected background b are determined with statistical errors.If we can reconstruct the distribution g(; N ) of the true value of the parameter of the function f(n; ) in the case of the observed N events then we can determine the relative number of correct decisions:

)(211

);();(1

);();(

^^

^^~

00

^

00

^

0

0

n

nb

n

nsb

dnfg

dnfg

ConclusionsConclusions

The probability language in terms of and is more acceptable for estimation of the quality of planned experiments than the language of standard deviations.

The proposed approach allows to use the relative number of correct decisions in the equal probability test as a measure of distinguishability of hypotheses.

This approach gives the easy way for including of the systematics and statistical uncertainties into the estimation of distinguishability of hypotheses.

December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov