8/6/2019 Minimal Sufficiency

1/2

Sufficient Statistics

Guy LebanonMay 2, 2006

A sufficient statistics with respect to is a statistic T (X 1 , . . . , X n ) that contains all the information thatis useful for the estimation of . It is useful a data reduction tool, and studying its properties leads to otheruseful results.

Denition 1. A statistic T is sufficient for if p(x 1 , . . . , x n |T (x1 , . . . , x n )) is not a function of .A useful way to visualize it is as a Markov chain T (X 1 , . . . , X n ) {X 1 , . . . , X n } (although inclassical statistics is not a random variable but a specic value). Conditioned on the middle part of the

chain, the front and back are independent.As mentioned above, the intuition behind the sufficient statistic concept is that it contains all the infor-

mation necessary for estimating . Therefore if one is interested in estimating , it is perfectly ne to get ridof the original data while keeping only the value of the sufficient statistic. The motivation connects to theformal denition by considering the concept of sampling a ghost sample: Consider a statistician who erasedthe original data, but kept the sufficient statistic. Since p(x 1 , . . . , x n |T (x 1 , . . . , x n )) is not a function of (which is unknown), we assume that it is a known distribution. The statistician can then sample x 1 , . . . , x nfrom that conditional distribution, and that ghost sample can be used in lieu of the original data that wasthrown away.

The denition of sufficient statistic is very hard to verify. A much easier way to nd sufficient statisticsis through the factorization theorem.

Denition 2. Let X 1 , . . . , X n be iid RVs whose distribution is the pdf f X i or the pmf pX i . The likelihood function is the product of the pdfs or pmfs

L (x 1 , . . . , x n |) =ni =1 f X i (x i ) X i is a continuous RV ni =1 pX i (x i ) X i is a discrete RV

.

The likelihood function is sometimes viewed as a function of x 1 , . . . , x n (xing ) and sometimes as afunction of (xing x 1 , . . . , x n ). In the latter case, the likelihood is sometimes denoted L ().

Theorem 1 (Factorization Theorem) . T is a sufficient statistic for if the likelihood factorizes into the following form

L (x 1 , . . . , x n |) = g(, T (x 1 , . . . , x n )) h (x 1 , . . . , x n ) for some functions g, h .

Proof. We prove the theorem only for the discrete case (the continuous case requires different techniques).First assume the likelihood factorizes as above. Then

p(x 1 , . . . , x n |T (x 1 , . . . , x n )) =p(x 1 , . . . , x n , T (x 1 , . . . , x n ))

p(T (x 1 , . . . , x n ))=

p(x 1 , . . . , x n )

y:T (y )= T (x ) p(y1 , . . . , y n )=

h (x 1 , . . . , x n )

y:T (y )= T (x ) h (y1 , . . . , y n

which is not a function of . Conversely, assume that T is a sufficient statistic for . Then

L (x 1 , . . . , x n |) = p(x 1 , . . . , x n |T (x 1 , . . . , x n ), ) p(T (x 1 , . . . , x n )|) = h (x 1 , . . . , x n )g(T (x 1 , . . . , x n ), ).

1

8/6/2019 Minimal Sufficiency

2/2