Tests

Jean-Yves Le Boudec

Contents

1. The Neyman Pearson framework

2. Likelihood Ratio Tests

3. ANOVA

4. Asymptotic Results

5. Other Tests

2

Tests

Tests are used to give a binary answer to hypotheses of a statistical natureEx: is A better than B?

Ex: does this data come from a normal distribution ?

Ex: does factor n influence the result ?

3

Example: Non Paired Data

Is red better than blue ?

For data set (a) answer is clear (by inspection of confidence interval) no test required

4

Is this data normal ?

5

5.1 The Neyman-Pearson Framework

6

Example: Non Paired Data

7

Is red better than blue ?

Critical Region, Size and Power

8

Example : Paired Data

9

Power

10

11

Grey Zone

12

p-value of a test

13

14

Tests are just tests

15

Test versus Confidence Intervals

If you can have a confidence interval, use it instead of a test

16

2. Likelihood Ratio Test

A special case of Neyman-Pearson

A Systematic Method to define tests, of general applicability

17

18

19

A Classical Test: Student Test

The model :

The hypotheses :

20

21

22

Here it is the same as a Conf. Interval

23

The “Simple Goodness of Fit” Test

Model

Hypotheses

24

1. compute likelihood ratio statistic

25

2. compute p-value

26

Mendel’s Peas

P= 0.92 ± 0.05 => Accept H0

27

3 ANOVA

Often used as “Magic Tool”

Important to understand the underlying assumptions

ModelData comes from iid normal sample with unknown means and same variance

Hypotheses

28

29

30

The ANOVA Theorem

We build a likelihood ratio statistic test

The assumption that data is normal and variance is the same allows an explicit computation

it becomes a least square problem = a geometrical problem

we need to compute orthogonal projections on M and M0

31

The ANOVA Theorem

32

Geometrical InterpretationAccept H0 if SS2 is small

The theorem tells us what “small” means in a statistical sense

33

34

ANOVA Output: Network Monitoring

35

The Fisher-F distribution

36

37

Compare Test to Confidence Intervals

For non paired data, we cannot simply compute the differenceHowever CI is sufficient for parameter set 1

Tests disambiguate parameter sets 2 and 3

38

Test the assumptions of the test…Need to test the assumptions

Normal In each group: qqplot…

Same variance

39

40

4 Asymptotic Results

41

2 x Likelihood ratio statistic

42

The chi-square distribution

43

Asymptotic Result

Applicable when central limit theorem holds

If applicable, radically simpleCompute likelihood ratio statistic

Inspect and find the order p (nb of dimensions that H1 adds to H0)

This is equivalent to 2 optimization subproblems

lrs = = max likelihood under H1 - max likelihood under H0

The p-value is

44

Composite Goodness of Fit Test

We want to test the hypothesis that an iid sample has a distribution that comes from a given parametric family

45

Apply the Generic Method

Compute likelihood ratio statistic

Compute p-value

Either use MC or the large n asymptotic

46

47

Is it normal ?

48

49

50

Mendel’s Peas

P= 0.92 ± 0.05 => Accept H0

51

Test of Independence

Model

Hypotheses

52

Apply the generic method

53

54

5 Other TestsSimple Goodness of Fit

Model: iid data

Hypotheses: H0 common distrib has cdf F()H1 common distrib is anything

Kolmogorov-Smirnov: under H0, the distribution of

is independent of F()

55

56

Anderson-Darling

An alternative to K-S, less sensitive to “outliers”

57

58

59

Jarque Bera test of normality (Chapter 4)

Based on Kurtosis and SkewnessShould be 0 for normal distribution

60

61

Robust TestsMedian Test

Model : iid sample

Hypotheses

62

Median Test

63

Wilcoxon Signed Rank Test

64

Wilcoxon Rank Sum Test

Model: Xi and Yj independent samples, each is iid

Hypotheses: H0 both have same distributionH1 the distributions differ by a location shift

65

Wilcoxon Rank Sum Test

66

Turning Point

67

Questions

What is the critical region of a test ?

What is a type 1 error ? Type 2 ? The size of a test ?

What is the p-value of a test ?

68

Questions

What are the hypotheses for ANOVA ?

How do you compute a p-value by Monte Carlo simulation ?

A Monte Carlo simulation returns p = 0; what can we conclude ?

What is a likelihood ratio statistic test ? What can we say about its p-value ?

69

We have data X_1,…,X_m and Y_1, …,Y_m. Explain how we can compute the p-value of a test that compares the variance of the two samples ?

We have a collection of random variables X[i,j] that corresponds to the result of the ith simulation when the machine uses configuration j. How can you test whether the configuration plays a role or not ?

70