Significance Tests:Significance Tests:

THE BASICSTHE BASICS

Could it happen by chance alone?Could it happen by chance alone?

Statistical InferenceStatistical Inference

Confidence Intervals—Use when you Confidence Intervals—Use when you want to estimate a population parameterwant to estimate a population parameter

Significance Tests—Use when you want Significance Tests—Use when you want to assess the evidence provided by data to assess the evidence provided by data about some claim concerning a about some claim concerning a populationpopulation– AN OUTCOME THAT WOULD RARELY HAPPEN AN OUTCOME THAT WOULD RARELY HAPPEN

BY CHANCE IF A CLAIM WERE TRUE IS GOOD BY CHANCE IF A CLAIM WERE TRUE IS GOOD EVIDENCE THAT THE CLAIM IS NOT TRUEEVIDENCE THAT THE CLAIM IS NOT TRUE

Overview of a Significance Overview of a Significance TestTest A A test of significancetest of significance is intended to assess the is intended to assess the

evidence provided by data against a evidence provided by data against a null null hypothesis hypothesis HH0 0 in favor of an alternate hypothesis in favor of an alternate hypothesis HHaa..

The statement being tested in a test of significance The statement being tested in a test of significance is called the is called the null hypothesisnull hypothesis. Usually the null . Usually the null hypothesis is a statement of “no effect” or “no hypothesis is a statement of “no effect” or “no difference.” difference.”

A A one-sided one-sided alternate hypothesis exists when we alternate hypothesis exists when we are interested only in deviations from the null are interested only in deviations from the null hypothesis in one directionhypothesis in one direction

HH00 : : =0 =0 HHa a : : >0 (or >0 (or <0)<0)

If the problem does not specify the direction of the If the problem does not specify the direction of the difference, the alternate hypothesis is difference, the alternate hypothesis is two-sidedtwo-sided

HH00: : =0=0HHaa: : ≠≠00

HYPOTHESESHYPOTHESESNOTE: Hypotheses ALWAYS NOTE: Hypotheses ALWAYS

refer to a population parameter, refer to a population parameter, not a sample statistic.not a sample statistic.

The alternative hypothesis The alternative hypothesis should express the hopes or should express the hopes or suspicions we have BEFORE we suspicions we have BEFORE we see the data. Don’t “cheat” by see the data. Don’t “cheat” by looking at the data first.looking at the data first.

CONDITIONSCONDITIONS These should look the same as in the last chapter (for confidence These should look the same as in the last chapter (for confidence

intervals)intervals)– RandomRandom

Data is from an SRS or from a randomized experimentData is from an SRS or from a randomized experiment– NormalNormal

For means—population distribution is Normal or you have a For means—population distribution is Normal or you have a large sample size (nlarge sample size (n≥30) to ensure a Normal sampling ≥30) to ensure a Normal sampling distribution for the sample meandistribution for the sample mean

For proportions—np≥10 and n(1-p)≥10 (meaning the sample is For proportions—np≥10 and n(1-p)≥10 (meaning the sample is large enough to ensure a Normal sampling distribution for large enough to ensure a Normal sampling distribution for

– IndependentIndependent Either you are sampling with replacement or you have a Either you are sampling with replacement or you have a

population at least 10 times as big as the sample to make using population at least 10 times as big as the sample to make using the formula for st. dev. okay. the formula for st. dev. okay.

p̂

CAUTIONCAUTIONBe sure to check that the Be sure to check that the

conditions for running a conditions for running a significance test for the significance test for the population mean are satisfied population mean are satisfied before you perform any before you perform any calculations.calculations.

Test StatisticTest Statistic A test statistic comes from sample data A test statistic comes from sample data

and is used to make decisions in a and is used to make decisions in a significance testsignificance test– Compare sample statistic to hypothesized Compare sample statistic to hypothesized

parameterparameter– Values far from parameter give evidence Values far from parameter give evidence

against the null hypothesis (Hagainst the null hypothesis (H00))

– Standardize your sample statistic to obtain Standardize your sample statistic to obtain your TEST STATISTICyour TEST STATISTIC

PP-values & statistical -values & statistical significancesignificance

The probability (computed assuming HThe probability (computed assuming H00 is true) is true) that the test statistic would take a value as that the test statistic would take a value as extreme or more extreme than that actually extreme or more extreme than that actually observed is called the observed is called the PP-value-value of the test. The of the test. The smaller the smaller the PP-value, the stronger the evidence -value, the stronger the evidence against the null hypothesis provided by the against the null hypothesis provided by the data.data.

““Significant” in the statistical sense doesn’t Significant” in the statistical sense doesn’t mean “important”. It means simply “not likely mean “important”. It means simply “not likely to happen just by chance.”to happen just by chance.”

The The significance level significance level α α is the decisive value of is the decisive value of the the PP-value. It makes “not likely” more exact.-value. It makes “not likely” more exact.

If the If the PP-value is as small or smaller than -value is as small or smaller than α, α, we we say that the data is say that the data is statistically significant statistically significant at levelat level α.α.

INFERENCE TOOLBOX (p INFERENCE TOOLBOX (p 705)705)

1—1—PPARAMETER—Identify the population of ARAMETER—Identify the population of interest and the parameter you want to draw a interest and the parameter you want to draw a conclusion about. STATE YOUR HYPOTHESES!conclusion about. STATE YOUR HYPOTHESES!

2—2—CCONDITIONS—Choose the appropriate ONDITIONS—Choose the appropriate inference procedure. VERIFY conditions inference procedure. VERIFY conditions ((Random, Normal, Independent) Random, Normal, Independent) before using it.before using it.

3—3—CCALCULATIONS—If the conditions are met, ALCULATIONS—If the conditions are met, carry out the inference procedure.carry out the inference procedure.

4—4—IINTERPRETATION—Interpret your results in NTERPRETATION—Interpret your results in the context of the problem. CONCLUSION, the context of the problem. CONCLUSION, CONNECTION, CONTEXT(meaning that our CONNECTION, CONTEXT(meaning that our conclusion about the parameter connects to our conclusion about the parameter connects to our work in part 3 and includes appropriate work in part 3 and includes appropriate context)context)

Steps for completing a SIGNIFICANCE TEST:

DO YOU REMEMBER WHAT THE STEPS ARE???

Step 1—PARAMETERStep 1—PARAMETER

Read through the problem and Read through the problem and determine what we hope to show determine what we hope to show through our test.through our test.

Our null hypothesis is that no change has Our null hypothesis is that no change has occurred or that no difference is evident.occurred or that no difference is evident.

Our alternative hypothesis can be either Our alternative hypothesis can be either one or two sided.one or two sided.

Be certain to use appropriate symbols Be certain to use appropriate symbols and also write them out in words.and also write them out in words.

Step 2—CONDITIONSStep 2—CONDITIONS

Based on the given information, determine which Based on the given information, determine which test should be used. Name the procedure.test should be used. Name the procedure.

State the conditions.State the conditions. Verify (through discussion) whether the Verify (through discussion) whether the

conditions have been met. For any assumptions conditions have been met. For any assumptions that seem unsafe to verify as met, explain why.that seem unsafe to verify as met, explain why.

Remember, if data is given, graph it to help Remember, if data is given, graph it to help facilitate this discussionfacilitate this discussion

For each procedure there are several things that For each procedure there are several things that we are assuming are true that allow these we are assuming are true that allow these procedures to produce meaningful results.procedures to produce meaningful results.

Step 3—CALCULATIONSStep 3—CALCULATIONS

First write out the formula for the test First write out the formula for the test statistic, report its value, mark the value on statistic, report its value, mark the value on the curve.the curve.

Sketch the density curve as clearly as Sketch the density curve as clearly as possible out to three standard deviations on possible out to three standard deviations on each side.each side.

Mark the null hypothesis and sample statistic Mark the null hypothesis and sample statistic clearly on the curve.clearly on the curve.

Calculate and report the Calculate and report the PP-value-value Shade the appropriate region of the curve.Shade the appropriate region of the curve. Report other values of importance (standard Report other values of importance (standard

deviation, df, critical value, etc.)deviation, df, critical value, etc.)

Step 4—INTERPRETATIONStep 4—INTERPRETATION There are really two parts to this step: decision & There are really two parts to this step: decision &

conclusion. TWO UNIQUE SENTENCES.conclusion. TWO UNIQUE SENTENCES. Based on the Based on the PP-value, make a decision. Will you -value, make a decision. Will you

reject Hreject H00 or fail to reject H or fail to reject H00.. If there is a predetermined significance level, If there is a predetermined significance level,

then make reference to this as part of your then make reference to this as part of your decision. If not, interpret the decision. If not, interpret the PP-value -value appropriately.appropriately.

Now that you have made a decision, state a Now that you have made a decision, state a conclusion IN THE CONTEXT of the problem.conclusion IN THE CONTEXT of the problem.

This does not need to, and probably should not, This does not need to, and probably should not, have statistical terminology involved. DO NOT have statistical terminology involved. DO NOT use the word “prove” in this statement.use the word “prove” in this statement.

Example 1Example 1

Your buddy (Jake) claims to be an A Your buddy (Jake) claims to be an A student (meaning he has a 90 average). student (meaning he has a 90 average). You don’t know all of his grades but based You don’t know all of his grades but based on what you have seen you think this on what you have seen you think this claim is an overstatement. You took a claim is an overstatement. You took a simple random sample of his grades and simple random sample of his grades and recorded them. They are: 92, 87, 86, 90, recorded them. They are: 92, 87, 86, 90, 80, 91. You also know that all his grades 80, 91. You also know that all his grades in the class have a standard deviation of in the class have a standard deviation of 3.5.3.5.

Step 1Step 1 We want to determine whether Jake is We want to determine whether Jake is

accurate in his measure of his course accurate in his measure of his course grade.grade.

Our null hypothesis is that Jake has a Our null hypothesis is that Jake has a course average of 90.course average of 90.

Our alternative hypothesis is that Jake’s Our alternative hypothesis is that Jake’s course average is below a 90.course average is below a 90.

HH00: : = 90 = 90

HHaa: : < 90 < 90

Step 2Step 2 Since we know the population standard deviation Since we know the population standard deviation

we will be performing a z-test of significance. we will be performing a z-test of significance. (NOTE-in practice, we rarely know sigma) (NOTE-in practice, we rarely know sigma)

We were told that our selection of grades was an We were told that our selection of grades was an SRS of Jake’s scores.SRS of Jake’s scores.

The box plot shows moderate left skewness. Our The box plot shows moderate left skewness. Our sample is not large so we must assume that the sample is not large so we must assume that the population of all of Jake’s grades are population of all of Jake’s grades are approximately normal in distribution in order for approximately normal in distribution in order for our sampling distribution to be approximately our sampling distribution to be approximately normal. Using the IQR(1.5) method for normal. Using the IQR(1.5) method for determining outliers we see that there are no determining outliers we see that there are no outliers in this sample of grades.outliers in this sample of grades.

Provided Jake has at least 60 overall grades, we Provided Jake has at least 60 overall grades, we are safe assuming independence and using the are safe assuming independence and using the necessary formula for standard deviation.necessary formula for standard deviation.

Step 3Step 3 A curve should be drawn, labeled, and A curve should be drawn, labeled, and

shaded.shaded. You can use the formula to calculate your You can use the formula to calculate your

z test statistic for this problemz test statistic for this problem In this case z = -1.6330In this case z = -1.6330

Mark this on your sketch.Mark this on your sketch. Based on our calculations the Based on our calculations the PP-value is -value is

0.0512.0.0512. , , σσ=3.5, n=6=3.5, n=6

0xz

n

87.6667x

Step 4Step 4 Since there is no predetermined level of Since there is no predetermined level of

significance if we are seeking to make a significance if we are seeking to make a decision, this could be argued either way. If decision, this could be argued either way. If Jake were correct about being an A student, we Jake were correct about being an A student, we would only get a sample of grades with an would only get a sample of grades with an average this low in roughly 5.1% of all samples. average this low in roughly 5.1% of all samples.

There is not overwhelming evidence against HThere is not overwhelming evidence against H00, , however, this is enough to convince me that Hhowever, this is enough to convince me that H00 can be rejected.can be rejected.

Our evidence may not be strong enough to Our evidence may not be strong enough to convince Jake that he is wrong. However, convince Jake that he is wrong. However, based on this evidence, I do not believe Jake is based on this evidence, I do not believe Jake is accurate about his average being a 90. It accurate about his average being a 90. It doesn’t appear that Jake is the A student he doesn’t appear that Jake is the A student he claims to be.claims to be.

WARNINGSWARNINGS Tests of significance assess evidence against

H0

If the evidence is strong, reject H0 in favor of Ha

Failure to find evidence against H0 means only that data are consistent with H0, not that we have clear evidence that H0 is true

If you are going to make a decision based on statistical significance, then the significance level αα should be stated before the data are produced.