CAUSE Teaching and Learning Webinar - January 11, 2011

Introducing Informal Introducing Informal Inference Using Data-Inference Using Data-Centric Lab ExercisesCentric Lab Exercises

Rakhee Patel, Rob Gould and Gretchen Rakhee Patel, Rob Gould and Gretchen DavisDavisUCLA Department of StatisticsUCLA Department of Statistics

CAUSE Teaching and Learning Webinar - January CAUSE Teaching and Learning Webinar - January 11, 201111, 2011

OutlineOutline

Why teach informal inference early?Why teach informal inference early? Informal inference using randomizationInformal inference using randomization Introductory computer lab assignmentsIntroductory computer lab assignments

ImplementationImplementation GoalsGoals Lab example 1: tuberculosis dataLab example 1: tuberculosis data Lab example 2: births and smoking dataLab example 2: births and smoking data

DiscussionDiscussion

CAUSE Teaching and Learning Webinar - January 11, 2011 Slide 2 of 26

Informal InferenceInformal Inference

CAUSE Teaching and Learning Webinar - January 11, 2011

Students often encounter difficulties with Students often encounter difficulties with formal inference methods later in formal inference methods later in introductory coursesintroductory courses

Particularly, Particularly, abstractabstract concepts are difficult concepts are difficult What is a null distribution/hypothesis?What is a null distribution/hypothesis? How do we know when to reject the null?How do we know when to reject the null? What is a p-value?What is a p-value?

Introducing informal inference early in a Introducing informal inference early in a course may help with these strugglescourse may help with these struggles Can do so without using complex vocabulary Can do so without using complex vocabulary

and mathematical machinery and mathematical machinery Slide 3 of 26

Randomization TestsRandomization Tests


Randomization tests are a good way to Randomization tests are a good way to illustrate these concepts in a way that illustrate these concepts in a way that students can students can see with their own eyessee with their own eyes Also a good way to illustrate power of Also a good way to illustrate power of

computerscomputers Can use many measuresCan use many measures In some cases, they more closely match the In some cases, they more closely match the

design of experimentdesign of experiment See Cobb (2007), The Introductory See Cobb (2007), The Introductory

Statistics Course: A Ptolemaic Curriculum?Statistics Course: A Ptolemaic Curriculum?

Slide 4 of 26

Computer Lab AssignmentsComputer Lab Assignments


Students complete eight weekly lab Students complete eight weekly lab assignments assignments

Each lab uses real data to answer a Each lab uses real data to answer a particular investigative questionparticular investigative question Lab also includes intermediate questions that Lab also includes intermediate questions that

help guide students to answer the investigative help guide students to answer the investigative questionquestion

Students ultimately turn in set of summary Students ultimately turn in set of summary questions regarding: questions regarding: ultimate goals of labultimate goals of lab methods usedmethods used conclusions about research question conclusions about research question applications of concepts to real worldapplications of concepts to real world Slide 5 of 26

Implementation GoalsImplementation Goals


To understand how concepts in lecture To understand how concepts in lecture are applied to understanding real dataare applied to understanding real data

Design labs so that learning the software Design labs so that learning the software does not get in the way of doing does not get in the way of doing statistical analysisstatistical analysis First lab assignment introduces Fathom First lab assignment introduces Fathom

softwaresoftware Subsequent labs provide additional Subsequent labs provide additional

instructions for any new methods usedinstructions for any new methods used TAs will talk less, assist more.TAs will talk less, assist more.

Slide 6 of 26

Informal Inference GoalsInformal Inference Goals


To understand why permuting observations To understand why permuting observations simulates a null distributionsimulates a null distribution

To understand that null sampling To understand that null sampling distribution is used to make decisions about distribution is used to make decisions about rejecting null hypothesisrejecting null hypothesis Develop intuition behind Develop intuition behind chance modelschance models

To understand how to use the null To understand how to use the null distribution to estimate the p-value & make distribution to estimate the p-value & make decisionsdecisions Use Use chance modelchance model to make inference about to make inference about

actual dataactual dataSlide 7 of 26

Example Lab 1: TB or Not Example Lab 1: TB or Not TB?TB?


Investigative Question Investigative Question Is Streptomycin an effective treatment for Is Streptomycin an effective treatment for

tuberculosis?tuberculosis? ObjectivesObjectives

Using two-way tables, we can see how to make Using two-way tables, we can see how to make conclusions about cause-and-effect by conclusions about cause-and-effect by comparing actual results to a chance model.comparing actual results to a chance model.

DataData Outcomes from Austin Bradford Hill’s first Outcomes from Austin Bradford Hill’s first

randomized study in 1948 examining the randomized study in 1948 examining the effects of the antibiotic Streptomycin vs. a effects of the antibiotic Streptomycin vs. a control on 107 tuberculosis patientscontrol on 107 tuberculosis patients

Slide 8 of 26

Example Lab 1: Intermediate Example Lab 1: Intermediate Question 1Question 1


Based on Hill’s study, does it Based on Hill’s study, does it seemseem that that treatment and outcome are independent treatment and outcome are independent or dependent?or dependent?

Slide 9 of 26

Treatment and Treatment and outcome outcome appearappear to be to be dependentdependent. 26% . 26% died in control group died in control group while only 7% died in while only 7% died in Streptomycin group.Streptomycin group.



IfIf treatment and outcome treatment and outcome werewere independent, how many patients in the independent, how many patients in the Streptomycin group, according to Hill’s Streptomycin group, according to Hill’s study, would we study, would we expectexpect to die if we to die if we replicated the study again?replicated the study again?

Slide 10 of 26

We would expect a We would expect a proportion of 18/107 = proportion of 18/107 = 0.168 to die overall. So 0.168 to die overall. So of the 55 patients in the of the 55 patients in the Streptomycin group, we Streptomycin group, we expect 0.168 of them, expect 0.168 of them, or or 9.259.25 patients patients (roughly 9), to die.(roughly 9), to die.



Design and execute an appropriate simulation in Design and execute an appropriate simulation in Fathom using a chance model to replicate Hill’s Fathom using a chance model to replicate Hill’s study study under the assumption that treatment under the assumption that treatment and outcome are and outcome are independentindependent. Repeat the . Repeat the simulation 100 times and then use the results from simulation 100 times and then use the results from the chance model to determine whether (a) or (b) is the chance model to determine whether (a) or (b) is the most reasonable explanation for the actual data the most reasonable explanation for the actual data in Hill’s study.in Hill’s study. (a) Streptomycin is a significantly more effective (a) Streptomycin is a significantly more effective

treatment for tuberculosis than bed rest. Thus, treatment treatment for tuberculosis than bed rest. Thus, treatment and outcome are dependent.and outcome are dependent.

(b) The actual difference between treatments is due to (b) The actual difference between treatments is due to chance variation and Streptomycin may have no effect chance variation and Streptomycin may have no effect on tuberculosis. Thus, it is possible that treatment and on tuberculosis. Thus, it is possible that treatment and outcome are independent.outcome are independent.

Slide 11 of 26



Student discussions are crucial in determining Student discussions are crucial in determining how to set up the simulations, as there are many how to set up the simulations, as there are many waysways Question 2 might guide them to look at the number of Question 2 might guide them to look at the number of

deaths in the Streptomycin group for each simulationdeaths in the Streptomycin group for each simulation

Slide 12 of 26

As expected, the chance As expected, the chance model shows an average of model shows an average of about 9 to 10 deaths in the about 9 to 10 deaths in the Streptomycin group. Only 4 Streptomycin group. Only 4 died in the actual data which died in the actual data which never occurs in these never occurs in these simulations of the model, so simulations of the model, so the chance model does not the chance model does not fit well and fit well and (a)(a) is more is more reasonable.reasonable.

Example Lab 1: Common Example Lab 1: Common MisconceptionsMisconceptions


Students confuse Students confuse randomizedrandomized data and data and expectations with expectations with actualactual data: data:

Students look at the Students look at the distancedistance between between the actual and the chance model rather the actual and the chance model rather than than how oftenhow often the actual occurs under the actual occurs under the chance modelthe chance model

Slide 13 of 26

If outcome is independent of treatment, If outcome is independent of treatment, we expect 9.25 deaths in the we expect 9.25 deaths in the Streptomycin group. The chance model Streptomycin group. The chance model also shows an average of between 9 and also shows an average of between 9 and 10 deaths, so we cannot reject chance.10 deaths, so we cannot reject chance.

Since the actual value of 4 is far from the Since the actual value of 4 is far from the expected value of 9.25 then we reject the expected value of 9.25 then we reject the chance model.chance model.



Is Hill’s study an example of an observational Is Hill’s study an example of an observational study or an experiment? If we had seen a real study or an experiment? If we had seen a real difference between the Streptomycin group and difference between the Streptomycin group and the control group and concluded that the control group and concluded that Streptomycin was effective (and maybe you did Streptomycin was effective (and maybe you did in Question 3), can we conclude that the in Question 3), can we conclude that the causecause is the antibiotic? Why?is the antibiotic? Why?

Slide 14 of 26

Hill’s study is an experiment because he Hill’s study is an experiment because he imposed the Streptomycin and control imposed the Streptomycin and control treatments on the patients, while keeping all treatments on the patients, while keeping all other factors constant. If we found other factors constant. If we found Streptomycin to be effective (which we did), Streptomycin to be effective (which we did), we can conclude that the antibiotic is the we can conclude that the antibiotic is the cause of the lower number of deaths cause of the lower number of deaths because Hill could completely control the because Hill could completely control the study and treatments.study and treatments.

Example Lab 1: Example Lab 1: ObservationsObservations


Without using the terms null hypothesis or p-value, Without using the terms null hypothesis or p-value, students should be able to find a significant students should be able to find a significant difference between the groups and a causal difference between the groups and a causal relationshiprelationship

Students may have trouble with the concepts and Students may have trouble with the concepts and software used for this lab, but subsequent labs software used for this lab, but subsequent labs repeat the same techniques to illustrate the same repeat the same techniques to illustrate the same general inference steps for various other types of general inference steps for various other types of datadata

It is important to show students several realizations It is important to show students several realizations from the chance model to illustrate what happens from the chance model to illustrate what happens under the “no difference” hypothesisunder the “no difference” hypothesis Any differences that occur are strictly Any differences that occur are strictly by chanceby chance

Slide 15 of 26

Example Lab 2: Compared Example Lab 2: Compared to What?to What?


Investigative Question Investigative Question How does birth weight differ between How does birth weight differ between

smoking and non-smoking mothers?smoking and non-smoking mothers? ObjectivesObjectives

Using statistical summaries and the chance Using statistical summaries and the chance model, we can explore whether mothers model, we can explore whether mothers who smoke have babies with quantifiably who smoke have babies with quantifiably different weights than mothers who do not.different weights than mothers who do not.

DataData Characteristics of mothers and their babies Characteristics of mothers and their babies

for a sample of 1,000 births in North for a sample of 1,000 births in North Carolina in 2004Carolina in 2004

Slide 16 of 26



If a physician told you that a mother's If a physician told you that a mother's smoking affected the birth of her baby, smoking affected the birth of her baby, how much, typically, would you say the how much, typically, would you say the mother's smoking changes a baby's mother's smoking changes a baby's weight at birth, based on the weight at birth, based on the actual actual datadata??

Slide 17 of 26

I would guess (based on the I would guess (based on the sample we have) that sample we have) that smoking during pregnancy smoking during pregnancy typically lowers the baby’s typically lowers the baby’s weight by around weight by around 0.25 lbs0.25 lbs, , the difference in median the difference in median birth weights.birth weights.



We will work with a chance model that assumes We will work with a chance model that assumes that smoking has that smoking has NONO effect on the babies' effect on the babies' weights.weights.

Using this chance model, simulate a set of data Using this chance model, simulate a set of data and calculate the difference in median birth and calculate the difference in median birth weights between the smoking and non-smoking weights between the smoking and non-smoking groups. Repeat this 100 times. groups. Repeat this 100 times.

According to the chance model (your 100 According to the chance model (your 100 simulations), what would you estimate is the simulations), what would you estimate is the typical difference in medians between the two typical difference in medians between the two groups? What would you consider to be an groups? What would you consider to be an unusually large difference, according to the unusually large difference, according to the chance model?chance model?

Slide 18 of 26



Again, it is important for students to Again, it is important for students to work together to determine how to set work together to determine how to set up the simulationup the simulation

Slide 19 of 26

I would expect the typical I would expect the typical difference in median birth difference in median birth weights to be close to zero, weights to be close to zero, which is reiterated by the which is reiterated by the average difference of -0.011 average difference of -0.011 shown in the summary table. shown in the summary table. The plot shows that an The plot shows that an unusually large difference unusually large difference seems to be around 0.2 lbs.seems to be around 0.2 lbs.



In the actual data, the true difference in In the actual data, the true difference in median birth weights between the smoking median birth weights between the smoking mothers and non-smoking mothers is 0.25 mothers and non-smoking mothers is 0.25 pounds. How often did the chance model pounds. How often did the chance model produce a difference this large or larger?produce a difference this large or larger?

Slide 20 of 26

In the 100 chance model In the 100 chance model simulations, I rarely saw a simulations, I rarely saw a difference in medians as difference in medians as large as 0.25 lbs (the actual large as 0.25 lbs (the actual difference) as indicated by difference) as indicated by the dot plot. In fact, only 2 the dot plot. In fact, only 2 out of the 100 simulations out of the 100 simulations resulted in a difference as resulted in a difference as large as the actual result.large as the actual result.



We have mentioned two possible reasons for We have mentioned two possible reasons for differences in the weights of babies from differences in the weights of babies from smoking mothers versus non-smoking smoking mothers versus non-smoking mothers in the actual data:mothers in the actual data: Smoking mothers really do have lighter babies. Smoking mothers really do have lighter babies. The difference between birth weights from The difference between birth weights from

smokers and non-smokers is due to chance. smokers and non-smokers is due to chance. Use your answer to Question 3 to justify which Use your answer to Question 3 to justify which

of these explanations seems most reasonableof these explanations seems most reasonable

Slide 21 of 26

Since the actual difference in birth weights between Since the actual difference in birth weights between the smoker and non-smoker groups rarely occurs the smoker and non-smoker groups rarely occurs using the 100 chance model simulations, the chance using the 100 chance model simulations, the chance model does not seem to fit the data very well and model does not seem to fit the data very well and therefore it seems that smoking mothers actually do therefore it seems that smoking mothers actually do have lighter babies.have lighter babies.



Is this study an example of an observational study Is this study an example of an observational study or an experiment? Justify your answer. If we had or an experiment? Justify your answer. If we had seen a real difference between birth weights from seen a real difference between birth weights from smokers and non-smokers and concluded that smokers and non-smokers and concluded that smoking and birth weight are associated (and smoking and birth weight are associated (and maybe you did in Question 4), can we conclude maybe you did in Question 4), can we conclude that smoking causes low birth weight? Why?that smoking causes low birth weight? Why?

Slide 22 of 26

This is an observational study because the smoking This is an observational study because the smoking was not imposed on the mothers. If we had found a was not imposed on the mothers. If we had found a real difference in birth weights between smokers and real difference in birth weights between smokers and non-smokers (which we did), we still cannot conclude non-smokers (which we did), we still cannot conclude that smoking is the cause of the lighter babies. This that smoking is the cause of the lighter babies. This due to the fact that with an observational study, we due to the fact that with an observational study, we cannot control all factors of the study (such as other cannot control all factors of the study (such as other bad habits, for example) and so these other outside bad habits, for example) and so these other outside factors could affect the weight of the babies as well.factors could affect the weight of the babies as well.

Example Lab 2: Example Lab 2: ObservationsObservations


Randomizing allows students to examine Randomizing allows students to examine any measure on a data set (such as the any measure on a data set (such as the median)median)

Again, students should see several Again, students should see several iterations from the chance model to iterations from the chance model to illustrate the “no difference” hypothesisillustrate the “no difference” hypothesis

Once students practice using randomization Once students practice using randomization to make inference, the intuition should to make inference, the intuition should become more and more clearbecome more and more clear Does chance model fit the data well?Does chance model fit the data well? If not, what does that mean?If not, what does that mean?

Slide 23 of 26

DiscussionDiscussion


Repetition is keyRepetition is key Seeing is believingSeeing is believing Some future labs follow same structureSome future labs follow same structure

Fit model to dataFit model to data Make inference about validity of modelMake inference about validity of model

Normal models can be thought of as nice Normal models can be thought of as nice shortcut to use when assumptions are metshortcut to use when assumptions are met

Students generally show less frustration Students generally show less frustration once the formal methods are taughtonce the formal methods are taught Performance on inference related problems has Performance on inference related problems has

improvedimprovedSlide 24 of 26

Final Impressions and Final Impressions and Future WorkFuture Work


Based on graded lab examples and Based on graded lab examples and evaluations:evaluations: Students seem to understand why permuting Students seem to understand why permuting

simulates a null distribution, but are not sure simulates a null distribution, but are not sure what the null distribution itself representswhat the null distribution itself represents

Students don't know how to use the nullStudents don't know how to use the null Need more work to integrate the lessons Need more work to integrate the lessons

learned in lab into the lecturelearned in lab into the lecture Perhaps more support materials are necessaryPerhaps more support materials are necessary

Integrating randomization-based testing with Integrating randomization-based testing with parametric testing is more difficult than parametric testing is more difficult than anticipatedanticipated

Slide 25 of 26

Thank YouThank You


This work has been supported by grant This work has been supported by grant NSF DUE‐0737126NSF DUE‐0737126

Contact information: Contact information: Rakhee Patel: Rakhee Patel: [email protected] Rob Gould: Rob Gould: [email protected]

Slide 26 of 26

CAUSE Teaching and Learning Webinar - January 11, 2011

Documents

Transcript of CAUSE Teaching and Learning Webinar - January 11, 2011