STAT 360: Take-Home Final Name(s):_________________________Fall 2014Points: 100 points ________________________

Consider the Fan Cost Index which is used to track the cost of attendance for a family of four to major sporting events. According to their website, Team Marking Report is the leading publisher of sports marketing and sponsorship information used daily by executives from collegiate and professional sports properties, sponsor companies, marketing agencies and media partners.

https://www.teammarketing.com/btSubscriptions/fancostindex/index

The calculation of the Fan Cost Index (FCI) is described on the above website and can be generally expressed as.

FCI=4∗Seating+4∗Pop+2∗Beer+4∗Hotdog⏟Food

+2∗Program+Parking+2∗Cap

1. Verify the calculation of FCI for the Boston Red Sox’s in the above table. (2 pts)

I have collected data on the Fan Cost Index for all teams in the MLB, NFL, and NHL leagues from their website. The “Food” component of the Fan Cost Index was not collected.

1

2. Consider the following mean function for Fan Cost Index (FCI).

E (FCI|Seating ,Program , Parking,Cap )=β0+β1∗Seating+β2∗Parking+ β3∗Program+β4∗Cap

a. Fit the above mean function in JMP. Paste a copy of the model output here. (3 pts)

b. Are the estimated regression coefficients from your regression model fairly close to the values we’d expect them to be considering how FCI is computed? Discuss. (3 pts)

c. Consider any one of the estimated regression coefficients; give a practical interpretation of this quantity. (2 pts)

d. Use the added variable plots (or as JMP calls them Leverage plot) to confirm the notation that Seating is the most important predictor. Copy and paste the added variable plot for Seating below. Explain how we know Seating is the most important predictor as compared to the other predictors. (3 pts)

Consider a second regression model that includes only Seating.

E (FCI|Seating )=β0+β1∗Seating

e. Does this model do a decent job of modeling Fan Cost Index? Discuss. (3 pts)

f. Compare the R2 value from the model that includes all the predictors (i.e. full model) to the model with only Seating (i.e. reduced model). From this comparison, discuss whether or not these additional predictors are important in helping to understand the variability in Fan Cost Index. (2 pts)

g. The “Big Test” (see Handout #10) can be used to more formally evaluate the importance of these additional predictor variables. In particular, the “Big Test” can be used to directly compare the full and reduced models proposed above. Conduct the “Big Test” to simultaneously verify that the statistical importance of these additional predictors, i.e. Parking, Program, and Cap. (4 pts)

2

h. One predictor that was included in the dataset was not used. In particular, Premium Seating was included in this dataset, but was not considered in the model proposed above. I have included the pairwise correlation matrix for our predictor variables here.

From this correlation matrix, I decide two things: i) I should not include both Seating and Premium Seating, and ii) I should include Seating over Premium Seating in my model. Carefully explain how I arrived at both of these conclusions. (2 pts)

3. Suppose I fit add the Baseball Indicator and Football Indicator variables to the original model fit above. This Fit Dialog box and estimated regression coefficients are shown here for this model.

Fit Model Dialog Window Estimated Regression Coefficients

a. What is the effect of including the Baseball Indicator and Football Indicator predictors in this model? It may help to write-out the model using coefficients like we did in class (see notes from Handout #11). (5 pts)

b. The p-value for Baseball Indicator is 0.1540; thus, this predictor does not appear to be useful. What are the ramifications of such an outcome on our model? Discuss. (3 pts)

c. Complete the “Big Test” to confirm that the Baseball Indicator predictor nor Football Indicator predictor are necessary for this model. (4 pts)

d. Practically speaking, what does the outcome from the test run in part c. imply about Fan Cost Index for these three sports? (3 pts)

3

4. Consider the following study done in Japan that investigated the effects of sulfur dioxide (SO2) on the prevalence ratio of chronic bronchitis. The data from this study is provided below.

Comment: Observations 1 – 8 are from a non-polluted area of the city – this is evident as

sulfur dioxide measurements are low for this area, i.e. less than 0.30. Observations 9 – 17 are from a polluted area of the city

Data from study Scatterplot

It is known that the general population has a baseline prevalence ratio for chronic bronchitis. This baseline can be easily estimated by computing the average prevalence ratio using the data from the non-polluted area of the city. The researcher knows that low levels of sulfur dioxide will not increase this baseline level; however, at some point an increased level of sulfur dioxide will start to adversely affect the prevalence of chronic bronchitis. This point is called the threshold point and will be denoted by x0 here.

Model Setup Response: Prevalence Ratio Predictor: SO2 Threshold value: x0 Mean Functions

E (Prevalence Ratio|SO 2¿={ AverageSO2 SO2≤ x0β0+ β1∗SO 2 SO 2>x0

Variance Functions

4

VAR (Prevalence Ratio|SO 2¿={σ 12 SO 2≤x0σ 22 SO 2> x0

a. The first thing the researchers has requested in an estimate of the threshold value. The following process will provide such as estimate.

Finding an estimate ofx0Step 1: Estimate the average prevalence in the non-polluted area.Step 2: Obtain the mean function for prevalence ratio in the polluted area.Step 3: Set the value from Step 1 equal to the mean function from Step 2.

Solving for x in this equation provides an estimate of x0

What value do you obtain for the estimated threshold value? (3 pts)

x0=¿¿

b. The following can be used to obtain the predicted prevalence ratio for our two part mean function.

Predicted={ AverageSO2 SO 2≤ x0β0+ β1∗SO2 SO2> x0

Create the following plot of the two part estimated mean function using the predictive values computed above. (2 pts)

c. Compute the R2 for this two part mean function. (2 pts)

R2=¿¿

d. Consider the following improvements that an experienced modeler, i.e. me or you, might attempt use to improve the above model.

o Improvement #1: After careful consideration, you realize that the predicted values for data at SO2 = 0.9, 1.0, and 1.15 were computed using AverageSO2 because these observations were less than the threshold value. You decide to slide the threshold value to the left slightly, say to 0.75 to improve the prediction of these points.

5

Predicted={ AverageSO2 SO 2≤0.75β0+ β1∗SO2 SO2>0.75

o Improvement #2: Use a quadratic mean function and obtain the predicted values in the usual manner.

Mean Function: E (Prevalence Ratio|SO 2¿=β0+β1∗SO2+β2∗SO 22

Verify, via R2, that the above suggestions are actually improvements. (4 pts)

Method of obtaining predicted values R2

Two part mean function using threshold value = x0Two part mean function using threshold value = 0.75

Using quadratic mean function

e. Use the three sets of predicted values computed above to create the following plot via Graph > Overlay Plot. (2 pts)

6

f. You have worked hard to improve the predictions for this model. Unfortunately, the researcher is *not* impressed. In fact, she sarcastically laughs and refuses to consider your improvements. She encourages you to consider what SO2 level would minimize the prevalence ratio for each “improvement” method you suggest. She informs you that the smallest prevalence ratio should be occurring at very small SO2 values, i.e. close to 0.

i. Which SO2 value produces the smallest predicted prevalence ratio if improvement method #1 is used? Discuss. (2 pts)

ii. Which SO2 value produces the smallest predicted prevalence ratio if improvement method #2 is used? (3 pts)

This problem requires calculus, i.e. compute the derivative, set it equal to 0, and solve for SO2. I have computed the derivative below for those that have not had calculus.

JMP uses a centering option when fitting a quadratic mean function. This problem is easier if this is not done. I used R to obtain the estimated quadratic mean function (without centering).

Quadratic Mean Function

(without JMP’s centering option)

¿

Derivative ¿ β1+2∗β2∗SO 2¿−0.005+2∗0.00553∗SO 2

7

5. For this problem, we will modeling data from school districts in Texas. The data was obtained from the Texas Education Agency.

Source: http://ritter.tea.state.tx.us/perfreport/snapshot/

Consider the following:

Response: 4 Year Graduation Rate Potential predictor variables: Identified in the table below. Usual form for the mean and variance function.

o E (4 YearGradRate∨??? ? )=β0+…o Var (4YearGradRate∨??? ? )=σ2

Possible predictor variables that will be considered initially are listed here. We will ignore the demographic variables for now.

Variable RoleDistrictName Demographic

CountyNumberName DemographicRegion Demographic

DistictSize DemographicCommunityType DemographicWealthIndictor Demographic

TAXRATE DemographicAccountabilityRating Demographic

%AfricianAmerican Students Predictor%Hispanic Students Predictor

%White Students Predictor%AmericanIndian Students Predictor

%Asian Students Predictor%EconomicDisadvantaged Students Predictor

%EnglishLanguageLearners Students Predictor%SpecialEd Students Predictor

%Billingual/ESL Students Predictor%CareerTechnical Students Predictor%GiftedTalented Students Predictor

AttendenceRate PredictorDropOutRate Predictor

4YearGradRate ResponseSTAAR%Level2orHigherALL (Standardized tests) Predictor

%CollegeAdmitted PredictorSATAverage PredictorACTAverage Predictor

SalaryTeacherAvg PredictorStudentsPerTeacher Predictor

Teacher%5orFewerYrsExp PredictorTeacherAvgYearExp PredictorTeacherTurnoverRate Predictor

RevenuePerPupil PredictorExpendituresActualPerPupil Predictor

Expenditures%Athletics Predictor

8

a. In JMP, use Analyze > Multivariate Methods and obtain a correlation matrix that includes the response, all the potential predictors identified above.

Identify on the table below which predictor variables, if any, will be removed due to concerns of multicollinarity. (5 pts)

Varaible Role

Removed because of Multicollinearity?

(Yes / No)DistictName Demographic  

CountyNumberName Demographic  Region Demographic  

DistictSize Demographic  CommunityType Demographic  WealthIndictor Demographic  

TAXRATE Demographic  AccountabilityRating Demographic  %AfricianAmerican Predictor  

%Hispanic Predictor  %White Predictor  

%AmericanIndian Predictor  %Asian Predictor  

%EconomicDisadvantaged Predictor  %EnglishLanguageLearners Predictor  

%SpecialEd Predictor  %Billingual/ESL Predictor  

%CareerTechnical Predictor  %GiftedTalented Predictor  AttendenceRate Predictor  

DropOutRate Predictor  4YearGradRate Response  

STAAR%Level2orHigherALL Predictor  %CollegeAdmitted Predictor  

SATAverage Predictor  ACTAverage Predictor  

SalaryTeacherAvg Predictor  StudentsPerTeacher Predictor  

Teacher%5orFewerYrsExp Predictor  TeacherAvgYearExp Predictor  

TeacherTurnoverRate Predictor  RevenuePerPupil Predictor  

ExpendituresActualPerPupil Predictor  Expenditures%Athletics Predictor  

b. Use the Minimum AICc criterion with Forward selection to obtain a preliminary list of the necessary predictor variables. Provide a screen capture of the Step History portion of the output below. Also, in the table below identify the preliminary importance of each predictor variable via this approach. ( 5 pts)

c. Use the P-value Threshold criterion with Mixed (or Stepwise) selection. Use 0.25 for Prob to Enter and 0.10 for Prob to Leave. Once again, provide a screen capture of the Step History portion of the output and again identify the preliminary importance of each predictor variable via this approach in the table below. (5 pts)

9

Varaible Role

Was this predictor in your final model

when Minimum AICc was used?

Was this predictor in your final model

when P-Value Threshold was used?

%AfricianAmerican Predictor  %Hispanic Predictor  

%White Predictor  %AmericanIndian Predictor  

%Asian Predictor  %EconomicDisadvantaged Predictor  

%EnglishLanguageLearners Predictor  %SpecialEd Predictor  

%Billingual/ESL Predictor  %CareerTechnical Predictor  %GiftedTalented Predictor  AttendenceRate Predictor  

DropOutRate Predictor  4YearGradRate Response  

STAAR%Level2orHigherALL Predictor  %CollegeAdmitted Predictor  

SATAverage Predictor  ACTAverage Predictor  

SalaryTeacherAvg Predictor  StudentsPerTeacher Predictor  

Teacher%5orFewerYrsExp Predictor  TeacherAvgYearExp Predictor  

TeacherTurnoverRate Predictor  RevenuePerPupil Predictor  

ExpendituresActualPerPupil Predictor  Expenditures%Athletics Predictor  

d. Discuss any differences between Minimum AICc Forward, and the P-Value Threshold – Mixed selections procedures. (3 pts)

10

e. Decide on a final model. Identify the predictors you are going to use in your final model in the table below.

Varaible Role

Identify which predictor variables will be used in

my final model.%AfricianAmerican Predictor  

%Hispanic Predictor  %White Predictor  

%AmericanIndian Predictor  %Asian Predictor  

%EconomicDisadvantaged Predictor  %EnglishLanguageLearners Predictor  

%SpecialEd Predictor  %Billingual/ESL Predictor  

%CareerTechnical Predictor  %GiftedTalented Predictor  AttendenceRate Predictor  

DropOutRate Predictor  4YearGradRate Response  

STAAR%Level2orHigherALL Predictor  %CollegeAdmitted Predictor  

SATAverage Predictor  ACTAverage Predictor  

SalaryTeacherAvg Predictor  StudentsPerTeacher Predictor  

Teacher%5orFewerYrsExp Predictor  TeacherAvgYearExp Predictor  

TeacherTurnoverRate Predictor  RevenuePerPupil Predictor  

ExpendituresActualPerPupil Predictor  Expenditures%Athletics Predictor  

f. Obtain the regression output for your final model. Does this model do a decent

job of predicting 4 year graduation rate? Discuss. (3 pts)

g. Use the added variable plots (or as JMP calls them Leverage plot) to identify the most important predictor variable in your model. Copy the added variable plot below. Discuss. (2 pts)

11

h. Consider the information regarding Cook’s Distance from Handout #9. Cook’s Distance can be easily obtained in JMP by selecting Save Columns > Cook’s D Influence from the red-down menu.

Cook’s DistanceCook (1977) developed a measure of the effect of an individual observation by combining the magnitude of the internally studentized residual with the magnitude of leverage for this observations. This statistic is simply referred to as Cook’s Distance or Cook’s D.

Coo k ' s Di=( student ri )

2

(k+1)⏟residual

∗hi

(1−hi)⏟

leveragewhere

ri=internally studentized residual hi=leverage k=number of predictors∈model

Suggested Rules for Cook’s Distance An observation whose Cook’s Distance is substantially larger than others should be

investigated Cook suggests it is always important to investigate observations whose Cook’s D > 1

Remove any observations that have a Cook’s Distance larger than 1 from your dataset. Refit the model. Provide the updated regression output here. In what ways is this model different than your final model you obtained above? (3 pts)

g. Next, consider an investigation of the studentized residuals for your model. These can be easily obtained by selecting the red drop-down menu and selecting Save Columns > Studentized Residuals.

Studentized Residual i=e i−0

√ σ2∗(1−h ii)

Any observation whose studentized residual is less than -2.0 or larger than 2.0 would indicate a poor prediction. In particular, a studentized residual less than -2.0 would indicate a substantial over-prediction and a studentized residual more than 2.0 would indicate a substantial under-prediction for 4 year graduation rate by the model. Identify all school districts (using their name) for the following situations. (5 pts)

School districts for which the model substantially over predicted:

School districts for which the model substantially under predicted:

12

h. Create a plot that has the predicted 4 year graduation rate on the x-axis and the actual 4 year graduation rate on the y-axis. This provides a visualization of the R2 value from our model. Add the y=x line to this plot for reference.

Does the model appear to performing well? Discuss. (3 pts)

13

6. Consider again the cross-validation procedures discussed in Handout #18. For this problem we will again use the Grandfather_Clocks dataset as was done in this handout.

Recall, the following visualization for the Train / Test Cross-Validation, i.e. the simple 2-fold cross-validation approach discussed in Handout #18.

Train / Test, i.e. 2-Fold, Cross ValidaitonDivide data into 2-folds or 2 parts; Training dataset is used to build model; Test dataset is used to obtain RMSE to measure predictive ability.

The following describes the k-fold cross-validation approach. The Grandfather_Clocks dataset has 32 observations, so an 8-fold cross-validation will be used. Thus, each fold will consist of 4 observations each.

Iteration k-Fold Approach0 Create k unique subsets of the data1 Folds 1, 2, 3, 4, 5, 6, 7, __ are

used to create Training dataset; Fold 8 is used as Test dataset; Obtain RMSE using observations in Fold 8.

2 Folds 1, 2, 3, 4, 5, 6, __ , 8 are used to create Training dataset; Fold 7 is used as Test dataset; Obtain RMSE using observations in Fold 7.

3-7 Repeat process for Folds 3 - 78 Folds __, 2, 3, 4, 5, 6, 7, 8 are

used to create Training dataset; Fold 1 is used as Test dataset; Obtain RMSE using observations in Fold 1.

a. Type the following into R.

What is the difference between folds and random.folds? Discuss (3 pts)

14

b. Suppose the name of the data frame that contains the Grandfather Clock dataset is Grandfather_Clocks. Type the following in R (or something similar if your data frame has a different name).

What is this doing to the Grandfather_Clocks dataset? Discuss (3 pts).

c. The following sequence of commands will obtain the RMSE using Fold #1 as the Test dataset.

What is the predictive RMSE when Fold #1 is used for the Test dataset? (2 pts)

Predictive RMSE: _____________

d. Repeat part c for each of the 8 folds. Enter the predictive RMSE from each Fold in the table below. (4 pts)

Fold used for Test dataset

Predictive RMSE

12345678

e. Obtain the average predictive RMSE value for the k=8 fold cross-validation approach used here.

Average Predictive RMSE: ______________

f. The RMSE from the full model (when no cross-validation is used) is 133.1. Is the average predictive RMSE larger than 133.1? Why would we expect this to be the case? Discuss. (2 pts)

15