Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker

30
Office of Institutional Research Antelope Valley College Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker Office of Institutional Research and Planning Antelope Valley College [email protected]

description

Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker Office of Institutional Research and Planning Antelope Valley College [email protected]. Note From Ted: - PowerPoint PPT Presentation

Transcript of Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker

Office of Institutional Research

Antelope Valley College

Probability Based Advising for Basic Skills Courses

By

Ted Younglove and Aaron Voelcker

Office of Institutional Research and PlanningAntelope Valley College

[email protected]

Office of Institutional Research

Antelope Valley College

Note From Ted:

If you would like to try the ‘cheat sheet’ part of this project I can send you an example file for the data, and the data can be run for an

hourly fee ($105/hr) by:

Scott M Lesch, Ph.D. Principal Consulting Statistician C&C / Statistical Consulting

Collaboratory University of California Riverside [email protected]

Office of Institutional Research

Antelope Valley College

Modeling

Prediction

Intervention

Evaluation

Office of Institutional Research

Antelope Valley College

Modeling and Prediction Update

• Predictive models for success and persistence

developed in 2006 using

• SAS Stepwise Discriminant Analysis

• 11 parameter models and

• 4 parameter models

• Persistence defined as continued enrollment from

Fall to Spring

• Success defined as Success in all courses taken

(A,B,C,P,CR)

Office of Institutional Research

Antelope Valley College

Modeling and Prediction Update

• 4 parameter model selected for ease of use

• Success

• Age at start of term

• Ethnicity Black (Yes/No)

• Enrolled in at least one basic skills course

(Yes/No)

• Units completed beyond 30

Office of Institutional Research

Antelope Valley College

Modeling and Prediction Update

• Validated in Fall 2007 on independent data

PersistenceActual (Fall to Spring)

Prediction Probability Group Prediction Group 0 1 % CorrectPr (0) .75 - 1.00 Strong No Return (0) 100 50 67%Pr (0) .50 - .749 Weak No Return (0) 2472 2664 48%Pr (0) .25 - .499 Weak Return (1) 1758 5354 75%Pr (0) 0.00 - .249 Strong Return (1) 161 709 81%

SuccessAll Students Actual (Fall Success)Prediction Probability Group Prediction Group 0 1 % CorrectPr (0) .75 - 1.00 Strong Unsuccessful (0) 574 139 81%Pr (0) .50 - .749 Weak Unsuccessful (0) 4169 2940 59%Pr (0) .25 - .499 Weak Successful (1) 1722 3009 64%Pr (0) 0.00 - .249 Strong Successful (1) 191 524 73%

Office of Institutional Research

Antelope Valley College

Modeling and Prediction Update

• Validated in 2008 on independent data.

Persistence (Fall to Spring)Prediction Probability Group Prediction Group 0 1 % CorrectPr (0) .75 - 1.00 Strong No Return (0) 305 116 72%Pr (0) .50 - .749 Weak No Return (0) 3380 3476 49%Pr (0) .25 - .499 Weak Return (1) 1173 5719 83%Pr (0) 0.00 - .249 Strong Return (1) 90 798 90%

SuccessPrediction Probability Group Prediction Group 0 1 % CorrectPr (0) .75 - 1.00 Strong Unsuccessful (0) 687 75 90%Pr (0) .50 - .749 Weak Unsuccessful (0) 6497 1341 83%Pr (0) .25 - .499 Weak Successful (1) 2271 3397 60%Pr (0) 0.00 - .249 Strong Successful (1) 197 592 75%

Office of Institutional Research

Antelope Valley College

Prediction Works!

• So what do we do about it?

• More of the same?

• New efforts?

Office of Institutional Research

Antelope Valley College

Intervention

• Many (90% Fall 2009) students enter AVC with reading, writing and/or math skills that are below college level,

• Students are placed into courses by their performance on an entrance exam,• 3 levels of English (ENGL 095, 097, and 099)• 2 levels of Reading (READ 097 and 099)• 3 levels of Math (MATH 050, 060, 070)

• Students may be placed into one or more Basic Skills courses.

Office of Institutional Research

Antelope Valley College

Intervention

• Lack of success in Basic Skills courses is a significant impediment to persistence and success at AVC,

• Given that a new student tests into a specific Basic Skills course and can successfully pass this course, what other college level courses can this student concurrently enroll in and pass?

• Can we help these students by providing guidance to counselors and students based on past students success (or lack of success)?

Office of Institutional Research

Antelope Valley College

Intervention

• Our solution to the problem posed previously is a Logistic Regression model (specifically a Logistic ANOCOVA),

• Logistic regressions are used to predict probabilities of occurrence of binary variables,

• Frequently used in medical research and marketing,• Predicting the effect smoking has on the

probability of a heart attack,• Predicting the probability a customer will purchase

a product.

Office of Institutional Research

Antelope Valley College

Intervention

• One possible alternative solution: simply calculate the percent success in concurrent classes for students in the different basic skills courses,

• Logistic regression chosen to provide a statistical framework.

Office of Institutional Research

Antelope Valley College

Intervention

• In our case, we are interested in the covariates, the courses taken with the Basic Skills course.

• For ENGL095, ENGL097, ENGL099 the model would be:

Represents the global mean

Represents the specific non ENGL course effect

Represents the adjustment effects of placement into ENGL095 and ENGL097

Represents the effect of passing the ENGL course

Office of Institutional Research

Antelope Valley College

Intervention

• Past data was used to estimate the parameters of the logistic equation,

Office of Institutional Research

Antelope Valley College

Intervention

• The model was estimated using SAS proc logistic,

• Minimum sample size for a course to be included was 30,

• Two important assumptions:

Individual student effect assumed to be random and negligible,

Individual instructor effect assumed to be random and negligible,

• Because of the large number of students and instructors these effects can not be easily estimated.

Office of Institutional Research

Antelope Valley College

Intervention

• Once the model has been estimated, the parameter estimates are then be used to calculate the specific probabilities for passing each analyzed secondary course for each Basic Skills English level,

• All probabilities are estimated for the case where the student has passed the ENGL course.

Office of Institutional Research

Antelope Valley College

Intervention

Counseling ‘Cheat Sheets’

• ‘Cheat Sheets’ were produced to help counselors and students in selecting courses to improve success in the other courses,

• It is hoped that by improving selection of concurrent courses success will improve in ENGL as well,

• The project has been implemented in the Intersession 2009 and Spring 2009 registration period.

Office of Institutional Research

Antelope Valley College

Intervention

Counseling ‘Cheat Sheets’

• 122 concurrent courses had sufficient data for estimates for ENGL 095, 097, and 099

• 108 concurrent courses had sufficient data for estimates for MATH 050, 060, and 070

• 20 concurrent courses had sufficient data for estimates for READ 097, and 099

Office of Institutional Research

Antelope Valley College

InterventionExample: ENGL095

Rank Course Pass Probability Rank Course Pass Probability1 PE 220 98% 62 MATH 050 60%2 PE 120 96% 63 BUS 101 59%3 MUS 126 92% 64 ENGR 115 59%4 PE 115 92% 65 HD 101 59%5 PE 140 91% 66 MOA 101 59%6 AJ 203 88% 67 MATH 050A 58%7 PE 101 88% 68 MATH 060 58%8 PE 175 87% 69 MUS 131 57%9 ART 113 86% 70 BUS 105 56%

10 HIST 111 84% 71 BUS 201 56%11 BUS 199 83% 72 THA 101 56%12 PE 170 83% 73 COMM 101 55%13 MATH 020 82% 74 HIST 102 54%14 DFST 102 81% 75 NF 102 54%15 MUS 121 81% 76 SOC 101 54%16 PE 160 81% 77 BUS 212 53%17 PE 135 80% 78 AJ 205 52%18 PHOT 101L 80% 79 CT 050 52%19 ART 110 79% 80 PHIL 105 52%20 EOPS 060 79% 81 CHEM 101 50%21 PE 150 79% 82 HIST 113 50%22 CG 101 78% 83 MATH 115 49%23 CHEM 101L 78% 84 PSY 101 49%24 PE 103 78% 85 ASTR 101 48%25 ASTR 101L 77% 86 CIS 101 48%26 DA 103 77% 87 COMM 103 48%27 HIST 110 76% 88 MUS 103 48%28 PE 109 75% 89 MATH 070A 47%29 PHOT 101 75% 90 POLS 101 47%30 AJ 101 74% 91 ART 100 46%31 CG 101L 74% 92 FTV 101 46%32 HD 103 74% 93 MUS 104 46%33 PE 155 74% 94 NF 100 46%34 PE 191 74% 95 OT 101 46%35 ID 100 72% 96 GEOG 101 45%36 THA 110 72% 97 MATH 135 45%37 DA 104 71% 98 PHIL 106 44%38 DA 106 71% 99 PSY 055 44%39 MATH 050C 71% 100 CFE 101 43%40 CFE 102 70% 101 BIOL 101L 41%41 ART 140 69% 102 ECON 102 41%42 CA 103 68% 103 MATH 070 41%43 MGT 101 68% 104 MATH 102 41%44 AJ 102 67% 105 ART 102 39%45 MKTG 101 67% 106 HIST 107 39%46 PSY 212 67% 107 ECON 101 38%47 AJ 103 66% 108 MATH 100 37%48 ED 140 66% 109 BIOL 101 36%49 HD 100 66% 110 HIST 108 36%50 MATH 100A 66% 111 MATH 130 35%51 PE 102 66% 112 AUTO 100 34%52 READ 099 66% 113 HIST 104 34%53 SPAN 101 66% 114 MATH 080 33%54 FTEC 111 65% 115 RE 101 33%55 HD 105 65% 116 BIOL 100 29%56 READ 097 65% 117 GEOL 101 28%57 DFST 101 64% 118 HIST 101 28%58 ENGL 066 63% 119 MUS 101 27%59 GEOG 105 62% 120 PSCI 101 22%60 HE 101 61% 121 ART 101 18%

Office of Institutional Research

Antelope Valley College

Intervention

Discussion:

What guidance do you give to counselors?

Office of Institutional Research

Antelope Valley College

Modeling

Prediction

Intervention

Evaluation

Office of Institutional Research

Antelope Valley College

Evaluation

Intersession/Spring 2009 Plan

• Test for changes in registration pattern,

• Test for increase in percent success• Overall• Basic Skills

• Test for differential effect on students predicted not likely to succeed.

Office of Institutional Research

Antelope Valley College

Evaluation

Intersession/Spring 2009 -Complications

• New variable created for tracking which students were advised using new method was not used consistently,

• Consistency in identification of students provided counseling lacking,• Counseling = 1, student received

counseling during this term, probably advised using ‘cheat sheets’

• Counseling = 0, student probably did not receive counseling during this term, probably not advised using ‘cheat sheet’.

Office of Institutional Research

Antelope Valley College

Evaluation (Registration Behavior)

Spring 2009 - ENGL

Group Count 2008 Count 2009 2008 (%) 2009 (%)Best (.70-1.0) 145 38 77% 73%Middle (.50-.69) 36 14 19% 27%Worst (<.50) 7 0 4% 0%

188 52

Group Count 2008 Count 2009 2008 (%) 2009 (%)Best (.70-1.0) 255 77 89% 94%Middle (.50-.69) 31 4 11% 5%Worst (<.50) 1 1 0% 1%

287 82

Group Count 2008 Count 2009 2008(%) 2009(%)Best (.70-1.0) 17 9 18% 19%Middle (.50-.69) 67 33 70% 69%Worst (<.50) 12 6 13% 13%

96 48

095

097

099

Office of Institutional Research

Antelope Valley College

Evaluation (Registration Behavior)

Spring 2009 - MATH

050

060

070

Group Count 2008 Count 2009 2008 (%) 2009 (%)Best (.70-1.0) 277 109 87% 84%Middle (.50-.69) 35 20 11% 16%Worst (<.50) 7 0 2% 0%

319 129

Group Count 2008 Count 2009 2008 (%) 2009 (%)Best (.70-1.0) 217 129 98% 98%Middle (.50-.69) 3 3 1% 2%Worst (<.50) 2 0 1% 0%

222 132

Group Count 2008 Count 2009 2008 (%) 2009 (%)Best (.70-1.0) 219 80 97% 100%Middle (.50-.69) 7 0 3% 0%Worst (<.50) 0 0 0% 0%

226 80

Office of Institutional Research

Antelope Valley College

Evaluation (Registration Behavior)

Spring 2009 – ENGL 099

ENGL 099Group Counseling 1 Counseling 0

Best (.70-1.0) 94% 57%Middle (.50-.69) 5% 40%Worst (<.50) 1% 3%

Group Cheatsheet No CheetsheetBest (.70-1.0) 94% 89%Middle (.50-.69) 5% 11%Worst (<.50) 1% 0%

Office of Institutional Research

Antelope Valley College

Evaluation

Intersession/Spring 2009 -Complications

• Additional evaluation suggestions?

• Two focus areas:• Change in registration behavior,• Change in Success percentage.

Office of Institutional Research

Antelope Valley College

Evaluation

Spring 2009

After the end of the term:

• Are persistence rates higher in the students who were advised using the ‘cheat sheets’?

• Are success rates higher in the students who were advised using the ‘cheat sheets’?

Office of Institutional Research

Antelope Valley College

Conclusions

Spring 2009

• LANOCOVA analysis provides a workable way to estimate pass probabilities for concurrent courses,

• Adoption by counselors is under way and leading to changes in registration behavior,

• Effects on success may be difficult to estimate on Spring data.

Office of Institutional Research

Antelope Valley College

Discussion

Spring 2009

• Suggestions on improving use of the ‘cheat sheets’?,

• Suggestions on analysis of success?

• Other courses?