USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY...
-
Upload
kellie-rose -
Category
Documents
-
view
217 -
download
0
Transcript of USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY...
![Page 1: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/1.jpg)
USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA
Bobbie E. FryeBobbie E. FryeCentral Piedmont Community CollegeCentral Piedmont Community CollegeDr. James E. Bartlett Dr. James E. Bartlett North Carolina State UniversityNorth Carolina State University
The Council for the Study of Community Colleges
![Page 2: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/2.jpg)
AgendaAgenda
IntroductionIntroduction Statement of Problem/PurposeStatement of Problem/Purpose Research QuestionsResearch Questions Theoretical FrameworkTheoretical Framework MethodsMethods AnalysesAnalyses ResultsResults ContributionsContributions ConclusionsConclusions Future ResearchFuture Research
![Page 3: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/3.jpg)
Introduction
Remedial [developmental] courses are a fixture in the American community college• Allow students that are not prepared for college level work to enroll in college
programs• Annual cost of remediation: 1.9 – 2.3 billion dollars
Placement in developmental course• Varies by institution (i.e. Placement scores, Prerequisites, Statewide policies, Program
requirements, etc.) Remedial Evidence
• Achieving the Dream (ATD) a national community college student success initiative which targets low income and under-represented populations, found that 62% of full-time students in community colleges needed developmental math.
• Data analyses of ATD colleges have revealed consistent patterns of entering students who require developmental education, and who experience low completion rates in developmental courses (ATD, 2010).
• A National Educational Longitudinal Study (NELS) followed 8th grade students from 1988 to the year 2000, and found that 58% of those students who attended a community college took at least one remedial course, 44% took between one and three courses, and 14% took more than three remedial courses (Attewell, Domina, Lavin & Levey, 2006).
![Page 4: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/4.jpg)
Nature of ProblemNature of Problem
Developmental math education: it’s a Developmental math education: it’s a gauntlet that many students run but few gauntlet that many students run but few students (25%) emerge from successfully students (25%) emerge from successfully (Bailey, 2009, Clery, 2009).(Bailey, 2009, Clery, 2009).
About 60% of new students place into About 60% of new students place into developmental math (Bailey, et al., 2010).developmental math (Bailey, et al., 2010).
““Though a primary point of access to higher Though a primary point of access to higher education, community colleges struggle to education, community colleges struggle to ensure that all of their students earn a ensure that all of their students earn a credential” (Kolenovic, Linderman, & Karp, credential” (Kolenovic, Linderman, & Karp, 2013, p.272).2013, p.272).
In addition to financial costs, there are social In addition to financial costs, there are social and personal costs for students who move and personal costs for students who move through the developmental education through the developmental education sequences (Bailey et al., 2010).sequences (Bailey et al., 2010).
![Page 5: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/5.jpg)
Past ResearchPast Research
The studies have methodological The studies have methodological limitations to particular states and limitations to particular states and sub-populations of students. sub-populations of students.
Failure to control for selection bias is Failure to control for selection bias is a common concern in the study of a common concern in the study of developmental education developmental education (Bettinger & Long, 2005). (Bettinger & Long, 2005).
So far, results have been mixed. So far, results have been mixed.
![Page 6: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/6.jpg)
Statement Of ProblemStatement Of Problem
Researchers acknowledge the limitations of comparing student performance, progression, or retention in a non-scientific study wherein participants are not randomly assigned nor even equivalent in terms of motivation, intentions, background, or skill level (Titus, 2007).
Comparisons of student outcomes using propensity matching has been used to yield less biased results than are derived using simple raw comparisons (Rojewski et al., 2009).
Proposed remedies to self-selection bias in educational research need to be explored in terms of feasibility and usefulness, as it is critical that researchers apply methodologies that control for selection bias. Resources are scarce and monies need to be allocated to educational programs that are making a difference in students’ success, retention rates, and long-term outcomes.
![Page 7: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/7.jpg)
What is the Purpose?What is the Purpose?
The purpose of this study is to determine if there is a significant The purpose of this study is to determine if there is a significant difference in characteristics and college level outcomes between difference in characteristics and college level outcomes between developmental math students who complete college math developmental math students who complete college math coursework compared to students who do not complete math coursework compared to students who do not complete math coursework.coursework.
Students referred to developmental math are the participants, Students referred to developmental math are the participants, defined as students referred into one or more areas of defined as students referred into one or more areas of developmental math coursework and enrolled in at least one developmental math coursework and enrolled in at least one developmental math course during the study period. The developmental math course during the study period. The population consists of 2007-08 new student cohorts at seven NC population consists of 2007-08 new student cohorts at seven NC community colleges. community colleges.
![Page 8: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/8.jpg)
Purpose
The purpose of this study was to determine if there is a difference in outcomes of grade point average, credits earned, credentials
earned, transfer, and retention between two groups.
Comparison Group:Group of students who
complete developmental math coursework, attempt but do no
complete college level math with a C or better
Study Group:Group of students who
complete developmental math coursework and subsequently college level math with a C or
better
Propensity matching was used to create two equivalent groups of students matched on the propensity to
complete college level math
![Page 9: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/9.jpg)
Research QuestionsResearch Questions
Prior to propensity matching, is there a difference in Prior to propensity matching, is there a difference in student and academic characteristics between student and academic characteristics between developmental students who complete college math developmental students who complete college math coursework compared to students who do not complete coursework compared to students who do not complete math coursework?math coursework?
Prior to propensity matching, is there a difference in Prior to propensity matching, is there a difference in student and academic characteristics between student and academic characteristics between developmental students who complete college math developmental students who complete college math coursework compared to students who do not complete coursework compared to students who do not complete math coursework between institutions?math coursework between institutions?
After propensity matching, is there a difference in After propensity matching, is there a difference in college level outcomes between developmental college level outcomes between developmental students who complete college math coursework students who complete college math coursework compared to students who do not complete math compared to students who do not complete math coursework?coursework?
After propensity matching, is there a difference in After propensity matching, is there a difference in college level outcomes between developmental college level outcomes between developmental students who complete college math coursework students who complete college math coursework compared to students who do not complete math compared to students who do not complete math coursework between institutions ?coursework between institutions ?
![Page 10: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/10.jpg)
Theoretical Framework – Theoretical Framework – 10
Student retention theoriesStudent retention theories Tinto – Academic and social integration Tinto – Academic and social integration
Behaviors (1993)Behaviors (1993)
Bean and Metzner Academic integration Bean and Metzner Academic integration and environmental factors – Beliefs (1995) and environmental factors – Beliefs (1995)
![Page 11: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/11.jpg)
![Page 12: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/12.jpg)
![Page 13: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/13.jpg)
Four Models Examined
Variable Name Model IStudent
Demographics
Model IIAcademic Integration
Model IIIIndividual &
Institution
Model IVIndividual &
InstitutionGender X X X XAge X X X XRace* X X X XTransfer Program X X X XPell Recipient X X X XFull-Time in 1st Term X X X XFirst Dev. Math Level X X X XRetained 2nd term X X XDev. English X X XSuccess Course X X XFirst Term GPA X X X% Minority** X% Pell Recipients** X% Dev. English *** X % Dev. Math *** XInstitution Member X
*Race categories: White, African American, Hispanic, International, Asian, Other ** Number of students as a percentage of fall 2007 enrollment***Number of students as a percentage of 2007-08 institutional FTE
![Page 14: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/14.jpg)
MethodsMethods Non-Experimental (Propensity Score Matching)Non-Experimental (Propensity Score Matching)
Study Group 5,548 students who placed into Study Group 5,548 students who placed into developmental math coursework and of those 2,102 (38%) developmental math coursework and of those 2,102 (38%) of students who subsequently attempted college level of students who subsequently attempted college level mathmath
Purposeful sampling techniquePurposeful sampling technique
VariablesVariables Matching variables-IndependentMatching variables-Independent Grouping (Variable-A-C Completion/Non-Completion College Grouping (Variable-A-C Completion/Non-Completion College
Math)Math) Dependent (Student Outcomes)Dependent (Student Outcomes)
Data Collection- Student term records Fall 2007-Summer Data Collection- Student term records Fall 2007-Summer 20122012
Data AnalysisData Analysis Multi-Level Propensity Score MatchingMulti-Level Propensity Score Matching
![Page 15: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/15.jpg)
There are six essential steps to PSM:
Data pre-screening Covariate identification Propensity score estimation Matching of propensity scores Determination of matching success Presentation of results
![Page 16: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/16.jpg)
Analyses
To analyze whether there is a difference in demographics and academic characteristics of the two study groups prior to propensity score matching t-tests will be used. Means, standard deviation, significance levels of p<=05, t values and cohen’s d (effect size) will be reported using t-tests analyses (Rojewski, et al., 2009; 2010).
Bias percentages (standardized effect sizes) before and after matching were also reported (Oakes & Johnson, 2006). The standardized percent bias is the percent difference of the sample means in the treated and non-treated (full or matched) sub-samples as a percentage of the square root of the average of the sample variances in the treated and non-treated groups (Rosenbaum & Rubin, 1985)
Logistic regression will also be used to determine the background and academic factors that explain membership in the two groups of study and to create a propensity score that is used in the matching technique. Logistic stepwise regression will yield significant independent variables that predict membership in the control or treatment groups. Nakelkerke R-squared, chi-squared, beta coefficients and independent variables with p value <=.05 were reported and retained in the model (Rojewski, et al., 2009; 2010).
Using the propensity score generated through the logistic regression technique, equivalent groups of students were propensity matched using a matching procedure in STATA software (Titus, 2007). The technique known as psmatch2 creates two matched groups based on the propensity to complete college level math.
![Page 17: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/17.jpg)
Logistic Regression Results (Sig. prior to Match)Covariates Model I Model
IIModel III
Model IV
DV: Complete CL Math Exp(B) Exp(B) Exp(B) Exp(B)
Gender (Female)1.411** 1.369** 1.377** 1.345**
Age1.018*
African American 0.532** 0.607* 0.487*** 0.484***
Pell Recipient 0.736** 0.736** 0.756* 0.754*
First Term GPA 1.235*** 1.254*** 1.235***
% Minority 1.055*** NA
% Pell Awarded 0.945* NA
% Dev English 0.914* NA
College B 1.350*
College F 2.963***
College G 0.459***
R-Squared 0.044 0.075 0.090 0.115
Notes: *p < .05 **p < .01 ***p < .001
![Page 18: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/18.jpg)
Model IV T-Tests Prior to Matching(Sig. variables)
![Page 19: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/19.jpg)
Model IV T-Tests Prior to Matching (continued)
Covariates N M SD t p d % bias
College B 539 0.345 0.475 2.410 0.016* 0.122 12.100
1,563 0.404 0.490
College F 539 0.043 0.202 5.740 0.000*** 0.320 31.800
1,563 0.131 0.338
College G 539 0.134 0.341 -4.690 0.000*** -0.219 -21.700
1,563 0.068 0.253
Average BiasR-Squared
13.4000.072
Cohen’s d = Mt – Mc / σpooled; where σpooled = √ σ2t + σ2c / 2 *p < .05 **p < .01 ***p < .001
First line of each variable is the control group; second line is the treatment group.
![Page 20: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/20.jpg)
Model IV T-Tests After Matching
![Page 21: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/21.jpg)
Model IV T-Tests After Matching (Continued)
CovariateN M SD t p d % bias
Full-Time 512 0.656 0.475 0.351 0.725 0.023 2.600
507 0.667 0.472First Dev. Math 512 1.984 0.735 0.256 0.798 0.016 1.400
507 1.996 0.722
Dev. Eng 512 0.453 0.498 0.206 0.837 -0.012 3.900
507 0.459 0.499
Success 512 0.359 0.480 -0.342 0.732 -0.021 -0.900
507 0.349 0.4771st Term GPA 512 2.141 1.460 -0.609 0.543 -0.038 -7.200
507 2.084 1.556Retained to 2nd 512 0.901 0.300 -0.156 0.875 -0.013 -1.200
507 0.897 0.304
![Page 22: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/22.jpg)
Model IV T-Tests After Matching (Continued) Covariate N M SD t p d % bias
College B 512 0.359 0.480 -0.541 0.589 -0.034 -4.200
507 0.343 0.475
College C 512 0.051 0.220 -0.556 0.578 -0.038 -4.600
507 0.043 0.204
College D 512 0.008 0.088 -0.070 -0.366 -0.024 -2.200
507 0.006 0.077
College E 512 0.043 0.203 -0.123 0.902 -0.010 -1.500
507 0.041 0.199
College F 512 0.049 0.207 -0.597 0.550 -0.060 -3.200
507 0.037 0.190
College G 512 0.121 0.326 0.430 0.667 -0.028 0.400
507 0.112 0.316
Average Bias R-Squared
3.300
0.063
Cohen’s d = Mt – Mc / σpooled; where σpooled = √ σ2t + σ2c / 2 *p < .05 **p < .01 ***p < .001First line of each variable is the control group; second line is the treatment group.
![Page 23: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/23.jpg)
Summary Before and After Matching
Bias Pre
Bias Post
%Change
R2
PreR2
PostChange
Sig. CV’s Pre
Sig. CV’s Post
Change
Model I 10.4% 10.3% -0.9% 0.020 0.041 +0.021 4 3 -1
Model II 12.5% 4.20% -66.4% 0.046 0.040 -0.006 6 0 -6
Model III 13.0% 2.80% -78.5% 0.061 0.050 -0.011 8 1 -7
Model IV 13.4% 3.30% -75.4% 0.072 0.063 -0.090 8 0 -8
![Page 24: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/24.jpg)
Final AnalysesFinal Analyses
After matching, to determine if there a difference in college level outcomes between developmental math students who complete college math coursework compared to students who do not complete math coursework, T-Tests were used. Means, standard deviation, significance levels of p<=.05, t values and cohen’s d were reported using t-tests analyses (Rojewski, et al., 2010).
Completion outcomes were categorical and chi-square analyses were used to determine the measures of association between the students who completed college math coursework compared to students who did not complete math coursework in terms of completion categories (Rojewski, et al, 2010).
![Page 25: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/25.jpg)
Model IV T-Tests Outcomes after Matching
College Courses
N M SD t p dCredits Attempted 512 51.259 27.090 10.083 0.000*** 0.632
507 67.852 25.406Credits Completed 512 31.516 23.827 16.453 0.000*** 1.031
507 55.884 23.447Credits A-C
512 26.750 21.654 17.905 0.000*** 1.122
507 51.525 22.511Retention
Terms Retained
512 6.139 2.785 8.245 0.000*** 0.516
507 7.524 2.576
![Page 26: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/26.jpg)
Model IV T-Tests Outcomes after Matching
Outcomes N M SD t p dDevelopmental CoursesCredits Attempted 512 13.504 8.894 -0.565 0.572 -0.035
507 13.191 8.761Credits Completed 512 10.080 7.133 2.169 0.030* 0.136
507 11.087 7.673Credits A-C
512 9.402 6.834 2.189 0.029* 0.137
507 10.368 7.258Developmental Math
Credits Attempted 512 8.101 4.603 -0.871 0.384 -0.054
507 7.854 4.470Credits Completed 512 5.906 3.752 2.036 0.042* 0.128
507 6.379 3.652Credits A-C
512 5.398 3.469 2.019 0.044* 0.126
507 5.826 3.295
![Page 27: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/27.jpg)
Model IV Completion OutcomesCompletion Categories
Passed College-Level Math
Grade of C or Better
Row Percent
Not Passed College-Level Math
Row Percent
Total
No Outcome Observed 133 29.23% 322 70.77% 455
Expected 226.4 228.6 455.0
Associate/Certificate Observed 81 77.88% 23 22.12% 104
Expected 51.7 52.3 104.0
Transfer 2 Year Observed 44 72.13% 17 27.87% 61
Expected 30.4 30.6 61.0
Transfer 4 Year Observed 177 70.52% 74 29.48% 251
Expected 124.9 126.1 251.0
Still Enrolled 30 Plus Credits
Observed 72 48.65% 76 51.35% 148
Expected 73.6 74.4 148.0
Total 49.75% 50.25% 1019
TotalAlpha= 0.050Chi-Squared= 165.159Degrees of Freedom= 4p value < .001Cramer’s V= 0.403
![Page 28: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/28.jpg)
Conclusions
After matching, the average means differ between the two groups on key progress indicators.
Specifically, the study group fared better than the comparison group and completed, on average, 25 more college credits at the community college.
In addition, the study group was retained an average of 1.5 more semesters than the comparison group.
![Page 29: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/29.jpg)
Conclusions (Continued)
In terms of expected versus observed frequencies, a higher number (71%) of students in the non-completion group earned no award compared to students in the completion group.
Completers of college math with C or better earned significantly more associate degrees (78%) than non-completers.
However, in the four models, non-completers of college level math were twice as likely to earn no outcome after five years when compared to the completer group.
Transfers to four-year and two-year institutions were common in both groups of students, and the math completion group was twice as likely to transfer out of the institution.
![Page 30: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/30.jpg)
Contributions
Applied propensity score matching methods across multiple community colleges
Demonstrated that students did not differ to a large extent
Demonstrated that propensity matching reduced the bias at least 75% for the multi-level model
Demonstrated the need to conduct additional research and examine variations among colleges
![Page 31: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/31.jpg)
Significance of StudySignificance of Study
The proposed study examines a state that has received The proposed study examines a state that has received little attention in the literature. little attention in the literature.
Contributes to program evaluation by using a propensity Contributes to program evaluation by using a propensity score matching multivariate approach.score matching multivariate approach.
Examines institutional variance utilizing a multi-level Examines institutional variance utilizing a multi-level propensity matching methodology.propensity matching methodology.
Relevant to the math redesign efforts and broader Relevant to the math redesign efforts and broader
developmental education reform.developmental education reform.
![Page 32: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/32.jpg)
LimitationsLimitations
No research methodology is free of bias (Bostian, 2008).No research methodology is free of bias (Bostian, 2008).
The propensity score method reduces the number of cases The propensity score method reduces the number of cases through the matching procedure. The elimination of cases through the matching procedure. The elimination of cases occurs during the matching process when a match is not occurs during the matching process when a match is not found based on the propensity score (Caliendo & Kopeinig, found based on the propensity score (Caliendo & Kopeinig, 2008). 2008).
While the colleges range in size from small, medium to large While the colleges range in size from small, medium to large student populations, there is no indication that the colleges student populations, there is no indication that the colleges are representative of most community colleges. are representative of most community colleges.
Some important variables found to be significant indicators Some important variables found to be significant indicators of student success, are not available in the dataset and of student success, are not available in the dataset and introduce the potential for unobserved differences. (Gelman introduce the potential for unobserved differences. (Gelman & Hill, 2007). & Hill, 2007).
![Page 33: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/33.jpg)
Future Research
Future research is needed to examine sensitivity to missing data among the various techniques utilized in multi-level propensity score research (Kim & Seltzer, 2007).
Future research is needed to provide guidance about adequate sample sizes in multi-institutional studies.
Future research should explore the timing of developmental math courses and college-level math in order to provide policy guidance to community colleges currently revising advising models.
Future qualitative research examining institutional variation is needed to help explain the contextual variation that yielded different outcomes for students within institutions.
![Page 34: USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA Bobbie E. Frye Central Piedmont.](https://reader030.fdocuments.us/reader030/viewer/2022032604/56649e5f5503460f94b58ddb/html5/thumbnails/34.jpg)
QuestionsQuestions