USING PROPENSITY SCORE MATCHING TO MODEL RETENTION OF DEVELOPMENTAL MATH STUDENTS IN COMMUNITY COLLEGES IN NORTH CAROLINA

Bobbie E. FryeBobbie E. FryeCentral Piedmont Community CollegeCentral Piedmont Community CollegeDr. James E. Bartlett Dr. James E. Bartlett North Carolina State UniversityNorth Carolina State University

The Council for the Study of Community Colleges

AgendaAgenda

IntroductionIntroduction Statement of Problem/PurposeStatement of Problem/Purpose Research QuestionsResearch Questions Theoretical FrameworkTheoretical Framework MethodsMethods AnalysesAnalyses ResultsResults ContributionsContributions ConclusionsConclusions Future ResearchFuture Research

Introduction

Remedial [developmental] courses are a fixture in the American community college• Allow students that are not prepared for college level work to enroll in college

programs• Annual cost of remediation: 1.9 – 2.3 billion dollars

Placement in developmental course• Varies by institution (i.e. Placement scores, Prerequisites, Statewide policies, Program

requirements, etc.) Remedial Evidence

• Achieving the Dream (ATD) a national community college student success initiative which targets low income and under-represented populations, found that 62% of full-time students in community colleges needed developmental math.

• Data analyses of ATD colleges have revealed consistent patterns of entering students who require developmental education, and who experience low completion rates in developmental courses (ATD, 2010).

• A National Educational Longitudinal Study (NELS) followed 8th grade students from 1988 to the year 2000, and found that 58% of those students who attended a community college took at least one remedial course, 44% took between one and three courses, and 14% took more than three remedial courses (Attewell, Domina, Lavin & Levey, 2006).

Nature of ProblemNature of Problem

Developmental math education: it’s a Developmental math education: it’s a gauntlet that many students run but few gauntlet that many students run but few students (25%) emerge from successfully students (25%) emerge from successfully (Bailey, 2009, Clery, 2009).(Bailey, 2009, Clery, 2009).

About 60% of new students place into About 60% of new students place into developmental math (Bailey, et al., 2010).developmental math (Bailey, et al., 2010).

““Though a primary point of access to higher Though a primary point of access to higher education, community colleges struggle to education, community colleges struggle to ensure that all of their students earn a ensure that all of their students earn a credential” (Kolenovic, Linderman, & Karp, credential” (Kolenovic, Linderman, & Karp, 2013, p.272).2013, p.272).

In addition to financial costs, there are social In addition to financial costs, there are social and personal costs for students who move and personal costs for students who move through the developmental education through the developmental education sequences (Bailey et al., 2010).sequences (Bailey et al., 2010).

Past ResearchPast Research

The studies have methodological The studies have methodological limitations to particular states and limitations to particular states and sub-populations of students. sub-populations of students.

Failure to control for selection bias is Failure to control for selection bias is a common concern in the study of a common concern in the study of developmental education developmental education (Bettinger & Long, 2005). (Bettinger & Long, 2005).

So far, results have been mixed. So far, results have been mixed.

Statement Of ProblemStatement Of Problem

Researchers acknowledge the limitations of comparing student performance, progression, or retention in a non-scientific study wherein participants are not randomly assigned nor even equivalent in terms of motivation, intentions, background, or skill level (Titus, 2007).

Comparisons of student outcomes using propensity matching has been used to yield less biased results than are derived using simple raw comparisons (Rojewski et al., 2009).

Proposed remedies to self-selection bias in educational research need to be explored in terms of feasibility and usefulness, as it is critical that researchers apply methodologies that control for selection bias. Resources are scarce and monies need to be allocated to educational programs that are making a difference in students’ success, retention rates, and long-term outcomes.

What is the Purpose?What is the Purpose?

The purpose of this study is to determine if there is a significant The purpose of this study is to determine if there is a significant difference in characteristics and college level outcomes between difference in characteristics and college level outcomes between developmental math students who complete college math developmental math students who complete college math coursework compared to students who do not complete math coursework compared to students who do not complete math coursework.coursework.

Students referred to developmental math are the participants, Students referred to developmental math are the participants, defined as students referred into one or more areas of defined as students referred into one or more areas of developmental math coursework and enrolled in at least one developmental math coursework and enrolled in at least one developmental math course during the study period. The developmental math course during the study period. The population consists of 2007-08 new student cohorts at seven NC population consists of 2007-08 new student cohorts at seven NC community colleges. community colleges.

Purpose

The purpose of this study was to determine if there is a difference in outcomes of grade point average, credits earned, credentials

earned, transfer, and retention between two groups.

Comparison Group:Group of students who

complete developmental math coursework, attempt but do no

complete college level math with a C or better

Study Group:Group of students who

complete developmental math coursework and subsequently college level math with a C or

better

Propensity matching was used to create two equivalent groups of students matched on the propensity to

complete college level math

Research QuestionsResearch Questions

Prior to propensity matching, is there a difference in Prior to propensity matching, is there a difference in student and academic characteristics between student and academic characteristics between developmental students who complete college math developmental students who complete college math coursework compared to students who do not complete coursework compared to students who do not complete math coursework?math coursework?

Prior to propensity matching, is there a difference in Prior to propensity matching, is there a difference in student and academic characteristics between student and academic characteristics between developmental students who complete college math developmental students who complete college math coursework compared to students who do not complete coursework compared to students who do not complete math coursework between institutions?math coursework between institutions?

After propensity matching, is there a difference in After propensity matching, is there a difference in college level outcomes between developmental college level outcomes between developmental students who complete college math coursework students who complete college math coursework compared to students who do not complete math compared to students who do not complete math coursework?coursework?

After propensity matching, is there a difference in After propensity matching, is there a difference in college level outcomes between developmental college level outcomes between developmental students who complete college math coursework students who complete college math coursework compared to students who do not complete math compared to students who do not complete math coursework between institutions ?coursework between institutions ?

Theoretical Framework – Theoretical Framework – 10

Student retention theoriesStudent retention theories Tinto – Academic and social integration Tinto – Academic and social integration

Behaviors (1993)Behaviors (1993)

Bean and Metzner Academic integration Bean and Metzner Academic integration and environmental factors – Beliefs (1995) and environmental factors – Beliefs (1995)

Four Models Examined

Variable Name Model IStudent

Demographics

Model IIAcademic Integration

Model IIIIndividual &

Institution

Model IVIndividual &

InstitutionGender X X X XAge X X X XRace* X X X XTransfer Program X X X XPell Recipient X X X XFull-Time in 1st Term X X X XFirst Dev. Math Level X X X XRetained 2nd term X X XDev. English X X XSuccess Course X X XFirst Term GPA X X X% Minority** X% Pell Recipients** X% Dev. English *** X % Dev. Math *** XInstitution Member X

*Race categories: White, African American, Hispanic, International, Asian, Other ** Number of students as a percentage of fall 2007 enrollment***Number of students as a percentage of 2007-08 institutional FTE

MethodsMethods Non-Experimental (Propensity Score Matching)Non-Experimental (Propensity Score Matching)

Study Group 5,548 students who placed into Study Group 5,548 students who placed into developmental math coursework and of those 2,102 (38%) developmental math coursework and of those 2,102 (38%) of students who subsequently attempted college level of students who subsequently attempted college level mathmath

Purposeful sampling techniquePurposeful sampling technique

VariablesVariables Matching variables-IndependentMatching variables-Independent Grouping (Variable-A-C Completion/Non-Completion College Grouping (Variable-A-C Completion/Non-Completion College

Math)Math) Dependent (Student Outcomes)Dependent (Student Outcomes)

Data Collection- Student term records Fall 2007-Summer Data Collection- Student term records Fall 2007-Summer 20122012

Data AnalysisData Analysis Multi-Level Propensity Score MatchingMulti-Level Propensity Score Matching

There are six essential steps to PSM:

Data pre-screening Covariate identification Propensity score estimation Matching of propensity scores Determination of matching success Presentation of results

Analyses

To analyze whether there is a difference in demographics and academic characteristics of the two study groups prior to propensity score matching t-tests will be used. Means, standard deviation, significance levels of p<=05, t values and cohen’s d (effect size) will be reported using t-tests analyses (Rojewski, et al., 2009; 2010).

Bias percentages (standardized effect sizes) before and after matching were also reported (Oakes & Johnson, 2006). The standardized percent bias is the percent difference of the sample means in the treated and non-treated (full or matched) sub-samples as a percentage of the square root of the average of the sample variances in the treated and non-treated groups (Rosenbaum & Rubin, 1985)

Logistic regression will also be used to determine the background and academic factors that explain membership in the two groups of study and to create a propensity score that is used in the matching technique. Logistic stepwise regression will yield significant independent variables that predict membership in the control or treatment groups. Nakelkerke R-squared, chi-squared, beta coefficients and independent variables with p value <=.05 were reported and retained in the model (Rojewski, et al., 2009; 2010).

Using the propensity score generated through the logistic regression technique, equivalent groups of students were propensity matched using a matching procedure in STATA software (Titus, 2007). The technique known as psmatch2 creates two matched groups based on the propensity to complete college level math.

Logistic Regression Results (Sig. prior to Match)Covariates Model I Model

IIModel III

Model IV

DV: Complete CL Math Exp(B) Exp(B) Exp(B) Exp(B)

Gender (Female)1.411** 1.369** 1.377** 1.345**

Age1.018*

African American 0.532** 0.607* 0.487*** 0.484***

Pell Recipient 0.736** 0.736** 0.756* 0.754*

First Term GPA 1.235*** 1.254*** 1.235***

% Minority 1.055*** NA

% Pell Awarded 0.945* NA

% Dev English 0.914* NA

College B 1.350*

College F 2.963***

College G 0.459***

R-Squared 0.044 0.075 0.090 0.115

Notes: *p < .05 **p < .01 ***p < .001

Model IV T-Tests Prior to Matching(Sig. variables)

Model IV T-Tests Prior to Matching (continued)

Covariates N M SD t p d % bias

College B 539 0.345 0.475 2.410 0.016* 0.122 12.100

1,563 0.404 0.490

College F 539 0.043 0.202 5.740 0.000*** 0.320 31.800

1,563 0.131 0.338

College G 539 0.134 0.341 -4.690 0.000*** -0.219 -21.700

1,563 0.068 0.253

Average BiasR-Squared

13.4000.072

Cohen’s d = Mt – Mc / σpooled; where σpooled = √ σ2t + σ2c / 2 *p < .05 **p < .01 ***p < .001

First line of each variable is the control group; second line is the treatment group.

Model IV T-Tests After Matching

Model IV T-Tests After Matching (Continued)

CovariateN M SD t p d % bias

Full-Time 512 0.656 0.475 0.351 0.725 0.023 2.600

507 0.667 0.472First Dev. Math 512 1.984 0.735 0.256 0.798 0.016 1.400

507 1.996 0.722

Dev. Eng 512 0.453 0.498 0.206 0.837 -0.012 3.900

507 0.459 0.499

Success 512 0.359 0.480 -0.342 0.732 -0.021 -0.900

507 0.349 0.4771st Term GPA 512 2.141 1.460 -0.609 0.543 -0.038 -7.200

507 2.084 1.556Retained to 2nd 512 0.901 0.300 -0.156 0.875 -0.013 -1.200

507 0.897 0.304

Model IV T-Tests After Matching (Continued) Covariate N M SD t p d % bias

College B 512 0.359 0.480 -0.541 0.589 -0.034 -4.200

507 0.343 0.475

College C 512 0.051 0.220 -0.556 0.578 -0.038 -4.600

507 0.043 0.204

College D 512 0.008 0.088 -0.070 -0.366 -0.024 -2.200

507 0.006 0.077

College E 512 0.043 0.203 -0.123 0.902 -0.010 -1.500

507 0.041 0.199

College F 512 0.049 0.207 -0.597 0.550 -0.060 -3.200

507 0.037 0.190

College G 512 0.121 0.326 0.430 0.667 -0.028 0.400

507 0.112 0.316

Average Bias R-Squared

3.300

0.063

Cohen’s d = Mt – Mc / σpooled; where σpooled = √ σ2t + σ2c / 2 *p < .05 **p < .01 ***p < .001First line of each variable is the control group; second line is the treatment group.

Summary Before and After Matching

Bias Pre

Bias Post

%Change

R2

PreR2

PostChange

Sig. CV’s Pre

Sig. CV’s Post

Change

Model I 10.4% 10.3% -0.9% 0.020 0.041 +0.021 4 3 -1

Model II 12.5% 4.20% -66.4% 0.046 0.040 -0.006 6 0 -6

Model III 13.0% 2.80% -78.5% 0.061 0.050 -0.011 8 1 -7

Model IV 13.4% 3.30% -75.4% 0.072 0.063 -0.090 8 0 -8

Final AnalysesFinal Analyses

After matching, to determine if there a difference in college level outcomes between developmental math students who complete college math coursework compared to students who do not complete math coursework, T-Tests were used. Means, standard deviation, significance levels of p<=.05, t values and cohen’s d were reported using t-tests analyses (Rojewski, et al., 2010).

Completion outcomes were categorical and chi-square analyses were used to determine the measures of association between the students who completed college math coursework compared to students who did not complete math coursework in terms of completion categories (Rojewski, et al, 2010).

Model IV T-Tests Outcomes after Matching

College Courses

N M SD t p dCredits Attempted 512 51.259 27.090 10.083 0.000*** 0.632

507 67.852 25.406Credits Completed 512 31.516 23.827 16.453 0.000*** 1.031

507 55.884 23.447Credits A-C

512 26.750 21.654 17.905 0.000*** 1.122

507 51.525 22.511Retention

Terms Retained

512 6.139 2.785 8.245 0.000*** 0.516

507 7.524 2.576

Model IV T-Tests Outcomes after Matching

Outcomes N M SD t p dDevelopmental CoursesCredits Attempted 512 13.504 8.894 -0.565 0.572 -0.035

507 13.191 8.761Credits Completed 512 10.080 7.133 2.169 0.030* 0.136

507 11.087 7.673Credits A-C

512 9.402 6.834 2.189 0.029* 0.137

507 10.368 7.258Developmental Math

Credits Attempted 512 8.101 4.603 -0.871 0.384 -0.054

507 7.854 4.470Credits Completed 512 5.906 3.752 2.036 0.042* 0.128

507 6.379 3.652Credits A-C

512 5.398 3.469 2.019 0.044* 0.126

507 5.826 3.295

Model IV Completion OutcomesCompletion Categories

Passed College-Level Math

Grade of C or Better

Row Percent

Not Passed College-Level Math

Row Percent

Total

No Outcome Observed 133 29.23% 322 70.77% 455

Expected 226.4   228.6   455.0

Associate/Certificate Observed 81 77.88% 23 22.12% 104

Expected 51.7   52.3   104.0

Transfer 2 Year Observed 44 72.13% 17 27.87% 61

Expected 30.4   30.6   61.0

Transfer 4 Year Observed 177 70.52% 74 29.48% 251

Expected 124.9   126.1   251.0

Still Enrolled 30 Plus Credits

Observed 72 48.65% 76 51.35% 148

Expected 73.6   74.4   148.0

Total 49.75% 50.25% 1019

TotalAlpha= 0.050Chi-Squared= 165.159Degrees of Freedom= 4p value < .001Cramer’s V= 0.403

Conclusions

After matching, the average means differ between the two groups on key progress indicators.

Specifically, the study group fared better than the comparison group and completed, on average, 25 more college credits at the community college.

In addition, the study group was retained an average of 1.5 more semesters than the comparison group.

Conclusions (Continued)

In terms of expected versus observed frequencies, a higher number (71%) of students in the non-completion group earned no award compared to students in the completion group.

Completers of college math with C or better earned significantly more associate degrees (78%) than non-completers.

However, in the four models, non-completers of college level math were twice as likely to earn no outcome after five years when compared to the completer group.

Transfers to four-year and two-year institutions were common in both groups of students, and the math completion group was twice as likely to transfer out of the institution.

Contributions

Applied propensity score matching methods across multiple community colleges

Demonstrated that students did not differ to a large extent

Demonstrated that propensity matching reduced the bias at least 75% for the multi-level model

Demonstrated the need to conduct additional research and examine variations among colleges

Significance of StudySignificance of Study

The proposed study examines a state that has received The proposed study examines a state that has received little attention in the literature. little attention in the literature.

Contributes to program evaluation by using a propensity Contributes to program evaluation by using a propensity score matching multivariate approach.score matching multivariate approach.

Examines institutional variance utilizing a multi-level Examines institutional variance utilizing a multi-level propensity matching methodology.propensity matching methodology.

Relevant to the math redesign efforts and broader Relevant to the math redesign efforts and broader

developmental education reform.developmental education reform.

LimitationsLimitations

No research methodology is free of bias (Bostian, 2008).No research methodology is free of bias (Bostian, 2008).

The propensity score method reduces the number of cases The propensity score method reduces the number of cases through the matching procedure. The elimination of cases through the matching procedure. The elimination of cases occurs during the matching process when a match is not occurs during the matching process when a match is not found based on the propensity score (Caliendo & Kopeinig, found based on the propensity score (Caliendo & Kopeinig, 2008). 2008).

While the colleges range in size from small, medium to large While the colleges range in size from small, medium to large student populations, there is no indication that the colleges student populations, there is no indication that the colleges are representative of most community colleges. are representative of most community colleges.

Some important variables found to be significant indicators Some important variables found to be significant indicators of student success, are not available in the dataset and of student success, are not available in the dataset and introduce the potential for unobserved differences. (Gelman introduce the potential for unobserved differences. (Gelman & Hill, 2007). & Hill, 2007).

Future Research

Future research is needed to examine sensitivity to missing data among the various techniques utilized in multi-level propensity score research (Kim & Seltzer, 2007).

Future research is needed to provide guidance about adequate sample sizes in multi-institutional studies.

Future research should explore the timing of developmental math courses and college-level math in order to provide policy guidance to community colleges currently revising advising models.

Future qualitative research examining institutional variation is needed to help explain the contextual variation that yielded different outcomes for students within institutions.

QuestionsQuestions