The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring
description
Transcript of The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring
![Page 1: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/1.jpg)
The Effects of Student Coaching:
An Evaluation of a Randomized Experiment in Student Mentoring
October 3, 2012
Eric Bettinger, Stanford UniversityRachel Baker, Stanford University
![Page 2: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/2.jpg)
Defining the problem: Trends in college attendance and completion
![Page 3: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/3.jpg)
More students are taking classes online
SOURCE: Going the Distance: Online Education in the United States, 2011 Survey by Babson Survey Research Group
![Page 4: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/4.jpg)
College attendance in the United States has consistently increased over the last four decades
SOURCE: The College Board, Trends in College Pricing 2010, Figure 17A and Figure 17B.
![Page 5: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/5.jpg)
College completion has not
SOURCE: Turner 2004.
![Page 6: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/6.jpg)
Voicing concerns about completion
• President Obama (2011): “This country needs and values the talents of every American. That is why we will provide the support necessary for you to complete college and meet a new goal by 2020: America will once again have the highest proportion of college graduates in the world.”
• Vice President Biden (2011): “We have to make the same commitment to getting folks across the graduation state that we did to getting them to the registrar’s office.”
![Page 7: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/7.jpg)
• Financial barriers/liquidity constraints (e.g. Dynarski & Deming 2010)
• No access to appropriate channels of information
(e.g. Bettinger, Long, Oreopolous and Sanbonmatsu 2010)
• Weak academic preparation and performance (e.g. Adelman & Gonzalez 2006)
• Lack of social and academic integration (e.g. Bloom & Sommo 2005, Tinto 1975)
Why do students not complete college?
![Page 8: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/8.jpg)
Today’s Focus is on Student Coaching
• What is coaching?– Individualized instruction/guidance aimed at
helping students overcome barriers• Why coaching?
– Help students to build study skills– “Nudge” students to complete complex tasks– Provide information related to college success
• Previous studies have looked at similar interventions
![Page 9: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/9.jpg)
InsideTrack
• Student coaching service• Business model focuses on being an external,
third-party advising service– Claim to build an economy of scale for counseling
services
• Partners with a number of types of institutions– Most students are studying in vocational tracks.
![Page 10: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/10.jpg)
InsideTrack’s Coaching
• Emphasis on training and hiring coaches• Coaching takes place via phone, email, and
text. • Coaching is “Active” not “Passive”
Our key goal is to identify the effects of this coaching on student retention.
![Page 11: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/11.jpg)
Our Experiment
• InsideTrack wanted to “prove” itself to college partners. They used randomized trials to show colleges their impact.
![Page 12: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/12.jpg)
Selection into Randomization
1. Colleges selected the number of students to be treated and submitted lists of students to InsideTrack.
2. InsideTrack randomly divided college lists into two groups.
![Page 13: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/13.jpg)
Selection into Randomization, con’t
3. InsideTrack presented the list to the schools.
4. Colleges chose which group would receive treatment.
![Page 14: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/14.jpg)
Basic Descriptive Statistics and Balance
Characteristic Control Group Mean
Difference for Treatment(std error)
Sample Size Number of Lotteries
Female.488 .009
(.009)12,525 15
Missing Gender.675 -.001
(.001)13,555 17
Age30.5 .123
(.209)9,569 8
Missing Age.294 .0001
(.0010)13,555 17
SAT Score886.3 -11.01
(16.19)1,857 4
Missing SAT.827 .001
(.002)13,555 17
Living On Campus
.581 -.005(.017)
1,955 4
![Page 15: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/15.jpg)
0.0
1.0
2.0
3.0
4.0
5
0 20 40 60 80Age
Treatment Age Control Age
0.0
005
.001
.001
5.0
02
0 500 1000 1500SAT
Treatment Control0
.2.4
.6.8
0 1 2 3 4HS GPA
Treatment Control
Age Distributions SAT Score Distributions
HS GPA Distributions
Distributions of Treatment and Control Groups
![Page 16: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/16.jpg)
Significant Differences by LotteryLottery # Charac-
teristics# Significant Diff (90%)
1 (n=1583) 2 0
2 (n=1629) 2 0
3 (n=1546) 2 0
4 (n=1552) 2 0
5 (n=1588) 2 0
6 (n=552) 3 0
7 (n=586) 3 0
8 (n=593) 3 0
9 (n=974) 9 0
Lottery # Charact-eristics
# Significant Diff (90%)
10 (n=326) 6 0
11 (n=479) 6 0
12 (n=400) 2 0
13 (n=300) 1 0
14 (n=600) 1 0
15 (n=221) 3 1
16 (n=176) 14 0
17 (n=450) 12 0
![Page 17: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/17.jpg)
Methodology
• Basic Regression Analysis
Y = α + βTreatment + γ1Lottery1 +. . . + γ17Lottery17 +Xδ + ε
Y is an outcome of interest focusing on retention after 6, 12, 18 or 24 months
Treatment is a binary variable for being coached.Lottery# is a binary variable indicating student
participation in a specific lottery.X is a vector of student characteristics
![Page 18: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/18.jpg)
Baseline Results with Covariates
Model 6-month retention
12-month retention
18-month retention
24-month retention
Control Mean .580 .435 .286 .242
1. Baseline
Treatment Effect(std error)
.052***(.008)
.053***(.008)
.043***(.009)
.034**(.008)
Lottery Controls Yes Yes Yes Yes
N 13,552 13,553 11,149 11,153
2. Baseline w/ Covariates
Treatment Effect(std error)
.051***(.008)
.052***(.008)
.042***(.009)
.033**(.008)
Lottery Controls Yes Yes Yes Yes
N 13,552 13,553 11,149 11,153
![Page 19: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/19.jpg)
Robustness: Effects in Each Lottery
![Page 20: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/20.jpg)
Robustness:Effects Across Years
Model 6-month Retention
12-month Retention
18-month Retention
24-month Retention
Control Mean .617 .479 .381 .356
2004 Cohorts
Treatment Effect(std error)
.088***(.020)
.070***(.020)
.068***(.021)
.030(.020)
Covariates Yes Yes Yes Yes
N 1,774 1,745 1,520 1,524
2007 Cohorts
Control Mean .573 .426 .265 .217
Treatment Effect(std error)
.044***(.008)
.049***(.009)
.037***(.010)
.034***(.009)
Lottery Controls Yes Yes Yes Yes
N 11,808 11,808 9,629 9,629
![Page 21: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/21.jpg)
Robustness:50/50 Splits
Model6-month
Retention12-month Retention
18-month Retention
24-month Retention
Completed Degree
Control Mean
.769 .614 .366 .350 .312
1. Baseline Model
Treatment Effect
.037***(.012)
.050***(.014)
.070***(.021)
.027(.020)
.040*(.024)
Lottery Controls
Yes Yes Yes Yes Yes
N 3,527 3,527 1,344 1,348 1,346
2. Baseline w/ Covariates
Treatment Effect
.037***(.012)
.050***(.014)
.070***(.021)
.027(.020)
.040*(.024)
Lottery Controls
Yes Yes Yes Yes Yes
N 3,527 3,527 1,344 1,348 1,346
![Page 22: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/22.jpg)
Effects on Subgroups
Model 6-month Retention
12-month Retention
18-month Retention
24-month Retention
Females
Control Mean .661 .497 .346 .299
Treatment Effect(std error)
.025**(.012)
.045***(.013)
.033**(.014)
.022*(.013)
N 6,045 6,045 4,740 4,744
Males
Control Mean .536 .403 .260 .215
Treatment Effect(std error)
.061***(.012)
.054***(.012)
.047***(.012)
.047***(.011)
N 6,479 6,480 5,457 5,457
Effects by Gender
![Page 23: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/23.jpg)
Effects by Age GroupModel 6-month
Retention12-month Retention
18-month Retention
24-month Retention
Students Under 30
Control Mean .600 .438 .234 .184
Treatment Effect(std error)
.037***(.010)
.052***(.011)
.040***(.012)
.041***(.011)
N 7,850 7,850 5,671 5,671
Students Over 30
Control Mean .513 .400 .311 .266
Treatment Effect(std error)
.062***(.017)
.044***(.017)
.034**(.016)
.024(.015)
N 3,958 3,958 3,958 3,958
Effects on Subgroups
![Page 24: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/24.jpg)
Cost-Benefit Analysis• Most studied intervention focused on retention is
financial aid– Effect sizes are usually around 3 percentage points
per $1000 in aid.– Effect is contemporaneous and doesn’t extend into
future years.• InsideTrack cost about $1000 per year per
student– Contemporaneous effect was about 5 percentage
points– Effects persisted into subsequent year (3 percentage
points)
![Page 25: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/25.jpg)
Conclusion/Discussion• College advisement is a widespread intervention• “Adult” learners are becoming an increasingly important
group of students in higher education; effects were symmetric across age.
• Online education is also rising; multiple campuses in our study were online campuses.
• InsideTrack offers 3rd party advising/coaching– Attempts to exploit economy of scale.– Loosely affiliated with college.– Active rather than passive coaching.
• Effects were large and cost effective
![Page 26: The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Mentoring](https://reader035.fdocuments.us/reader035/viewer/2022062722/56813a4a550346895da23f32/html5/thumbnails/26.jpg)
Degree Completion
• Degree completion information come from 3 lotteries
• Definition of degree is generally four-year degree. It could include some two-year degrees.
• Control Group Graduation Rate = 31.2%• Treatment Effect = 4.0% with standard error
of (2.4%)