The impact of SMF programmes on participants’ …...• The UK has relatively low levels of...
Transcript of The impact of SMF programmes on participants’ …...• The UK has relatively low levels of...
![Page 1: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/1.jpg)
© Institute for Fiscal Studies
The impact of SMF programmes on participants’ early education outcomes
Ellen Greaves and Claire Crawford
31st March 2015
![Page 2: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/2.jpg)
© Institute for Fiscal Studies
Importance of evaluation
• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment
• Important to attempt to increase mobility and reduce socio-economic inequality
• What programmes are most effective at raising the attainment of disadvantaged pupils?
• Where is limited funding best allocated?
• Few charities undertake robust analysis to help determine this
• SMF and J.P. Morgan should be congratulated for their open approach, and other organisations should follow
• This will help to ensure that scarce resources are targeted most effectively and thus hopefully help narrow the attainment gap
![Page 3: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/3.jpg)
The SMF programmes
• Aspiring Professionals Programme (APP)
– Delivered to Year 12 students with high academic attainment and lower socio-economic status
– Mentoring
– Internship
– Skills development
– Events and trips to universities
– University application support
• J.P. Morgan Residential Programme
• Whitehall Social Mobility Internship Programme
© Institute for Fiscal Studies
![Page 4: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/4.jpg)
The SMF programmes
• Aspiring Professionals Programme (APP)
• J.P. Morgan Residential Programme
– Delivered to Year 12 students with high academic attainment and lower socio-economic status, living outside London and with an interest in a career in finance and economics
– Two week internship
– Mentor from J.P. Morgan
• Whitehall Social Mobility Internship Programme
© Institute for Fiscal Studies
![Page 5: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/5.jpg)
© Institute for Fiscal Studies
The IFS evaluation
• Evaluate the impact of SMF programmes on education and employment outcomes of participants
• Education outcomes:
– Higher Education (HE) participation
– Amongst those who go to university: • Participation at “high-status” institution
• Subject choice
• Participation outside home region
• Degree completion and classification
• Early employment outcomes:
– Activity status: whether in work, further education, or unemployed
– Whether working in a professional occupation
![Page 6: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/6.jpg)
© Institute for Fiscal Studies
The IFS evaluation
• Evaluate the impact of SMF programmes on education and employment outcomes of participants
• Education outcomes:
– Higher Education (HE) participation
– Amongst those who go to university: • Participation at “high-status” institution
• Subject choice
• Participation outside home region
• Degree completion and classification
• Early employment outcomes:
– Activity status: whether in work, further education, or unemployed
– Whether working in a professional occupation
![Page 7: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/7.jpg)
Gold-standard evaluation
• Use group of potential participants (e.g. successful applicants)
© Institute for Fiscal Studies
![Page 8: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/8.jpg)
Gold-standard evaluation
• Randomly assign potential participants to two groups
© Institute for Fiscal Studies
![Page 9: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/9.jpg)
Gold-standard evaluation
Programme group
Receive programme
Comparison group
Counterfactual for programme group
© Institute for Fiscal Studies
![Page 10: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/10.jpg)
Gold-standard evaluation
Programme group
60% attend Russell Group institutions
Comparison group
50% attend Russell Group institutions
© Institute for Fiscal Studies
In this example the impact of the programme is 10 percentage points (20% increase)
![Page 11: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/11.jpg)
Feasible evaluation
Programme group Comparison group
Choose individuals with very similar characteristics to
programme group
© Institute for Fiscal Studies
![Page 12: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/12.jpg)
Feasible evaluation
Programme group Comparison group
Choose individuals with very similar characteristics to
programme group
© Institute for Fiscal Studies
Disadvantage: we may not observe all the important ways in which treatment and comparison groups differ
![Page 13: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/13.jpg)
What do we know about SMF participants?
• Background characteristics from application form
– GCSE attainment
– Ethnic group
– Eligibility for free school meals/education maintenance allowance
– Postcode
• A-Level attainment from subsequent SMF survey
• HE destination and subject choice from subsequent SMF survey
© Institute for Fiscal Studies
![Page 14: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/14.jpg)
IFS approach to construct comparison group
• Use administrative data to find individuals with similar characteristics to act as our comparison group
• Eligibility for free school meals
• Local area characteristics
• Prior attainment
• Ethnic group
• But we cannot observe:
• Motivation
• Desired future occupation
• Parents’ level of education
• etc . . .
© Institute for Fiscal Studies
proxy for family income
![Page 15: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/15.jpg)
Concerns about approach
• Survey non-response
– If SMF participants (who answered the survey) are more motivated than individuals with similar attainment, etc, in admin data, then we will be overestimating the impact of the SMF programme
– Hope that this potential bias is relatively small – but can’t be sure
• Comparison group for 2011 and 2012 cohorts
– We do not yet have access to administrative data for these cohorts
– This means we have to use the 2010 cohort for our comparison group
– Not ideal because we know HE participation has been increasing over time (and SES gaps have been falling slightly)
– Impacts may be upward biased for the 2011 and 2012 cohorts
© Institute for Fiscal Studies
![Page 16: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/16.jpg)
Evaluation findings
© Institute for Fiscal Studies
![Page 17: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/17.jpg)
University participation
© Institute for Fiscal Studies
-25
-20
-15
-10
-5
0
5
10
15
20
2009 2010 2011 2012: APP
2012: J.P. Morgan
Perc
enta
ge
Poin
t Im
pac
t
SMF Cohort
Possibly due to earlier cohort used to construct comparison group
Equivalent to a 12% increase
![Page 18: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/18.jpg)
Russell Group participation (amongst those going to university)
© Institute for Fiscal Studies
0 5
10 15 20 25 30 35 40 45 50
2009 2010 2011 2012: APP
2012: J.P. Morgan
Perc
enta
ge
Poin
t Im
pac
t
SMF Cohort
Equivalent to a 17% increase
Equivalent to an 18% increase
Equivalent to a 72% increase
![Page 19: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/19.jpg)
“Top 10” participation (amongst those going to university)
© Institute for Fiscal Studies
-10
0
10
20
30
40
50
60
2009 2010 2011 2012: APP
2012: J.P. Morgan
Perc
enta
ge
Poin
t Im
pac
t
SMF Cohort
Equivalent to a 16% increase
Equivalent to a 13% increase
Equivalent to a 130% increase
![Page 20: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/20.jpg)
Other
• Little impact on subject choice, conditional on HE participation
– Some exceptions, including the probability of studying business and finance for participants on the J.P. Morgan Residential Programme
• Impact is largely similar for participants inside and outside London
• Little impact on the probability of attending an institution outside home region
– Positive impact for 2012 cohort
© Institute for Fiscal Studies
![Page 21: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/21.jpg)
Summary
• Amongst those who went to university, SMF programmes appear to have increased participation at high status institutions
– Increase at Russell Group institutions between 17% and 27% across SMF cohorts
– Increase at institutions most visited by top employers between 13% and 43% across SMF cohorts
• Larger impact for J.P. Morgan residential programme
• Findings are subject to assumptions
• Difficult to compare results across cohorts
– Changing selection criteria across cohorts
– Necessity to use earlier cohort to construct a comparison group
© Institute for Fiscal Studies
![Page 22: The impact of SMF programmes on participants’ …...• The UK has relatively low levels of intergenerational income mobility and large inequalities in educational attainment •](https://reader034.fdocuments.us/reader034/viewer/2022050111/5f487c9ea5c6d93a060a8f1f/html5/thumbnails/22.jpg)
Conclusion
• The SMF programmes seem to have had a sizeable positive effect on institution choice and/or Russell Group application success amongst high achieving young people from disadvantaged backgrounds who might otherwise have gone to different universities
• Magnitude is roughly equivalent to the difference between pupils who achieve three A grades at A-level and three A* grades at A-level, on average, conditional on participation
• Amongst those who go to university, probability of attending a Russell Group institution is higher than those from private schools
• Estimation method could be replicated for other programmes to determine the most effective ways to increase socio-economic mobility and access to professional occupations
© Institute for Fiscal Studies