UC Berkeley Admissions in 2015 and 2016: An Analysis of...
Transcript of UC Berkeley Admissions in 2015 and 2016: An Analysis of...
![Page 1: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/1.jpg)
UCBerkeleyAdmissionsin2015and2016:AnAnalysisoftheRoleofLettersofRecommendationandAugmentedReview
July13,2016
JesseRothstein,ProfessorofPublicPolicy&Economics
ExecutiveSummaryUCBerkeleymademanychangestoitsundergraduateadmissionprocessesin2016,includingtheadditionoflettersofrecommendation(LORs)formanyapplicantsandtheeliminationoftheseparate“AugmentedReview”(AR)pool.Thesechangeswereintendedinparttobetteridentifyhiddengemsintheapplicantpool–becausesomanymorestudentscouldbeaskedforLORsthanwereeverconsideredunderAR,ifLORswereanywherenearaseffectiveatidentifyingunderrepresentedstudentswhocouldsucceedatCalthenetimpactwouldbetoincreasediversity.Ontheotherhand,thechangesmighthaveraisedbarrierstoadmissionfordisadvantagedstudents,whomightnothaveaccesstoteachersorcounselorswillingandabletowritestronglettersandwhosehiddenstrengthsmightnothavebeenrecognizedwithoutAR.ViceChancellorKoshlandaskedmeinJunetoconductanindependentanalysisofBerkeleyundergraduateadmissions,focusingontheLORandARchanges.IhavedoneacademicresearchoncollegeadmissionsatUCandelsewhere,butIhaveneverbeeninvolvedwithBerkeleyadmissionsprocesses.Topreparethisreport,IconsultedextensivelywiththeadmissionsofficeandwiththeSenateCommitteeonAdmissions,Enrollment,andPreparatoryEducation(AEPE)inanefforttounderstandtheadmissionsprocess.Thisreportreflectsmyindependentanalysisandconclusions.Ireachtwomainconclusions.First,thenumberofapplicantsfromunderrepresentedgroups(low-income,first-generation,fromlow-APIhighschools,and/orunderrepresentedminorities)whowereadmittedrosein2016.Butbecausethenumberadmittedwhowerenotfromthesegroupsrosebyalargerproportion,theshareofadmittedstudentsfromunderrepresentedgroupsfellsomewhat.Second,neithertheadditionofLORsnortheremovalofARcontributedmeaningfullytothisdecline.Ifanything,askingforLORsraisedtherelativeadmissionsratesofapplicantsfromunderrepresentedgroups.IamunabletopreciselyidentifytheimpactoftheeliminationofAR–whilesomeestimatesindicatethatthisslightlyreducedadmissionsforthosewhowouldhavebeenconsideredviaARin2015,othersindicatezeroorevenapositiveeffect.Alloftheestimatesagreethattheimpactwassmallinanycase.Itisbeyondthescopeofthisreporttoidentifywhatdidcausetheshiftin2016.Mypreliminaryinvestigation,however,suggeststhatthedeclineintheshareofadmitsfromunderrepresentedgroupsisinlargepartastatisticalartifactduetotheexpandeduseofthewaitlistin2016.Therewerealsoreductionsintheadmissionschancesoftheunderrepresentedapplicantswiththestrongestnumericrecordsthatcannotbeattributedtothewaitlist.Futureinvestigationshouldfocusonunderstandingwhatinthescoringprocessharmedtheseapplicants.
![Page 2: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/2.jpg)
2
IntroductionBerkeleymadeanumberofchangestoitsundergraduateadmissionsprocessesin2016:
• Itrequestedlettersofrecommendation(LOR)formanyapplicants.• Iteliminatedtheaugmentedreview(AR)pool.• Itshiftedfromapointsystemforscoringapplicationstoathree-category
(Yes/No/Maybe)ratingsystem.• Readersbeganscoringapplicantsonalistofholistic/non-cognitivefactors.• Everyapplicationwasreadtwice,whereinthepastmanywerereadonlyonce• Athirdread,bymembersofthefaculty,wasaddedformanyapplications.• Thewaitlistwasusedmuchmoreextensivelythaninthepast,andmanyapplicants
whoin2015wouldhavebeenadmittedorrejectedoutrightwereinsteadofferedpositionsonthewaitlistin2016.
Table1presentssimplesummariesofadmissionsoutcomesin2015and2016,intheupperpanelpoolingallin-stateandout-of-stateapplicantsandinthelowerpanelrestrictingattentiontoCaliforniaresidentsnotbeingrecruitedasathletes.OfparticularconcernisthedeclineintheshareofadmittedstudentsfromdisadvantagedbackgroundsorgroupsthatareunderrepresentedatBerkeley.InTable1andthroughoutthisreport,Iconsiderfourgroupsofsuchstudents:low-incomestudents(withfamilyincomesbelow$40,000);first-generationcollegestudents(thosewhoseparentsdidnotgraduatefromcollege);studentsfromdisadvantagedschools(withAPIindexesof5andbelow);andunder-representedminoritystudents(UREMs).Irefertothemcollectivelyas“underrepresented.”Iwasaskedtoconductananalysisofadmissionsdataforthe2015and2016cycles,focusingontheimpactoftheadditionoflettersofrecommendationandtheeliminationofaugmentedreviewontheadmissionofunderrepresentedstudents.Inalloftheanalysesbelow,IrestrictattentiontoCaliforniaresidentapplicantswhowerenotclassifiedintheadmissionsprocessasrecruitedathletes.ThelowerportionofTable1showsstatisticsfortheseapplicants.TheoverallcontextIbeginwithseveralpreliminariesnecessarytounderstandingtheimpactsoflettersofrecommendationandaugmentedreviewonadmissionsoutcomes.ThecompositionoftheapplicantpoolThefirstplacetolookforanexplanationforchangingoutcomesischangesintheapplicantpoolitself.Iftheapplicantsfromunderrepresentedgroupswereweaker,onaverage,in2016thanin2015,thiscouldaccountfortheoverallobservedchanges,evenwithoutachangeinpolicy,andwouldconfoundmyLORandARanalyses.UnderstandingthedistributionofapplicantstrengthisalsohelpfulasawayofgaugingwhichtypesofstudentswereaffectedbytheLORandARprograms.
![Page 3: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/3.jpg)
3
Table1.Summaryofadmissionsoutcomes
Applicants
Admits
2015 2016
2015 2016
Allapplicants
Number 78,924 82,578
13,266 14,423
Admissionrate
16.8% 17.5%
Sharefromunderrepresentedgroups
Lowincome(<$40K) 24% 23%
19% 18%
Firstgeneration 17% 16%
13% 12%
LowAPI 13% 12%
12% 11%
UREM 23% 23%
18% 18%
Anyofthesefour 40% 38%
33% 31%
Californiaresidents,excludingrecruitedathletes
Number 45,570 45,626
8,570 9,610
Admissionrate
18.8% 21.1%
Sharefromunderrepresentedgroups
Lowincome(<$40K) 31% 30%
24% 22%
Firstgeneration 24% 23%
18% 15%
LowAPI 23% 22%
19% 16%
UREM 33% 34%
23% 23%
Anyofthesefour 53% 53%
42% 39%
Iconstructameasureofapplicantstrengthbyaggregatinganumberofavailablestudentcharacteristics.1My“admissionsscore”representsthepredictedprobabilitythatastudentwithagivensetofcharacteristicswouldhavebeenadmittedinthefirstround(i.e.,notoffthewaitlist,andnotthroughAugmentedReview),hadhe/sheappliedasaCaliforniaresidenttotheCollegeofLettersandSciencesin2015.2
1Thevariablesarethoseusedinthemodelusedbytheadmissionsofficetopredictapplicationreadscores,andincludemeasuresoftraditionalacademicstrength,measurescharacterizingthehighschool,(including,forexample,theextenttowhichthestudenttookadvantageoftheschool’sadvancedcourseofferings),andthreeofthefourdisadvantagemeasuresconsideredhere.Applicants’raceandethnicity,whichcannotbeconsideredinadmissions,isnotincluded.2IuseonlyL&SapplicantsbecausetheothercollegesmayweightcharacteristicsdifferentlythandoesLettersandSciences.Nevertheless,myL&S-basedscoreisnearlyperfectlycorrelatedwithascoreconstructedbasedonadmissionsintheCollegeofEngineering.
![Page 4: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/4.jpg)
4
Twoaspectsofthisscoremustbeemphasized:First,itcapturesonlythequantitativecharacteristicsthatarecodedintheadmissionsoffice’sdatabase;readersseemoreinformation,andmayidentifyapplicantsasstrongerorweakerthanisindicatedbymyscore.Second,thecharacteristicsareweightedtobestpredictadmissionin2015.Theweightputondifferentcharacteristics–say,onhighSATscoresvs.takingalloftheAPcoursesofferedatyourhighschool–mightvaryfromyeartoyear,andindeedseemstohavechangedsomewhatin2016(asdiscussedbelow).Butevenwiththesecaveats,theadmissionsscoreneverthelesspresentsausefulsummary.Totakeoneexample,38%ofapplicantsin2016havescoresunder1%.Whileaveryfewofthesestudentsmighthavecharacteristicsnotinthedatabasethatmeritadmission,thisisquiterare;thevastmajorityofstudentsinthisgroupwouldnotbeadmittedundertheregular2015processes.Indeed,only1.8%ofthemwereadmittedin2015,and2.5%in2016.Figure1showsthedistributionofadmissionsscoresacrossCaliforniaresident,non-athleteapplicantstoallcollegesin2015.Thisisheavilyleft-skewed:Mostapplicantshaveverylowchancesofadmission,thoughthereareafewwhoaresostrongonthedimensionscapturedbymyindexthatitisrareforotherfactorstopreventthemfrombeingadmitted.
Figure1.Distributionofadmissionsscoresin2015forCaliforniaresidentapplicants
Becausetheoveralldistributionofadmissionsscoresissodominatedbyapplicantswithextremelylowchancesofadmission,Ifindithelpfultofocusonthosewhoaremorelikelytobeadmitted.Figure2showsthedistributionofadmissionsscoresforthosewhowereactuallyadmittedin2015(includingARadmitsandthoseadmittedoffthewaitlist),whileFigure3repeatsthisforthefourunderrepresentedgroupsandFigure4repeatsitforapplicantsnotfromthesegroups.
0.1
.2.3
.4Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015
![Page 5: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/5.jpg)
5
Figure2.DistributionofadmissionsscoresforadmittedCaliforniaresidentapplicantsin2015
ItisapparentinFigure3thattheadmissionsscoredistributionisquiteskewedtotheleftforstudentsfromtheunderrepresentedgroups.Thisistrueeventhoughthepredictionmodelusedtogeneratetheadmissionsscoreincludesindicatorsforlowincome,firstgeneration,andlowAPI(butnotUREM)students.Evidently,manyofthestudentswhoareadmittedfromthesegroupsarepickedoutfromlargepoolswithsimilarobservablecredentialswhoarenotadmitted.Thisismuchlesstrueforstudentsnotfromthesegroups,forwhomthedistributionisshowninFigure4:Here,admittedstudentsaremuchmorelikelytohaveadmissionsscoresabove0.6.
Figure3.AdmissionsscoredistributionsforadmittedCaliforniaresidentapplicantsfromfourunderrepresentedgroupsin2015
0.0
1.0
2.0
3.0
4Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015For those actually admitted
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Low income
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
1st Generation
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Low API
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
UREM
Distribution of admissions scores in 2015For those actually admitted
![Page 6: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/6.jpg)
6
Figure4.AdmissionscoredistributionforCaliforniaresidentsnotfromunderrepresentedgroups,2015
Table2showssummariesofthedistributionofadmissionsscoresforCaliforniaresidentapplicantsandadmittedstudentsin2015and2016,bothforallapplicantsandforapplicantsfromtheunderrepresentedgroups.Itindicatesthatthedistributionofadmissionsscoreschangedsomewhatbetweenyears,withmorestudentswithverylowandveryhighadmissionsscoresin2016thanin2015.Theseroughlyoffseteachother,however,andaverageadmissionsscores,bothoverallandforapplicantsfromunderrepresentedgroups,werequitesimilarin2016asin2015.Overall,changesinthedistributionofobservablecharacteristicsamongapplicants,onitsown,wouldnotlikelyhaveproducedsubstantialchangesinapplicationoutcomes.
Table2.Distributionofadmissionsscoresin2015and2016 Californiaresidents,excludingrecruitedathletes
Allstudents
Underrepresentedgroups
Applicants
Admits
Applicants
Admits
2015 2016
2015 2016
2015 2016
2015 2016
Mean 0.17 0.17
0.49 0.45
0.13 0.13
0.42 0.40Fractionbelow1% 33% 38%
1% 3%
39% 45%
2% 5%
Fractionbelow5% 53% 57%
7% 12%
61% 64%
11% 15%
5thpercentile 0.00 0.00
0.04 0.02
0.00 0.00
0.02 0.0110thpercentile 0.00 0.00
0.08 0.04
0.00 0.00
0.05 0.03
25thpercentile 0.00 0.00
0.22 0.14
0.00 0.00
0.15 0.1050thpercentile 0.04 0.03
0.49 0.43
0.02 0.02
0.38 0.33
75thpercentile 0.22 0.22
0.76 0.75
0.14 0.13
0.67 0.6790thpercentile 0.61 0.65
0.89 0.90
0.45 0.49
0.85 0.87
95thpercentile 0.79 0.83
0.93 0.94
0.67 0.72
0.91 0.9299thpercentile 0.94 0.95
0.97 0.98
0.90 0.92
0.96 0.98
0.0
1.0
2.0
3.0
4.0
5Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015For those actually admitted
![Page 7: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/7.jpg)
7
Figure5showstheshareofstudentsateachadmissionsscorelevelwhowereconsideredunderARin2015(solidline)oraskedforLORsin2016(dashedline).ThefeaturethatjumpsoutthemostisthattheLORprogramwasmassivelylargerthanARhadbeen–evenatthelowestadmissionsscores,whereARstudentsareconcentrated,theshareof2016studentsfromwhomLORswererequestedgreatlyexceedstheshareconsideredunderARin2015.Thesecondthingtonoticeisthat,whileARstudentswereconcentratedaroundthe0.3mark–thatis,thesestudentswere,basedonobservablesalone,muchweakerthantheaverageapplicant–LORswereusedmostamongstudentswithmuchhigheradmissionsscores,50-80%.Inthisrange,nearlyall2016applicantswereaskedforLORs,but10%orlesswereconsideredunderARin2015.
Figure5.ShareofCaliforniaresidentapplicantsconsideredunderARoraskedforletters
ThewaitlistBeyondtheAR/LORshift,anotherimportantchangein2016wasagreatlyexpandeduseofthewaitlist.Theshareofapplicantsofferedpositionsonthewaitlistnearlydoubled(from4.5%to8.6%)in2016.Manystudentsdeclinedtheseoffers–fully4%of2016applicantsdeclinedofferedpositionsonthewaitlist,ascomparedto1.5%in2015.Thisgreatlycomplicatescomparisonsof2015and2016outcomes,assomeofthestudentswhoturneddownpositionsonthewaitlistin2016wouldhavebeenadmittedoutrightin2015.ManyBerkeleyapplicantsarechoosingbetweenBerkeleyandotherexcellentuniversities,andmanywhoareacceptedwindupgoingelsewhere.In2015,lessthanhalfofadmittedstudentscametoBerkeley,andthissharewassmallerforstrongerapplicants.Inmanycases,studentswillhavealreadydecidedtoenrollelsewherebythetimeBerkeley’sinitialadmissionsoffersaremade.Consider,forexample,astudentadmittedelsewhereunderanearlydecisionprogram.Inprinciple,thisstudentmightwithdrawherBerkeleyapplication,butthishasnotbeeneasytodo,andinanyeventsomestudentsmightnotbotherwithnothingatstake.
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
AR in 2015 LOR in 2016
All applicants
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
AR in 2015 LOR in 2016
From underrepresented groups
Share of applicants considered under AR and LOR
![Page 8: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/8.jpg)
8
Ifthesestudentsarechosenforinitialadmission,theycounttowardourstatisticsonadmittedstudents.Butiftheyareofferedpositionsonthewaitlist,theyarelikelytoturndownthisoption,andthuswillnotcountasadmissionseveniftheywouldeventuallyhavebeenadmittedoffthewaitlist.Thus,ashiftofsomeadmissionsfromthefirsttothesecondround,asoccurredin2016,willreducetheshareoftheseuninterestedstudentsintheadmitpool(and,asasideeffect,raisetheenrollmentrateamongthoseadmitted).Table3showsthedistributionofadmissionsoutcomes,aggregatingnon-athleteCaliforniaresidentsacrosseachoftheseparateapplicantpools(fordifferentcollegesanddivisions)butseparatingthedifferentstages.Thisillustratesthepotentialdistortioncausedbythewaitlist:Notethattheshareofunderrepresentedapplicantswhowereadmittedrosebyonly0.7percentagepoints(andtheshareadmittedinthefirstroundfellby0.2p.p.),whilethesharewhoenrolledroseby1.9p.p.Thisisbecausethe“yield”rateforadmittedstudentsroseby4percentagepoints,from47%to51%overall,andby10p.p.,from59%to69%,foradmittedstudentsfromtheunderrepresentedgroups.Table3.Stagesoftheadmissionsprocess
Californiaresidents,excludingrecruitedathletes
Shareofallapplicants(%)
Shareofapplicantsfrom4groups(%)
2015 2016 Change
2015 2016 Change
Initialadmit 16.1 18.1 2.0
13.3 13.0 -0.2Initialdeny 79.4 73.3 -6.1
83.8 78.9 -4.8
Offerwaitlist 4.5 8.6 4.1
3.0 8.0 5.1
Declinewaitlistoffer 1.5 4.0 2.5
1.2 4.3 3.1
Acceptwaitlistoffer 3.0 4.6 1.6
1.8 3.8 1.9
AdmittedfromWL 2.7 3.0 0.3
1.6 2.6 1.0
NotadmittedfromWL 0.3 1.6 1.3
0.2 1.2 1.0
Ultimateoutcomes
Admitted 18.8 21.1 2.3
14.9 15.6 0.7
Enrolled(SIR) 8.8 10.7 1.9
8.8 10.7 1.9
Didnotenroll 10.0 10.4 0.3
6.1 4.9 -1.2
Denied 79.7 74.9 -4.8
83.9 80.1 -3.9
WithdrewafterWLoffer 1.5 4.0 2.5
1.2 4.3 3.1
Table3reinforcesmyconcernthatissuesofself-selectionarequantitativelyimportant.Manystudentswhounder2015processeswouldhavebeenadmittedbutgoneelsewherewereinsteadin2016offeredpositionsonthewaitlistonlytodeclinetheoffers–perhapsasmanyas1%of2016applicants.Thesestudentswouldhavebeencountedasadmitsin2015butnotin2016.Importantly,thisaffectsthestatisticsforunderrepresentedstudents,asthesestudents
![Page 9: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/9.jpg)
9
weredisproportionatelylikelytodeclinepositionsonthewaitlist:47%ofallstudentsofferedpositionsonthewaitlistdeclinedthem,butthissharewas53%forstudentsfromtheunderrepresentedgroups.Unfortunately,thereisnoadmissionsmeasurethatisperfectlycomparableacrossyears–inparticular,thecompositionofboththepoolofinitialadmitsandthepoolofeventualadmitsisaffectedbytheincreaseduseofthewaitlist,evenwithnootherchanges.Inthisreport,Ipresentresultsforfourdifferentmeasures,eachimperfect:
- Initialadmissions(includingbothFallandSpringadmits)- Admittedintheinitialroundorofferedapositiononthewaitlist- Everadmitted,eitherintheinitialroundoroffthewaitlist- Admittedandenrolled(asproxiedbyfilinganSIR,eitherforFallorSpringenrollment)
Thelastofthese,ofcourse,reflectsstudentaswellascampusdecisions(asdoesthethird,whichreflectsstudentdecisionstoacceptaspotonthewaitlist).Nevertheless,inmyviewitistheclosesttocomparableacrossyears.Ifstudents’propensitytoacceptBerkeleyadmissionsoffers,ifmade,didnotchangeacrossyears,andifastudentwhowouldacceptaninitialofferisnotputoffbybeingadmittedoffthewaitlist,changesinthepoolofenrolledstudentscanbeattributedtochangesinadmissionscriteria.AugmentedReviewandLettersofRecommendationThisreportfocusesontheAugmentedReviewandLettersofRecommendationcomponentsoftheadmissionsprocess.Table4showsthenumberofstudentsconsideredunderARin2015,thenumberaskedforLORsin2016,andtheoutcomesofeachgroupofapplications.AsalreadyindicatedbyFigure5,thismakesclearthattheLORprogramwasmuch,muchlargerthantheARpool,whichIunderstandwaskeptsmallduetotheenormousstafftimerequiredtoreviewARapplications.Table4alsoshowsthat15%ofstudentswhowereaskedforlettersdidnotrequestany.Thismightreflectwhatmanywereconcernedabout,thatstudentswouldnothaveaccesstoteacherswillingtowriteletters.Buttheaboveself-selectiondiscussionpointstoanotherpotentialexplanation:StudentsadmittedEarlyDecisionelsewhere,andothersnotveryinterestedinBerkeley,mightsimplynothavebotheredtorequestletters.Forthisreason,Idonotemphasizerequestsoforreceiptoflettersasoutcomes,andfocusontheimpactofthelettersrequestonthestudent’slikelihoodofbeingadmittedorofenrolling.
![Page 10: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/10.jpg)
10
Table4.OutcomesforARandLORstudents
Californiaresidents,excludingrecruitedathletes
Allapplicants
Underrepresentedgroups
ARin2015
LORin2016
ARin2015
LORin2016
Numberaffected 3,046 14,406
2,793 6,337
Shareofallapplicants 7% 32%
12% 26%
LORoutcomes
Anyrequested
88%
85%
Tworequested
77%
72%
Anyreceived
87%
84%
#received=#requested
83%
79%
Tworeceived
73%
67%
Admissionsoutcomes(shares)
Initialadmit 27% 40%
27% 35%
AdmitorWLoffer 33% 59%
33% 54%
Everadmit 30% 46%
30% 40%
Admitandmatriculate 17% 23%
16% 19%
Admissionsoutcomes(numbers)
Initialadmit 816 5,833
759 2,200
AdmitorWLoffer 995 8,438
923 3,414
Everadmit 907 6,672
841 2,558
Admitandmatriculate 503 3,364
459 1,211
AssessingthelettersofrecommendationcomponentofthechangeIbeginmyanalysisbyattemptingtoassesstheimpactoftheletterofrecommendationcomponentofthe2016admissionsprocess.Analysesbytheadmissionsofficehavecontrastedthosewhoprovidedletterstothosewhowereaskedforlettersbutdidnotprovidethem.Theseareusefulinunderstandingwhichstudentsmayhavetroubleobtainingletters(thoughasnotedabove,astudentwhodoesnotobtainlettersmightjusthavedecidedtogoelsewhere).Itakeadifferentapproach:ThefactorthatisunderBerkeley’scontroliswhetherapplicantsareaskedforletters,soIattempttouncovertheimpactofthisonadmissionsoutcomes,withouttryingtodistinguisheffectscomingfromdifficultyinobtaininglettersfromthosecomingfrom(forexample)thesubmissionofweakletters.Asithappens,thewaythattheLORprocesswasimplementedallowsforacompellinganalysisoftheLORrequest,basedonacomparisonofstudentsaskedforLORswithnearlyidentical
![Page 11: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/11.jpg)
11
studentswhojustmissedbeingasked.ThereweretwowaysthatstudentswereselectedforLORs:
• Theadmissionsofficeestimatedastatisticalmodeltopredictreadscoresin2015.3Those2016applicantswhosereadscoreswerepredictedtobe2.5,2.75,or3wereautomaticallyaskedforLORs.
• Applicantswhowerescoredbythefirstreaderas“Possible”(onthe2016Yes/Possible/Noscale),whenthiswasdonebeforethedeadlineforrequestingLORs,wereaskedforLORs.Abouttwo-thirdsofinitialreadsscored“possible”werecompletedbythedeadline.
Approximately80%ofthoseaskedforLORswereidentifiedbythefirstmethod,and40%bythesecondmethod.(20%wereidentifiedbybothmethods.)Although2015readscoresuseddiscretecategories(witheachreaderassigningascoreof1,2,2.5,3,or4,andwithlowernumbersgiventostrongerapplicants),thestatisticalmodelusedforthefirstmethodgeneratedcontinuouspredictions–thatis,anapplicantmighthavebeenpredictedtogetareadscoreof2.47.Studentswithpredictedreadscoresbetween2.38and3.26wereallaskedforletters,whilestudentswithreadscoresjustoutsidethisrangewereaskedonlyiftheywerecapturedbythesecondmethod.Figure6showstheshareofapplicantsateachpredictedreadscorewhowereaskedtosubmitletters.Forapplicantswithpredictedscoresbetween2.38and3.26,thiswas100%.Butonlyabout60%ofstudentswhowerejustabitstrongerthanthisrange(predictedscoresof2.37)wereaskedforlettersduetothefirstreader’sscore,andonlyabout3%ofstudentswhowerejustabitweakerthanthisrange(predictedscoresof3.27)wereasked.Thesesharpbreakspermita“regressiondiscontinuity”analysisoftheeffectoftheLORrequestonadmissionsoutcomes.Studentswithpredictedscoresof2.37areessentiallyidentical,onaverage,tothosewithpredictedscoresof2.38,andwouldalmostcertainlyhavehadverysimilaradmissionsoutcomeshadLORsnotbeenrequestedforsomanymoreofthelatter.Thus,anydifferenceintheiroutcomescanbeattributedtotheLORrequest.43Thepredictedreadscoreissimilarinspiritto,andreliesonthesamevariablesas,myadmissionsscorediscussedabove.Theydifferbecauselowerreadscoresarebetter,andbecausethepredictedreadscoreweightscharacteristicstopredictthe2015readscore,whiletheadmissionsscoreweightsthecharacteristicstopredictfirst-round2015admissions.Thecorrelationbetweenthetwoscoresisaround-0.85.4Anotherstrategymightbetocompareadmissionsoutcomesofthosewithpredictedreadscoresbetween2.38and3.26in2015and2016,relativetothoseoutsidethisrange.Unfortunately,Idonothaveaccesstopredictedreadscoresfor2015applicants,andhavebeenunsuccessfulinre-creatingthem.Iexpecttobeabletoeventually,butasIwritethisthekeystaffperson(GregDubrow,DirectorofResearchandPolicyAnalysisintheOfficeofUndergraduateAdmissions)isonvacation.Ithusdeferthistofuturestudy.
![Page 12: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/12.jpg)
12
Figure6.FractionofCaliforniaresidentapplicantsaskedforlettersofrecommendation,bypredictedreadscore
Figure7shows2016admissionsoutcomesasfunctionsofthepredictedreadscore.Notsurprisingly,studentswithlowerpredictedreadscoresaremorelikelytobeadmitted.Butnoticetheareaaroundtheverticallinesat2.38and3.26,whereIallowfortheaverageadmissionsoutcomestochangediscontinuously.Applicantstotheleftofthefirstline,only60%ofwhomareaskedforLORs,aresomewhatmorelikelythanstudentstotherightoftheline,allofwhomwereaskedforLORs,tobeadmitted,tobeinvitedtothewaitlist,andtoenroll.BecausethereisnoreasontoexpectdifferencesinoutcomesbetweenthesestudentsexceptforthedifferenceintheirLORtreatment,thisisclearevidencethatforstudentswithpredictedreadscoresaround2.38–strongerthan95%ofapplicantsand81%ofadmitsin2016–beingaskedforanLORreducedtheprobabilityofadmission.
Figure7.Admissionsprobabilitiesbypredictedreadscore,2016,allCaliforniaresidentapplicants
Nowturntothesecondline.Theremaybesmallerdiscontinuitieshere,generallypointingtohigheradmissionsprobabilitiesforthe3.25swhoweredefinitelyaskedforlettersthanforthe
0.2
.4.6
.81
Frac
tion
aske
d fo
r LO
R
1.5 2 2.5 3 3.5 4 4.5Predicted read score from adm. office model
Likelihood of LOR request by predicted read score
0.2
.4.6
.81
Prob
abilit
y
2 2.5 3 3.5 4Predicted read score
Initial admitAdmit or WL offerEver admitSIR
All applicants
![Page 13: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/13.jpg)
13
3.27swhohadonlya30%chanceofbeingasked.Thediscontinuitiesaresmallerhereandmaybeentirelyattributabletostatisticalnoise.However,thereweremanymoreapplicantswithpredictedscoresinthisrangethanaround2.38,soevenasmalleffectofLORson3.25studentswouldbequantitativelyimportant.ItisnotpossibletosaywhethertheLORimpactsseeninFigure7reflectbetteradmissionsdecisionsorworse,asitisnotpossibletoidentifythespecificstudentswhowereadmittedifnotaskedforlettersbutnotadmittedotherwise(orviceversa).ButitisworthnotingthatthisisexactlywhatwewouldexpectiftheLORsprovidedusefulinformation–somestudentswhowouldhavegottenthebenefitofthedoubtduetotheirstrongnumericcredentialswithoutLORswererevealedbytheLORstobeweakerthantheyappeared,whileotherswhowouldnothavegottenthebenefitofthedoubtwererevealedbytheirLORstobeworthadmitting.Ofcourse,theresultsarealsoconsistentwiththepossibilitythatsomestrongstudentsweredeniedadmissionbecausetheywereunabletoprovideLORs.Figure8repeatstheanalysisforapplicantsfromtheunderrepresentedgroups.Thesedataarenoisier,duetothesmallernumberofobservationshere.(Dotsaroundthe2.38thresholdrepresentanaverageof60applicantsinFigure7,andonly10-15inFigure8.)ThereisnosignherethatthestrongerstudentswerehurtbytheLORrequest,onaverage:Thoseontheleftofthe2.38lineareadmittedatessentiallythesamerateasthoseontheright.Figure8.Admissionsoutcomesbypredictedreadscore,2016,Californiaresidentapplicantsfromfourunderrepresentedgroups
Table3presentsquantitativeestimatesoftheeffectofanLORrequestonadmissionsoutcomes,separatelyforthosenearthe2.38thresholdandforthosenearthe3.26threshold.55Intechnicalterms,theseareinstrumentalvariablesestimatesfromafuzzyregressiondiscontinuitydesign.TheyreflectthelocalaverageeffectoftheLORrequestonstudentsneartherelevantthreshold.
0.2
.4.6
.81
Prob
abilit
y
2 2.5 3 3.5 4Predicted read score
Initial admitAdmit or WL offerEver admitSIR
Applicants from underrepresented groups
![Page 14: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/14.jpg)
14
Inthefullapplicantpool,strongstudentsfromwhomLORswererequestedwere8-10percentagepointslesslikelytobeadmittedthantheywouldhavebeenhadtheynotbeenselectedforLORs.Forweakerstudents,theeffectwastoincreaseadmissionschancesby2-4percentagepoints.(Notethatthereareaboutfourtimesasmanystudentsnearthe3.26thresholdasnearthe2.38threshold,sotheimpliednumberofstudentsadmittedduetolettersneartheformeriscomparabletothenumberdeniedduetolettersnearthelatter.)Forunderrepresentedapplicants,the2.38thresholdeffectissmaller,suggestingthatLORrequestswerenotharmfultostrongstudentsfromthisgroup.
Table3.RegressiondiscontinuityestimatesoftheeffectofLORrequestsontheprobabilityofadmissionCaliforniaresidents,excludingrecruitedathletes.Standarderrorsinparentheses
Initialadmit
AdmitorWLoffer
Everadmit
SIR
Allapplicants
Strongerapplicants(lowpredictedreadscores)
-10.4% -6.3% -7.6% -13.6%
(5.5%) (4.7%) (5.4%) (5.2%)
Weakerapplicants(highpredictedreadscores)
+2.0% +3.6% +4.2% +3.6%
(2.3%) (2.5%) (2.4%) (2.0%)
Applicantsfromfourunderrepresentedgroups
Strongerapplicants(lowpredictedreadscores)
-6.5% -6.5% -4.1% -2.3%
(9.4%) (8.3%) (9.3%) (7.9%)
Weakerapplicants(highpredictedreadscores)
-1.8% +0.7% +1.6% +3.2%
(3.8%) (4.1%) (3.9%) (3.2%)
Netimpactonnumberofadmittedstudents
Allapplicants
Number -304 54 40 -300
Proportion -4% 0% 0% -6%
Fromfourunderrepresentedgroups
Number -181 -60 19 123
Proportion -5% -1% 1% 5%
ThefinalrowsofthetableattempttoestimatethenetimpactofLORsonadmissionsateachstage–positivenumbersindicateapositiveneteffect,andnegativenumbersanegativenet
![Page 15: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/15.jpg)
15
effect.6Focusingonthelastcolumn,weseethattheLORrequirementraisedthenumberofunderrepresentedstudentswhoenrolledby123,whilereducingthenumberofenrolleesfromothergroupsby423.Thesearerelativelysmallnumbers,butshownosignofnegativeeffectsofLORsondiversityandindeedimplythatLORsraisedtheunderrepresentedshareofenrolledstudentsbyseveralpercentagepoints.Asnotedabove,wecannottellwhethertheLORaspectofthe2016proceduresledtobetterorworsedecisions.Butthereisnoindicationthatitreducedadmissionschancesforunderrepresentedorweakerstudents,whoseemmostlikelytohavefacedchallengesinobtainingsuitableletters.AssessingtheAugmentedReviewcomponentofthechangeThesecondmajorquestionIaddressiswhethertheeliminationofAugmentedReviewmadeitharderforthetypesofstudentsformerlyidentifiedforARtobeadmitted,orwhetherotherchangesmadetoadmissionsprocesseswereabletocompensateforthelackofaseparateARpool.Unfortunately,thereisnoregressiondiscontinuityresearchdesignavailableforassessingtheimpactofAugmentedReview.Moreover,thereisnowaytoidentifyinthe2016dataexactlywhichapplicantswouldhavebeenreferredtoARhaditbeeninplace,sowecannotcompareoutcomesforthispoolovertime.Asanalternative,Iidentifycandidateswho,basedontheircharacteristics,wouldhavebeenlikelytobereferredtoARin2015,andexaminehowtheiroutcomeschangedovertimerelativetootherswho,basedontheircharacteristics,wouldhavebeenunlikelytohavebeenreferredtoAR.Specifically,Icreateyetanotherscorefromthesamevariablesconsideredtodate,thisonerepresentingthelikelihoodthatastudentwiththesecharacteristicswouldhavebeenreferredtoandconfirmedforARin2015.7
6Thesecalculationsrequireratherheroicassumptions.Iassumethattheestimatedeffectsfoundatthetwodiscontinuitiesextendidenticallyoutsidethem,andIlinearlyinterpolateeffectsbetweenthediscontinuities.Imakenoallowanceforsamplingerrorinthisextrapolation.7Specifically,Ifitalogisticregression,using2015data,wheretheoutcomeisanindicatorforhavingbeenreferredtoandconfirmedforARandpredictorsareaquarticinthepredictedreadscore,indicatorsforthethreedisadvantagegroupsandinteractionsamongthem;andindicatorsandseparatequadraticsinthepredictedreadscoreforthosewith1,2,or3disadvantagefactors.(Amoreflexiblemodelthatincludesalloftheunderlyingvariablesinthemodelusedtogeneratethepredictedreadscoredoesnotgeneratemeaningfullybetterpredictions.)
![Page 16: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/16.jpg)
16
50%ofapplicantshaveARprobabilitiesbelow2%,but10%haveprobabilitiesabove20%.Figure9showshowtheARprobabilityscorerelatestothereadscore,separatelyforstudentswithdifferentnumbersofdisadvantagefactors.(Themultiplelinesineachseriesrepresentdifferentcombinationsofwhichdisadvantagefactorsthestudenthas.)Acrossalldemographicgroups,studentswithpredictedreadscoresnear3.25aremorelikelytobeconfirmedforARthanthosewithhigherorlowerpredictedreadscores.Foranygivenpredictedreadscore,ARprobabilityscoresarehigherforthosewithmoreenumerateddisadvantagefactors.Forstudentswiththreedisadvantagefactorsandpredictedreadscoresbetween2.64and3.91,ARprobabilityscoresareabove0.2,andsometimessubstantiallyso.Studentswhohaveonlytwodisadvantagefactorsmusthavepredictedreadscoresinanarrowerrange,betweenabout3.1and3.7,toachieveARprobabilityscoresthishigh,whilestudentswithzerooronedisadvantagefactorsneverhaveARprobabilitiesabove0.12.
Figure9.EstimatedprobabilityofAugmentedReviewbypredictedreadscoreandnumberofunderrepresentationfactors,Californiaresidentsin2015
Unfortunately,whilethispredictionmodelisfairlysuccessful,itdoesnotachieveasharpdistinctionbetweenARandnon-ARstudents–eventhestudentswiththeabsolutehighestARprobabilitieshaveonlya40%chanceofbeingconfirmedforAR.Inlightofthis,IconsidertwodefinitionsofstudentsmostlikelytobeconsideredintheARpool:
- StudentswithARprobabilityscoresabove0.2(10%ofapplicantsand38%ofthoseconfirmedARin2015)
- Studentswhoarelowincome,firstgeneration,andfromlowAPIschools,withARprobabilityscoresabove0.2(7%ofapplicants,and28%ofthoseconfirmedforARin2015).
Table4showstheadmissionsoutcomesforstudentsineachofthesegroupsin2015and2016,aswellasfortheircomplements(studentswithlowerARprobabilityscores).Relativechangesatallmarginsexceptinitialoutcomesarepositiveorclosetozero.(Acrosseachdefinition,therelativechangesaremostinfavorofthehigh-riskgroupwhentheoutcomeisadmissionorthe
0.1
.2.3
.4Pr
obab
ility
refe
rred
and
confi
rmed
for A
R
2.5 3 3.5 4 4.5Predicted read score
3 factors2 factors1 factor0 factors
Probability of AR confirmation, by predicted read score andnumber of disadvantage factors, 2015
![Page 17: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/17.jpg)
17
offerofawaitlistspot–itseemsthatmanystudentswhowouldhavebeenintheARpoolin2015wereofferedwaitlistspotsin2016butnotadmitted,eitherbecausetheywerenot
Table4.Probabilityofadmission,byARprobability,2015and2016Californiaresidents,excludingrecruitedathletes
Initialadmit
AdmitorWLoffer Everadmit SIR
Definition1:PredictedARprobability>20%
HighprobabilityAR
2015 17.5% 21.5% 19.4% 9.9%
2016 18.4% 32.5% 22.5% 12.3%
Change 0.9 11.0 3.1 2.5
ChangeforlowprobabilityARgroup
2.7 6.7 2.9 2.3
Differenceinchanges -1.8 4.3 0.2 0.2
Definition2:3disadvantagefactorsandpredictedARprobability>20%
HighprobabilityAR
2015 16.3% 19.2% 17.5% 8.9%
2016 16.0% 28.5% 19.7% 11.0%
Change -0.3 9.3 2.2 2.0
ChangeforlowprobabilityARgroup
2.7 6.9 3.0 2.3
Differenceinchanges -3.0 2.4 -0.8 -0.3
ImpactofAReliminationonnumberadmitted
Definition1 -248 603 30 25
Definition2 -396 318 -105 -38
selectedfromthewaitlistorbecausetheydeclinedtheoffer.)NeitherofthedefinitionsindicatesmeaningfuleffectsofARonthenumberofstudentswhoenrolled,andingeneral,itishardtodiscernchangesofmeaningfulmagnitudeintheadmissionsoutcomesofAR-typestudentsbetweenyears,suggestingthatchangesinotheraspectsoftheadmissionsprocessenabledthesestudentstogettheextraconsiderationin2016thattheygotthroughARin2015.TowardanunderstandingoftheoverallimpactofadmissionprocesschangesTheresultsthusfarsuggestthatLORs,ifanything,increaseddiversityoftheenteringclassin2016,andthattheeliminationofARhadatrivialeffect.Buttherewereanumberofotherchangesmadein2016,andoveralltheimpactwassomewhatlessthansatisfactory–theshareofstudentsfromunderrepresentedgroupsamongadmittedstudentsfell,thoughtheshareamongstudentswhoenrolledwasstable.Inthisfinalsection,Ipresentsomeanalysesofoveralloutcomesthatpointtopossiblecontributingfactors.
![Page 18: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/18.jpg)
18
Figure10showsestimatesoftheshareofstudentsateachapplicationscorewhoweresuccessfulin2015and2016,foreachofthedefinitionsofsuccessdefinedearlier.Here,Iadjustthe2015applicantpooltomatchthedistributionacrosscollegesseenin2016,toremovetheinfluenceofshiftsacrossadmissionsprocessesthataremoreorlesscompetitive.Weseethatweakerapplicants(asmeasuredbyadmissionsscoresbetween0.1and0.4)weremorelikelytobeadmittedin2016thanin2015,butstrongerapplicants(scoresabove0.7)weresomewhatlesslikelytobeadmitted.Thelatterchangedisappearswhenweincludewaitlistedstudentswithinitialadmits,butitreappearsandisevenlargerwhenweexaminetheshareofapplicantswhowereeveradmitted(countingasfailuresthosewhowererejectedoutrightaswellasthosewhowereofferedwaitlistspotsbuteitherdeclinedthemorwerenotadmittedoffthewaitlist).Bycontrast,thestrongestapplicantsweremorelikelyin2016thanin2015tomatriculate.
Figure10.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentsin2015and2016
Thecontrastbetweenthe3rdand4thpanelsisinformative–itsuggeststhatsomeverystrongstudentswhowereadmittedin2015butmatriculatedelsewherewerereclassifiedasnon-admitsin2016,eitherbecausetheadmissionsdecisiontookaccountinsomewayofthelikelihoodofmatriculationorbecausethesestudentsdroppedoutatthewaitliststage.Inanyevent,weseethatboththeverystrongestandweakerapplicantsweremorelikelytomatriculatein2016thanin2015,whiletherewaslittlechangeforthoseinthemiddlerange(between0.4and0.8).Figure11repeatsthisexercise,thistimeonlyforapplicantsfromthefourunderrepresentedgroups.Asintheoverallpool,weseeincreasedadmissionschancesin2016forapplicantswithadmissionsscoresaround0.2.Buthereweseefairlydramaticdeclinesinadmissionsofapplicantswithscoresabove0.5thattranslateintoreducedmatriculationaswell.
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
20152016
Initial admit
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
Admit or WL offer
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
Ever admitted
0.1
.2.3
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
Matriculated
Admissions probabilities, 2015 and 2016All applicants
![Page 19: UC Berkeley Admissions in 2015 and 2016: An Analysis of ...jrothst/otherwriting/rothstein_report_july... · An Analysis of the Role of Letters of Recommendation and Augmented Review](https://reader033.fdocuments.us/reader033/viewer/2022060208/5f04222c7e708231d40c7a9e/html5/thumbnails/19.jpg)
19
Figure11.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentapplicantsfromunderrepresentedgroups
Evidently,somethingintheadmissionsprocessesusedin2016reducedtheadmissionschancesofthestudentsfromunderrepresentedgroupswhowere,by2015standards,thestrongestintheirobservedcharacteristics.Onecandidateexplanationistheuseofnon-cognitivescores,whichmighthavebeensubtlybiasedagainststudentsfromunderrepresentedgroups;anotheristhatreadersmighthaveputlessweightonthefactorsmeasuringstudentsrelativetotheirschoolsinevaluating2016applications.Unfortunately,inthelimitedtimeIhadtopreparethisreport,Iwasnotabletogettothebottomofthischange.Itbearsfurtherstudy.Itisworthnoting,however,thatFigures10and11constitutestrongevidenceagainsttheviewthattheeliminationofARplayedamajorrole–recallthatARstudentsareconcentratedaroundadmissionsscoresnear0.2,whereadmissionschanceswentupthemostin2016.ConclusionBerkeleyadmissionsoutcomesforunderrepresentedstudentswere,bysomemeasures,disappointingin2016:Althoughmorewereadmittedoverall,andtheirshareofenrolledstudentswassteady,theymadeupalowershareofadmissionsoffersandparticularlyoffirst-roundadmissionsoffers.ItwasnaturaltowonderwhethertheeliminationofAugmentedReviewandtheadditionofLettersofRecommendationcontributedtothischange.Myanalysisoffersnosupportforthesepossibilities.Lettersofrecommendationseemtohavehurttheadmissionschancesofotherwise-strongapplicantsnotfromunderrepresentedgroups,withsmallerornoeffectsonapplicantsfromthosegroups,andthustohaveraisedtheshareofunderrepresentedstudentsamongadmissions.ItisdifficulttoidentifyanycleareffectofAReitherway,butinanyeventitwassmall.Theexplanationforthechangeinoutcomesin2016mustlieelsewhere,inoneoftheotherchangesmadetoadmissionsprocesses.
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
20152016
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
0.0
5.1
.15
.2.2
5Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
Admissions probabilities, 2015 and 2016Applicants from underrepresented groups