Evaluating the Fairness, Impact, Use and Predictive ... the Fairness, Impact, Use and Predictive...

Evaluating the Fairness, Impact, Use and Predictive Validity of the New MCAT Exam

Catherine Lucey, MDVice Dean for Education

University of California, San Francisco School of Medicine

Joshua Hanson, MD, MPHAssistant Dean for Student Affairs, Assistant Professor

The University of Texas School of Medicine at San Antonio

Carol Terregino, MDSenior Associate Dean for Education, Associate Dean for Admissions; Professor of Medicine

Rutgers Robert Wood Johnson Medical School

Daniel Clinchot, MDAssociate Vice President for Health Sciences Education; Chair, Department of Education and Anatomy

The Ohio State University College of Medicine

CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2

Diversity is a Core Driver of High Quality Health Care Minority physicians are significantly more likely to

practice primary care and in underserved areas Similar race/ethnicity identification between

physicians and patients results in longer visits and increased patient satisfaction

Communicating in patients’ native language often improves adherence to treatment

Medical students uniformly agree that a diverse student body improves the experience of medical school; enhances classroom discussions

Whitla DK et al. Acad Med. 2003; 78: 460Hung R et al. Acad Med.2007; 82(2):184AAMC Diversity in the Physician Workforce Facts and Figures 2014

Black/African-Americans comprise only 4 percent of the physician workforce

The majority of younger minority physicians in the workforce are women


Slightly More Than Half of 2016 Matriculants Self-Identified as White

0.3%

21.3%

7.1% 6.3%0.1%

51.5%

8.8%1.7% 1.6% 1.3%

0%

10%

20%

30%

40%

50%

60%

70%

AmericanIndian orAlaskaNative

Black orAfrican

American

NativeHawaiian/

OtherPacific

Islander

More thanOne Race

UnknownRace/

Ethnicity

Percentage of U.S. Medical School Matriculants by Race/Ethnicity, 2016

Native Hawaiian/

Other PacificIslander

Asian Hispanic orLatino

White Non-U.S. Citizen/

Non-Perm.Resident

Note: The “Non-U.S. Citizen and Non-Permanent Resident” category may include students with unknown citizenship. Matriculants who declined to report sex are not reflected. Students who reported more than one race/ethnicity are included under “More than One Race”. During the 2016 application cycle, a technical malfunction in the collection of race/ethnicity data necessitated a request that applicants review and re-submit responses to the race/ethnicity question in their AMCAS applications. Applicants in previous years were not asked to review responses to this question.

Other


The Path to Becoming a Physician Starts Early

Pre-College Experiences• Early environment• Family support• K-12 education• Extracurricular

experiences

Premedical Preparation• # colleges attended• Quality of instruction• Coursework• Extracurricular

experiences

Medical Education• Family and financial

support• Academic and other

support• Culture and climate

Residency Program• Academic and other


Medical School Admissions

• Holistic review (Including MCAT)

• Diverse class

Residency Selection

• Holistic review (Including USMLE)

• Diverse cohort


Group Differences in Academic Achievement Are Associated with Societal Inequalities MCAT, LSAT, GRE, GMAT, and other exams show population

group differences Presence of differences does not equate to test bias

(i.e., construct irrelevant content or alterations in administration)

Structural racism and privilege likely contribute to the differences seen across the spectrum of exams

• More likely to experience adverse environmental factors (poverty, food insecurity, low quality day care)

• More likely to have had disrupted or low quality K-12 education

• Less likely to have high quality exam prep experiences or advising experiences in college

Compared with majority

examinees, minority

examinees:

Davis D et al. Acad Med. 2013;88:593Dietrichson, J., Martin B., Filges, T., and Jorgensen, AMK (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research. 87, 243-283.


Type of Fairness Definition Influence on New

MCAT ExamSocietal Fairness

Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam.

Blueprints, test preparation resources

Procedural Fairness

Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions.

New score scales, score reports, and resources for admissions officers

Exam Fairness

MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds.

Item development/review, test form development

Fairness was Front and Center in Designing and Developing the New Exam


Overview of the MCAT Validity Research Program


21 Medical Schools and 2 Pre-Health Advisors Are Working Together to Evaluate the New Exam


The MCAT Validity Research Program is Complex Multiple research questions in three broad areas Qualitative and quantitative data from

ExamineesApplicantsMedical studentsMedical schools

Multiple methods and data collection designs (longitudinal, cross sectional)

The validity study will last about nine years (2014 to 2023)


The MCAT Validity Research Addresses Multiple Goals Provides evidence about the value of the new

MCAT exam in admissions decisions Answers questions about the fairness and

consequences of introducing the new MCAT exam for examinees, applicants, and medical students

Presents data to admissions officers that they can act on to improve their admissions decisions

Uses findings about the needs of aspiring physicians from underrepresented backgrounds to improve test preparation resources and outreach


The MCAT Validity Research Agenda Includes Three Broad Areas

Predicting Medical Student

Performance

Admissions Decision Making

Academic Preparation,

Diversity, and Fairness


We’re Testing 3 Hypotheses about Predicting Medical Student Performance

Do scores from the new exam correlate with academic performance throughout medical school?

Evaluating the predictive validity of

the new exam

Will scores from the Psychological, Social, and Biological Foundations of Behavior section correlate with performance in medical school courses that call on the behavioral and social sciences better than section scores from the old exam?

Examining the predictive validity of

the newest test section

Do scores from the new exam add value to the academic information applicants already provide about themselves through applications and transcripts?

Comparing the predictive validity of the new MCAT exam to other predictors


We’re Testing 2 Hypotheses about Admissions Decision Making

Will medical schools increase the percentage of applicants with total scores in the middle of the MCAT score scale who are invited to interview and receive acceptance offers?

Acceptance of a wide range scores

Will admissions committees use information about applicants’ strengths and weaknesses from the MCAT score reports to identify applicants who best fit their academic missions and goals?

Use of section scores


We’re Testing 9 Hypotheses about Academic Preparation, Diversity, and Fairness, For Example:

Will more individuals learn about psychology, sociology, and biochemistry in preparation for the MCAT exam?What resources, information, and outreach will provide equity in access for students from sociodemographic groups underrepresented in medicine?

Change in breadth of academic preparation

Will the diversity of examinees, applicants, and medical students who took the new exam change?

Diversity of aspiring physicians

Will scores from the new exam predict academic performance equally well for medical students from different racial, ethnic, or disadvantaged backgrounds?

Fairness in score meaning


How well did scores from the new MCAT

exam predict students’

performance across the first year of medical school?

How did admissions officers work with new MCAT scores in 2017 admissions

decisions?

What can we learn about preparation

resources needed by examinees from

underrepresented backgrounds?

Today’s Presentation Will Focus on Three Questions

1616

How well did scores from the new MCAT exam predict students’ performance

across the first year of medical school?


Validity School

Participants

The Predictive Validity Research Includes National and Validity School Data (2015-2017 Entrants)

National Population with New Scores

National Population


MCAT Scores Will Be Correlated with Multiple Types of Student Performance Outcomes

Holistic Medical Student Academic Performance

Academic Readiness to Graduate

Academic Readiness for Clerkships

Year-End Academic PerformanceCourse-Based

Academic Performance

Holistic Outcomes

Granular Outcomes







Some Analyses Will Include Outcomes from the National Population

Holistic Outcomes

Granular Outcomes

National OutcomesProgression to M2, M3USMLE Step exams

Graduation







Other Analyses Will Include Outcomes from the Validity Schools

Holistic Outcomes

Granular Outcomes

Outcomes from Validity SchoolsCourse grades

NBME Subject examsClerkship gradesGPA/Class Rank


Today’s findings are from the 2016 entrants with new scores

w/ new scoresN=8,521

Participants

N=1,027 w/ new scores

2016 MatriculantsN = 21,030

Matriculants


2016 Matriculants and Validity School Participants with New Scores Are Similar to the 2016 Matriculant Population

Demographics Fee assistance status Parental education/occupation Race/ethnicity

Academic metrics MCAT total and section scores Undergraduate GPAs

w/ new scoresN=8,521

Participants

N=1,027 w/ new scores

2016 MatriculantsN = 21,030

Matriculants







Today’s Results Include Two M1 Year-End Outcomes

Holistic Outcomes

Granular Outcomes

National OutcomeProgression to M2

without delay

Outcome from Validity Schools

Performance Across M1 Courses


Nationally, 2016 Entrants w/ a Wide Range of Scores Progressed to M2 Without Delay

% S

tude

nts

Prog

ress

ing

to M

2

MCAT Total Score Range

50%

80% 81%

93% 94% 96% 97% 98% 98% 99%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528

% of 2016 Entrants Who Progressed to M2 Without Delay

(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)

Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.


At the Validity Schools, New Scores Predict Performance across M1 Courses

r Effect Size0.1 Small0.3 Medium0.5 Large

0.43

0.69

0.55

00.

20.

40.

60.

81.

0C

orre

cted

Cor

rela

tion

Overall(NSchool = 15)

Medium Effect

Correlation of MCAT Total Scores with Performance Across M1 Courses: Median and Interquartile Range

Overall, the associations of the new MCAT total scores with performance across M1 courses are medium to large

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155


These Validities Compare Well to Those for Other Exams

Author(Year) Exam

Type of ExamScore

Type of Outcome

Median Validity

CoefficientUnit of

AnalysisAAMC (2017) New Exam Total Score First-Year Course Average .55 School

(Nschool=15)AAMC (2017) Old MCAT Total score First-Year Course Average .53 School

(Nschool=17)Kuncel, et al. (2007) GMAT Total score First-Year Graduate GPA .47 Meta-

analysis of multiple studies

Stilwell, et al. (2011) LSAT Total score First-Year Average Course Grade .58 School(Nschool=170)







At the Validity Schools, We Also Analyzed Outcomes from Individual M1 Courses

Holistic Outcomes

Granular Outcomes


At the Validity Schools, New Scores Predict Performance in Individual M1 Courses

Overall, the associations of the new MCAT total scores with performance in M1 courses are medium to large


0.40

0.61

0.51

00.

20.

40.

60.

81.

0C

orre

cted

Cor

rela

tion

(NCourse = 109)

Medium Effect

Correlation ofMCAT Total Scores with

Performance in Individual M1 Courses:

Median and Interquartile Range


0.40 0.40 0.40

0.61 0.62 0.64

0.51 0.51 0.50

00.

20.

40.

60.

81.

0C

orre

cted

Cor

rela

tion

Overall(NCourse = 109)

GeneralPrinciples ofFoundational

Science(NCourse = 66)

BodySystems

(NCourse = 46)

Course Content Area

At the Validity Schools, New Scores Predict Performance in Courses about Foundational Sciences and Body Systems


Medium Effect

Correlation of MCAT Total Scores with Performance in Individual M1 Courses by Course Content Area:

Median and Interquartile Range


New Scores Predict Equally Well for Students from Different Sociodemographic Backgrounds Research studied these early relationships for students

grouped by: Race/ethnicity Socioeconomic status

So far, MCAT scores neither over- nor under-predict the performance of students from underrepresented backgrounds based on two types of performance outcomes: National outcome: Progression to M2 without delay Validity school outcome: Performance across M1

courses


Predictive Validity Findings Will Be Reported from 2017 through 2022 for Medical Students Who Took the New Exam

2016 Entrants

2017 Entrants 2018 2019 2020 2021 2022

We have a lot more to learn about how students do in their second year, in their clerkships and on their USMLE exams, and their graduation from undergraduate medical school

3232

How did admissions officers work with new MCAT scores

in 2017 admissions decisions?


Admissions Committees Put MCAT Scores in Context There are more qualified applicants to medical school than there

are places in medical school Medical schools weigh information about attributes,

experiences, and academic metrics in deciding whom to admit Applicants from sociodemographic groups underrepresented in

medicine obtain lower mean scores on the MCAT exam and lower undergraduate GPAs than applicants from other groups

Medical schools aim to select students who will succeed and who will help build a diverse physician workforce


The New Score Scales Draw Attention to the Center of the Scale Because on the old exam, students who entered

medical school with scores in the center of the MCAT score scale succeeded

The new scale uses a nice round number at 500 to draw attention to applicants who might otherwise be overlooked

The new test also includes more questions per section, providing better information about examinees’ strengths and weaknesses on the exam

The new score reports use confidence bands to describe measurement precision and score profiles to describe strengths and weaknesses


Admissions Committees Put MCAT Scores in Context in 2017 Selection

Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range

New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%

2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802


Some 2017 Applicants w/ High UGPAs and MCATs Weren’t Accepted



2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

11% of applicants with GPAs at or above 3.8 and

MCAT scores at or above 518 were

not admitted into any medical

schools


Other 2017 Applicants w/ Modest Credentials Were Accepted



2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

14% of applicants with GPAs 3.00 - 3.19

and MCAT scores 498-501 were

admitted into at least one medical

school


Progression of 2016 Entrants to M2 Without Delay, By New MCAT Total Score Range

% S

tude

nts

Prog

ress

ing

to M

2

MCAT Total Score Range

50%

80% 81%

93% 94% 96% 97% 98% 98% 99%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528

% of 2016 Entrants Who Progressed to M2 Without Delay

(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)

Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.



2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802


Let’s put these two data sources together


Early Data Suggest Students with Scores in the Middle of the New Scale Do Well

50%

80% 81%

93% 94% 96% 97% 98% 98% 99%

1% 1% 4%11%

22%

36%

50%

64%

75%84%

0%10%20%30%40%50%60%70%80%90%

100%

472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528

% of 2016 Entering Medical Students Who Progressed to M2 Without Delay, and % of 2017 Applicants Accepted to Medical School

Progression to M2 (2016 Entrants) Acceptance Rates (2017 Applicants)


As a community, should we ask ourselves if there is more we want to do?


As a Community, Should We Ask Ourselves if There Is More We Want to do? Admissions committees identify the applicants with the

ingredients for success, and they support these students once they’re admitted

Are we missing some opportunities to pay more attention to students with scores in the middle of the MCAT score scale?

There is still a lot to learn about students’ performance in the second year and in their clerkships

On the new exam, findings so far suggest that, just like they did with the old exam, students admitted with scores in the middle of the new MCAT score scale do well


Interested in Learning More about Your Colleagues’ Use of New Exam Scores?

Strategies for Mission-Oriented Admissions in Light of the New MCAT Exam Monday, November 6 at 3:00 PM ET

Center 304-306

Tanisha Price-Johnson, PhD, University of Arizona-Tuscon College of MedicineTheodore Hall, MD, David Geffen School of Medicine at UCLAAaron Saguil, MD, F. Edward Herbert School of Medicine, Uniformed Services University Sunny Nakae, PhD, Loyola University Chicago Stritch School of MedicineCarol Teitz, MD, University of Washington School of Medicine

4545

What can we learn about preparation resources needed by examinees from

sociodemographic groups underrepresented in medicine?


Type of Fairness Definition Influence on New

MCAT ExamSocietal Fairness

Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam.

Blueprints, test preparation resources

Procedural Fairness

Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions.

New score scales, score reports, and resources for admissions officers

Exam Fairness

MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds.

Item development/review, test form development

The New MCAT Exam Was Designed to Increase Access


The New Test Blueprints Were Developed to Advance Societal Fairness Test concepts widely taught at undergraduate institutions,

including minority-serving and under-resourced institutions Test psychology and sociology concepts like discrimination,

stereotype threat, and socio-economic inequalities Pay increased attention to population health, studies of

diverse cultures, and ethics Balance the percentage of questions devoted to natural

sciences concepts with the percentage devoted to behavioral and social sciences concepts and information processing


Preparation Resources Were Developed to Advance Societal Fairness The Khan Academy has over 1,100 free tutorials on exam content Practice materials and resources are available on AAMC’s website: What’s on the MCAT Exam? Interactive Content Outline Roadmaps to MCAT Content in Biochemistry, Psychology, and

Sociology Textbooks Guide to Creating a Study Plan How I Prepared for the MCAT Exam Testimonials Practice tests and question banks

Students-residents.aamc.org/mcatprep


New Outreach Strategies Address Societal Fairness Expand outreach directly to students from sociodemographic

groups underrepresented in medicine Expand outreach to students through their advisors, with a

particular focus on faculty at under-resourced institutions Distribute a monthly newsletter Premed Navigator with important

information, resources, and tips Work with prehealth advisors on the MCAT Validity Committee

to share findings and promote resources, such as the “Find an Advisor” resource for students at schools with no access to an advisor ([email protected])

mailto:[email protected]?subject=I%20am%20looking%20for%20a%20health%20professions%20advisor&body=Please%20indicate%20the%20undergrad%20school%20that%20you%20are%20attending,%20or%20have%20graduated%20from%20and%20when%20you%20graduated.


Societal Fairness: Expanding Outreach to Faculty at Under-resourced and Minority-Serving Institutions Developed new contacts at minority-serving and/or under-

resourced colleges and universities—adding 350 contacts from: Almost 90 HBCUs More than 200 Hispanic-serving institutions More than 30 Tribal colleges and universities

Add to the list multiple times/year

Please encourage your advisor colleagues to add their names to our mailing list. They can contact us at [email protected]

mailto:[email protected]


Societal Fairness: Expanding Outreach Directly to Students from Sociodemographic Groups Underrepresented in Medicine Reach out to students directly through emails and newsletters Participate in health professions career fairs focused on minority

populations (e.g., UC Davis Pre-Health Conference, Stanford University Minority Medical Alliance Conference)

Participate in virtual events like Health Professions Week Virtual Fair and AAMC Medical School Virtual Fair

Monitor students’ use of MCAT preparation products to determine if better outreach about available resources is needed


Despite This Work, Group Differences Still Exist• The group differences in scores on the new exam are similar to

those on the old exam• They are similar to those on other standardized tests:

o GREo LSATo GMAT


Now We’re Studying How Students Prepare for the MCAT Exam

Pre-College Experiences• Early environment• Family support• K-12 education• Extracurricular

experiences

Premedical Preparation• # colleges attended• Quality of instruction• Coursework• Extracurricular

experiences

Medical Education• Family and financial

support• Academic and other


Residency Program• Academic and other


Medical School Admissions

• Holistic review (Including MCAT)

• Diverse class

Residency Selection

• Holistic review (Including USMLE)

• Diverse cohort


Review Courses Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds

43% 40%

6% 9%

52% 51%

25% 27%

020

4060

8010

0

CommercialPrep. Courses

University-BasedPrep. Courses

Khan Academy None of the Three

Use of Preparation Courses and Khan to Prepare for the ExamBy URiM Status (Race/Ethnicity)

Non-URiMURiM

Note. Data came from examinees who took the MCAT exam and in 2017.

44%

33%

7% 7%

52% 52%

24%30%

020

4060

8010

0




Use of Preparation Courses and Khan to Prepare for the ExamBy First-Generation College Status

Non-First Generation College StudentsFirst-Generation College Students


Practice Resources Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds

76%71%

53% 51%

21% 24%

020

4060

8010

0

Practice Exams Questions Packs &Section Banks

None of the Two

Use of AAMC Resources to Practice for the ExamBy URiM Status (Race/Ethnicity)

Non-URiMURiM

Note. Data came from examinees who took the MCAT exam and responded to the Post-MCAT Questionnaire in 2017.

75% 73%

53% 53%

21% 23%

020

4060

8010

0


None of the Two

Use of AAMC Resources to Practice for the ExamBy First-Generation College Status

Non-First Generation College StudentsFirst-Generation College Students


Gaps in the Use of Review Courses and Practice Resources Are Larger for Those Underrepresented on Both Race/Ethnicity and Parental Education

Note. Data came from examinees who took the MCAT exam and responded to the Post-MCAT Questionnaire in 2017.

44%

33%

6% 9%

52% 50%

24%31%

020

4060

8010

0




Use of Preparation Courses and Khan to Prepare for the ExamBy URiM Status (Race/Ethnicity) and First-Generation College Status

Neither URiM Nor First-Generation College StudentsURiM and First-Generation College Students

76%71%

53% 52%

21% 24%

020

4060

8010

0


None of the Two

Use of AAMC Resources to Practice for the ExamBy URiM Status (Race/Ethnicity) and First-Generation College Status

Neither URiM Nor First-Generation College StudentsURiM and First-Generation College Students


We are Conducting Qualitative and Quantitative Research to Understand Students’ Preparation Strategies and Barriers What is easy and difficult for examinees when they prepare for

the MCAT exam? What is easy and difficult about using the AAMC’s free and low-

cost materials to prepare for the MCAT exam? Are these barriers different or greater for those from

sociodemographic groups underrepresented in medicine? What additional resources and information do examinees and

their advisors need?


Interview Students

andAdvisors

Revise PMQ

Administer PMQ and Compare

Results for Examinees

from Different Groups

Learn what barriers

exist

Develop new

information and

resources

Findings Will Be Used to Improve Access to Resources and Information


Questions?

Stay tuned for new research findings

aamc.org/validitycommittee


Don’t forget to join us tomorrow!

The New MCAT Exam:Where We Are and Where We’re Going

Time: 7:00 AM - 8:30 AM Location: Center Ballroom C


References AAMC. (March, 2017). Predicting medical student performance. Presented at

the MCAT Validity Committee meeting at the Association of American Medical Colleges in Washington, D.C.

Kuncel, N., Credé, M., & Thomas, L. (2007). A meta-analysis of the predictive validity of the graduate management admission test (GMAT) and undergraduate grade point average (UGPA) for graduate student academic performance. Academy of Management Learning & Education, 6(1), 51-68. Retrieved from http://www.jstor.org/stable/40214516

Stilwell, L. A., Reese, L. M., & Diamond-Dalessandro, S. P. (2011). Predictive validity of the LSAT: A national summary of the 2009–2010 LSAT correlation studies. LSAT Technical Report Series, 11(2), 1-19. Retrieved from https://www.lsac.org/docs/default-source/research-(lsac-resources)/tr-11-02.pdf

http://www.jstor.org/stable/40214516

https://www.lsac.org/docs/default-source/research-(lsac-resources)/tr-11-02.pdf

Evaluating the Fairness, Impact, Use and Predictive ... the Fairness, Impact, Use and Predictive...

Documents

Transcript of Evaluating the Fairness, Impact, Use and Predictive ... the Fairness, Impact, Use and Predictive...