Real and Imaginary Concerns in Assessing
College Readiness
Nathan R. Kuncel
Department of Psychology
Favored Career Choices: In Rank Order
1. Jedi Knight2. Star Fleet Captain3. Batman4. Playboy Millionaire5. Psychology Professor6. Evolutionary Biologist7. Economist8. Brilliant but Misunderstood Novelist
Jedi Knight Innate heritable ability, unrelated to social class, focused through mentoring and intense training, high self control
Star Fleet Captain Born but seasoned leaders who succeed through
managed team effectiveness, charismatic and bold
Batman Single formative childhood experience that motivates massive skill investment, augmented by tremendous family wealth, intense and disagreeable
Professor of Psychology Masochistic enthusiasm for studying nearly intractable research questions, fairly friendly
Stories We Tell About Achieving Success
• Predictable versus Unpredictable
• Book Smarts versus Street Smarts
• General versus Specific Aptitudes
• Social Class Dependent versus Not
• Talent versus Effort
• Linear versus Asymptotic
Stories about Tests– Tests don’t even predict grades well– Tests don’t add anything to prediction– Tests don’t predict anything other than grades– Cognitive tests only predict academic outcomes – Cognitive tests are only proxies for SES– Beyond a certain point, scores don’t matter
Stories about Traditional Alternatives– Letters of Recommendation– Personal Statements– Interviews
Frightening Cautionary Tales about the Future– Faking– Bias– Unreliability
Assertion: Tests Don’t Even Predict
Grades Well
What does “well” mean and how do we quantify it?
Zombie Plague
• Plague sweeps the world
• Plague transforms 60% of infected people into horrifying and mindless monsters.
A Partial Treatment Is Developed!
Zombie Live
No Treatment 60% 40%
Treatment 40% 60%
What is the correlation?
Tests and Grades
• Berry and Sackett (2009) examined SAT-grade r’s in a sample of 165,000 students from 41 schools
• Corrected for restriction of range using school-specific applicant pools
• Obtained individual course grades for each course, and computed validity for each course (148,072 validity coefficients)– This removes influence of student choice of courses– Estimate r between SAT and common course portfolio
Berry and Sackett (2009) results
SAT-First Year College GPA
r
observed .36
corrected for range restriction
.46
corrected for course difficulty
.55
Story:Tests Don’t Add Anything
to Prediction
Bridgeman, Burton, & Cline (2009)
Percent Earning a 3.8 or Higher Graduate GPA
Story: Tests only predict 1st year grades
• Paper submitted to American Psychologist: “As is well known, the SAT predicts nothing but first year grades” [with no citation]
Evidence Across All Admissions Tests: Kuncel & Hezlett (2007) Science
• 7 Admissions Tests– GRE-T– GRE-S– MCAT – LSAT– GMAT– MAT– PCAT
• 8 Outcomes– 1st Year GPA– Graduate GPA– Faculty Ratings– Degree Attainment– Citation Counts– Research Productivity– Quals./Comps. Exams– Licensing Exams
Results are based on over 600,000 students across over 3,000 independent samples
Kuncel & Hezlett (2007). Science, 315, 1080-1081.
If the critics were right….
Book Smarts vs. Street Smartsor
Academic vs. Practical Intelligenceor
Book Learning vs. Common Sense
Book Smarts vs. Street Smarts
• Fundamental argument is that different types of intelligence are needed in academic versus real world contexts
• It is argued that academic contexts require an “academic intelligence” or book smarts
• It is my belief that these arguments are based (in part) on an overly narrow conceptualization of the tasks that comprise academic performance
Academic versus Practical TasksAcademic
• Formulated by others• Well-defined• Presents complete
information• Single method for
obtaining answer• Not embedded in
ordinary experience• Little or no intrinsic
interest
Practical• Requires problem
recognition and formulation• Ill-defined• Requires information
seeking• Multiple acceptable
solutions• Multiple paths to solution• Embedded in ordinary
experience• Requires motivation and
personal involvement
Things Students Do Before Taking a Test
• Determine study objectives and methods
• Manage goal conflicts
• Coordinate work with other classmates
• Seek additional information or materials
• Handle school related finances
• Negotiate with peers and faculty
• Avoid counterproductive school behaviors
• Structure effective communications
A Model of Undergraduate Student Performance Dimensions• Traditional Classroom
Success• Written and Oral
Communication• Personal Discipline• Resolving Goal Conflicts• Studying and Learning
Proficiency• Sustained Goal Directed
Effort
• Interactive Learning and Team Performance
• Administration• Interpersonal Proficiency• Non-Classroom
Performance• Development of Life Goals
and Values
Kuncel (2002); Kuncel, Campbell, Hezlett, & Ones (2001)
Cross-Situational Validity: A Direct Test
• However, we still lack a direct test: – Examine the validity of a single test developed
for academic settings but used in both academic and work settings
– Ideally we would also establish the relationship between this test and other cognitive ability measures
• Unfortunately, a single ability measure is rarely used for both personnel selection and educational admissions decisions
• With one notable exception….
MAT – Academic Performance
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Time to Finish
Degree Attainment
ResearchProductivity
ComprehensiveExams
Faculty Ratings
1st Year Grad. GPA
Grad. GPA
Kuncel, Hezlett, & Ones (2004) JPSP
MAT – Transitional Variables and Creativity
0 0.1 0.2 0.3 0.4 0.5 0.6
InternshipPracticum Ratings
Student TeachingRatings
Conseling WorkSamples
Creativity Ratings
Career PotentialRatings
Conseling PotentialRatings
Kuncel, Hezlett, & Ones (2004) JPSP
MAT – Work Criteria
0 0.1 0.2 0.3 0.4 0.5 0.6
JobPerformance
CounselingPerformance
Ed. Admin.Performance
Member Prof.Org.
Kuncel, Hezlett, & Ones (2004) JPSP
Story: Predictive Power of Cognitive Ability
Tests is an artifact of SES
Its Storytime…
• In the interest of truth in advertising, the SAT should simply be called a “wealth test” - Guiner (Undated)
• “The SAT merely measures the size of student’s houses” - Kohn (2001)
• “Only thing the SAT predicts well now is socioeconomic status” – Colvin (1997)
Recent Claims in the American Psychologist
• “…SAT I scores lose any ability to predict freshman year grades if the regression analyses control for socioeconomic status” (p. 100) – Crosby, Iyer, Clayton, and Downing (2003)
• “…SAT scores used for college admissions do not predict freshman year grades when socioeconomic status is controlled” (p. 1023) – Biernat (2003)
• Atkinson and Geiser (2009) claim that SAT coefficients are “decisively diminished” when SES and HSGPA are controlled.
Consistent SES FindingsSource Sample rtest-grade rtest-grade.SES
U California 77,000 .37 .34
Sackett, Kuncel, et al. 2009a 155,000 .35 .33
Sackett, Kuncel, et al. 2009b 17,000 .37 .36
Moderate observed
relationship
Does not go to zero when
controlling for SES
University of California System Data
HSGPA .30
SAT I .22
Family Income .03
Parents Education .05
“Ok, but I know someone who had really high test scores and they didn’t do as well as another friend who had lower test scores. So scores only matters to a certain degree, right?”
More is also better at school
• Arneson and Sackett (2009)
1.5
2
2.5
3
3.5
4
600 700 800 900 1000 1100 1200 1300 1400 1500 1600
SAT Score
Ad
just
ed F
resh
man
GP
A
Coward & Sackett (1990)• 174 studies on the relationship between
intelligence and job performance• Studies used the 9 scale GATB (General
Aptitude Test Battery)• Total sample size across all studies was a
substantial 36,614• Found overwhelming support for a linear
relationship between ability and job performance
• The commonly held notion appears to be incorrect
Traditional Supplementary Predictors
• Letters of Recommendation
• Personal Statements
• Interviews
00.10.20.30.40.5
GPA
Gra
d.Sc
hool
Facu
ltyR
atin
g
Deg
ree
Com
plet
ion
Res
earc
hPr
oduc
tivity
00.10.20.30.40.5
GPA
Grad
.Sc
hool
Facu
ltyRa
ting
Degr
eeCo
mpl
etio
n
Rese
arch
Prod
uctiv
ity
Letters GRE
Result suggest no incremental validity for letters, save for degree completion. Note that all letters were structured and quantitative.
Kuncel, Vanelli, & Ones (2009)
Cooper-Murphy, Klieger, Borneman, & Kuncel (2007) College and University
Personal Statements
Admissions Interview: Results
N k 80% cred.
3540 37 0.12 0.11 0.04 .06 to .17
2641 23 0.15 0.1 0.04 .10 to .21
262 5 0.24 0.25 0.21 -.03 to .51
231 3 0.13 0.07 0 .13 to .13
robs SDobs SDr
Grade Point Average in Graduate/Medical School
Medical Clinical and Internship Performance Rating
Graduate School Performance Rating (non Medical)
Medical Board Examinations
Yet There is Good Stuff Out There
Study Habits, Skills, and Attitudes
Crede & Kuncel (2007) Perspectives in Psychological Science
N k r
Aggregate Measures 18,517 107 .33
Study Skills 24,547 87 .25
Study Habits 23,390 102 .23
Study Attitudes 7,211 37 .26
0
0.1
0.2
0.3
0.4
0.5
SHSA SAT HSGPA
Class Attendance: Being There
Crede, Roch, & Kieszczynka (2010) Review of Educational Research
N k r
Attendance 21,164 68 .41
Personality Predictors of Graduate Student Performance
Rigdon & Kuncel, 2010
N k robs SDobs SDrGraduate GPA
Emotional Stability 429 6 .24 .12 .03
Achievement 1,645 7 .16 .08 .04
Task Performance
Autonomy 518 3 .16 .02 .00
Ach. Via Indep. 132 3 .27 .12 .00
Qualifying Exam
Ach. Via Indep. 156 2 .26 .00 .00
Obtain Ph.D.
Emotional Stability 1,180 3 .09 .10 .08
Conscientiousness 1,559 5 .08 .09 .07
Cautionary Tales
• Faking and Coaching– Most alternatives that have been proposed would
be highly susceptible to deliberate faking or test preparation coaching
• Personality Assessments– Other ratings
• Study habits, attitude, and skill inventories– Placement and guidance
• Biodata– Verifiable content
• Situational judgments tests
Fairness and Bias
• Extensive research on standardized test scores for bias
• Social class has also been scrutinized.
• Data for letters of recommendation and personal statements in academic settings is thinner.
• A summary of current knowledge on the next slide…
New Alternatives Should be Subjugated to Extensive Study for Bias Before Implementation
Top Related