The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity...
Transcript of The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity...
TC 1
The Chimera of Validity
Bringing the Validity Conversations Home: When
Educational Measures Go Public
Eva L. Baker CRESST National Center for Research on Evaluation, Standards, & Student Testing UCLA
Educational Assessment, Accountability, and Equity: Conversations on Validity Around the World
Teachers College, Columbia University 29 March 2012, New York
© Regents of the University of California
TC 2
The Chimera of Validity
© Regents of the University of California
TC 3 © Regents of the University of California
TC 4
Quality Validity Zen and the Art of Motorcycle Maintenance
R.M. Pirsig, 1974
“Real Quality must be the source of the subjects and objects, the cone from which the tree must start.”
© Regents of the University of California
TC 5
Common Language Validity
1. Well grounded, sound, supportable
2. Producing the desired results; efficacious
3. Legally sound and effective; incontestable; binding
4. a. Containing premises from which the conclusion may be logically derived
b. Correctly inferred or deduced from a premise as a conclusion
© Regents of the University of California
TC 6
Validate!
• To declare or make legally valid;
• To mark (like a transfer or passport);
• To substantiate and verify (decisive, conclusive, capacity to resist attack, capable of ending a debate)
American Heritage Dictionary of the
English Language, 1981, p. 1414)
© Regents of the University of California
TC 7
Conflicting Interpretations
Public & Policy Makers
Testing Professionals
Users
© Regents of the University of California
TC 8 © Regents of the University of California
TC 9
Reasons for Early Scholarly Studies
• Thorndike
• Exploration of human capacities
• Uses for selection
• Normal distributions as foundation of design and interpretation
• Largely assumed stability (fundamentally not subject to change by schools or programs)
“A test is valid for anything it correlates with” Guilford, 1946
© Regents of the University of California
TC 10
Faces of Key Testing Innovators • Ralph Tyler • Art Lumsdaine • B.F. Skinner • E.F. Lindquist • Ben Bloom • Chester Harris • Lee Cronbach • Sam Messick • Bob Glaser • Ed Gordon
© Regents of the University of California
TC 11
Standards for Educational and Psychological Testing
© Regents of the University of California
TC 12
“Standards”
• Mid 20th century
• 1999 APA-AERA-NCME sponsored
• Synthesis of accepted state of knowledge and best practice
• Not “aspirational”
• Serve all areas in education and psychology using testing—neurology, clinical, etc.
© Regents of the University of California
TC 13
“Standards” Cont’d
• Revised every decade or so
• Documentary
• Guidance–not enforceable by organizations
• Power through legal precedent, prophylactic
© Regents of the University of California
TC 14
Emerging Notions of Validity
• Unitary concept • Not types of validity, like criterion, predictive
• Purpose-driven
• Validity argument–theory & evidence Vs. claims
• Assessments designed to measure learning rather than traits
© Regents of the University of California
TC 15
Key Elements of the 1999 Standards
• Enormous effort to involve the field
• Validity as unitary concept, argued with a range of theory, designs and data
• Purpose and claims
• Guidance for assessment designers involving cognitive demands and other test attributes
© Regents of the University of California
TC 16
1999 Standards
• More standards • Longer chapters to clarify concepts • Fairness chapter and standards • Weak technology chapter • Thin policy chapter • No rating scheme for Standards, choices
depended on professional judgment • Not universally acclaimed
© Regents of the University of California
TC 17
“Going Public” What Happened?
• Given my hopes, not much
• Swamped by NCLB requirements
• Ambitious schedules
for developers
• Restricted testing time
• BOTA
• Cost, cost, cost
© Regents of the University of California
TC 18
Making a Validity Argument • Consequences
• Meaningfulness
• Fairness
• Cognitive complexity
• Content quality
• Transfer and generalizability
• Instructional sensitivity
• Cost and efficiency
© Regents of the University of California
Linn, Baker, & Dunbar, 1991
TC 19
Key Practical Barriers
• Clarity of standards
• Technical quality of complex tasks
• Evidence of transfer and generalized learning
• Time to conduct studies before use
• Scalable evidence of classroom or other learning experiences
• Confounded by accountability
© Regents of the University of California
TC 20
Challenge of Multiple Purposes
• Assessments for certification, progress and improvement
• Assessments of certification and admissions
• Assessments of children used to determine adult performance (teacher evaluation)
• Research use as dependent measures
© Regents of the University of California
TC 21
Looking to New CCSSs
• Carefully designed–many standards
• Consortia specifications of useful assessments
• Graphics to clarify meaning and relationships to avoid narrowness and trivial alignment • Maps, blueprints, ontologies
• Strategies for data management and validity studies
© Regents of the University of California
TC 22
Ontology Design
Specifications
Relevant texts
Best Practices
Expert judgment
Principle
Procedure
Fundamental principles
Includes components
Concept
requires requires
definition
related-to
DO
MA
INS
© Regents of the University of California
TC 23
My Picks • Unitary validity argument
• Process measures of learning
• Evidence that assessment scores will rise because of high quality instruction/learning
• Translational graphics
• Interrupting predictions
• Choice by students, as in badges and qualifications
© Regents of the University of California
TC 24
Going Public in the Future
• Short term questions • Who is taking the test?
• Hacking
• Favorite solution: Zacharias–Give up security • Publish everything in advance, menu to guide
choice
© Regents of the University of California
TC 25
Technology Now and Soon • Technology affects behaviors, brain function,
preferences
• Personalized, point-to-point communication
• Crowdsourcing Vs. expertise
• Desire to be noticed–metric: millions over weeks
• Constrained choices
• Weakened formal institutions
© Regents of the University of California
TC 26
Everyone Can Know Everything About You
• Data capture from multiple sources (sensors, cameras, phone)
• Integrated profiles of individuals
• Achievement, tacit preferences or psychological states
• Reports
• Privacy, personal choice, new institutions
• Generational problem
© Regents of the University of California
TC 27
To Educate for the Unpredictable
• Focus on challenging skill learning in content: Adaptive problem solving, situation awareness, risk, decisions, metacognitive skills, social and collaborative skills
• Transfer and generalization tasks–learning to learn
• Emphasize choice
© Regents of the University of California
TC 28
Validity? • Pertaining to skills
• Managed by different institutions, corporations (who are just like people)
• Quality and mobility issues
• Competition
• Adaptability rather than prediction
© Regents of the University of California
TC 29
Why the Chimera?
• Imagine options
• Imagine validity as a hybrid
• Chimeras are scary
© Regents of the University of California
TC 30
http://www.cse.ucla.edu
Eva L. Baker
[email protected] 310-206-1530
© Regents of the University of California
TC 31
Credits for Figures • FACE:
http://www.google.com/imgres?q=face+child+black&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=CXdt2WKPZy5gDM:&imgrefurl=http://newsone.com/the-education-zone/associatedpress3/education-secretary-says-minority-students-face-harsher-discipline/&docid=urHjqFA9_HP0VM&imgurl=http://newsone.com/files/2012/03/black-kids-face-harsher-treatment-in-school-discipline-.jpg&w=640&h=380&ei=rWtqT43vAoSTiAKN6NjIBQ&zoom=1&iact=hc&vpx=878&vpy=496&dur=189&hovh=173&hovw=291&tx=187&ty=109&sig=115080220995703348838&page=2&tbnh=144&tbnw=209&start=36&ndsp=47&ved=1t:429,r:4,s:36
• http://www.google.com/imgres?q=face&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsa&tbnid=W3X2sZNZGUwm0M:&imgrefurl=http://freeestuffs.com/megan-fox/megan-fox-face-758/&docid=BJzuVhbicsGP6M&imgurl=http://freeestuffs.com/wp-content/uploads/2011/12/Megan-Fox-face-758.jpg&w=1600&h=1200&ei=ZmhqT5LsJYfYiQKQlPD4BA&zoom=1
• http://www.google.com/imgres?q=face+child+asian+american&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsb&tbnid=W2zGjs11BeRUcM:&imgrefurl=http://blog.world-mysteries.com/science/how-many-major-races-are-there-in-the-world/&docid=WiG9D5oVBXB5SM&imgurl=http://blog.world-mysteries.com/wp-content/uploads/2011/02/asian-child.jpg&w=425&h=282&ei=bG1qT_CBKsHciAKk2MXSBQ&zoom=1
• http://www.google.com/imgres?q=face+latina+girl&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=emyPNGe6zfReFM:&imgrefurl=http://stockshop.wordpress.com/&docid=czgeS27bjgGMqM&imgurl=http://stockshop.files.wordpress.com/2010/03/hispanic-girl-smile.gif&w=433&h=288&ei=5G9qT9zBA6mqiQLz3vS7BQ&zoom=1&iact=hc&vpx=1279&vpy=645&dur=1833&hovh=183&hovw=275&tx=201&ty=130&sig=115080220995703348838&page=1&tbnh=151&tbnw=207&start=0&ndsp=43&ved=1t:429,r:15,s:0
© Regents of the University of California
TC 32
Credits for Pictures • Glaser: http://www.fabbs.org/index.php?cID=351
• Cronbach: http://oaks.nvg.org/lee-cronbach.html
• Gordon: http://www.google.com/imgres?q=edmund+gordon&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnso&tbnid=qPa0aELm65205M:&imgrefurl=http://www.tc.columbia.edu/news.htm%3FarticleID%3D7866&docid=QIHvISuu5URgSM&imgurl=http://www.tc.columbia.edu/i/media/7866_EdmundGordon.jpg&w=250&h=364&ei=qHVqT8GjCeeWiAL-0IiBBQ&zoom=1&iact=hc&vpx=185&vpy=121&dur=6186&hovh=271&hovw=186&tx=79&ty=154&sig=115080220995703348838&page=1&tbnh=154&tbnw=116&start=0&ndsp=39&ved=1t:429,r:0,s:0
• Bloom: http://www.google.com/imgres?q=benjamin+bloom&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbol&tbnid=4KSwaC6jMQtOZM:&imgrefurl=http://www.smartkpis.com/performance_management_resources/Education_and_Learning/Bloom-Benjamin-1913-1999-&docid=R1czzBRamLn5eM&imgurl=http://www.smartkpis.com/photos/learning/Bloom-Benjamin-smartkpis-photo-12.com.jpg&w=200&h=300&ei=JHRqT-akLcWjiALChfGCBQ&zoom=1&iact=hc&vpx=522&vpy=622&dur=361&hovh=240&hovw=160&tx=59&ty=150&sig=115080220995703348838&page=1&tbnh=164&tbnw=109&start=0&ndsp=37&ved=1t:429,r:9,s:0
• Tyler: http://www.google.com/imgres?q=ralph+tyler&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbo&tbnid=u8pZGDhDp4h8GM:&imgrefurl=http://www.ecs.org/html/aboutECS/AwardsWinners.htm&docid=R7w7sBgPzz-qPM&imgurl=http://www.ecs.org/html/aboutECS/Photos/RalphTyler.jpg&w=115&h=156&ei=i3RqT-GgF8GfiALd_ozQBQ&zoom=1&iact=hc&vpx=987&vpy=228&dur=5253&hovh=124&hovw=92&tx=64&ty=137&sig=115080220995703348838&page=1&tbnh=124&tbnw=92&start=0&ndsp=45&ved=1t:429,r:6,s:0
© Regents of the University of California
TC 33
Credits for Figures • http://gpaper139.112.2O7.net/b/ss/gpaper139,gntbcstglobal/1/H.3--NS/0" height="1" width="1"
border="0" alt="" />
• http://www.education.com/reference/article/thorndike-edward-lee-
• Persig: http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg
• Test standards: teststandards.org
• Early test standards Amazon
• http://c2.ac-images.myspacecdn.com/images02/129/l_0157a4bec6d648a482fafb71d28befa9.jpg
• http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg
• http://www.bates-communications.com
• http://www.frontdoor.com
• http://www.bates-communications.com
• http://library.thinkquest.org/04oct/00492/Quiz_pencil.jpg
• http://crooksandliars.com/files/vfs/2010/11/gavel.jpg
• http://zeusiswatching.files.wordpress.com/2010/10/100_2816.jpg?w=510
© Regents of the University of California