The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity...
Transcript of The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity...
![Page 1: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/1.jpg)
TC 1
The Chimera of Validity
Bringing the Validity Conversations Home: When
Educational Measures Go Public
Eva L. Baker CRESST National Center for Research on Evaluation, Standards, & Student Testing UCLA
Educational Assessment, Accountability, and Equity: Conversations on Validity Around the World
Teachers College, Columbia University 29 March 2012, New York
© Regents of the University of California
![Page 2: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/2.jpg)
TC 2
The Chimera of Validity
© Regents of the University of California
![Page 3: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/3.jpg)
TC 3 © Regents of the University of California
![Page 4: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/4.jpg)
TC 4
Quality Validity Zen and the Art of Motorcycle Maintenance
R.M. Pirsig, 1974
“Real Quality must be the source of the subjects and objects, the cone from which the tree must start.”
© Regents of the University of California
![Page 5: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/5.jpg)
TC 5
Common Language Validity
1. Well grounded, sound, supportable
2. Producing the desired results; efficacious
3. Legally sound and effective; incontestable; binding
4. a. Containing premises from which the conclusion may be logically derived
b. Correctly inferred or deduced from a premise as a conclusion
© Regents of the University of California
![Page 6: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/6.jpg)
TC 6
Validate!
• To declare or make legally valid;
• To mark (like a transfer or passport);
• To substantiate and verify (decisive, conclusive, capacity to resist attack, capable of ending a debate)
American Heritage Dictionary of the
English Language, 1981, p. 1414)
© Regents of the University of California
![Page 7: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/7.jpg)
TC 7
Conflicting Interpretations
Public & Policy Makers
Testing Professionals
Users
© Regents of the University of California
![Page 8: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/8.jpg)
TC 8 © Regents of the University of California
![Page 9: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/9.jpg)
TC 9
Reasons for Early Scholarly Studies
• Thorndike
• Exploration of human capacities
• Uses for selection
• Normal distributions as foundation of design and interpretation
• Largely assumed stability (fundamentally not subject to change by schools or programs)
“A test is valid for anything it correlates with” Guilford, 1946
© Regents of the University of California
![Page 10: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/10.jpg)
TC 10
Faces of Key Testing Innovators • Ralph Tyler • Art Lumsdaine • B.F. Skinner • E.F. Lindquist • Ben Bloom • Chester Harris • Lee Cronbach • Sam Messick • Bob Glaser • Ed Gordon
© Regents of the University of California
![Page 11: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/11.jpg)
TC 11
Standards for Educational and Psychological Testing
© Regents of the University of California
![Page 12: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/12.jpg)
TC 12
“Standards”
• Mid 20th century
• 1999 APA-AERA-NCME sponsored
• Synthesis of accepted state of knowledge and best practice
• Not “aspirational”
• Serve all areas in education and psychology using testing—neurology, clinical, etc.
© Regents of the University of California
![Page 13: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/13.jpg)
TC 13
“Standards” Cont’d
• Revised every decade or so
• Documentary
• Guidance–not enforceable by organizations
• Power through legal precedent, prophylactic
© Regents of the University of California
![Page 14: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/14.jpg)
TC 14
Emerging Notions of Validity
• Unitary concept • Not types of validity, like criterion, predictive
• Purpose-driven
• Validity argument–theory & evidence Vs. claims
• Assessments designed to measure learning rather than traits
© Regents of the University of California
![Page 15: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/15.jpg)
TC 15
Key Elements of the 1999 Standards
• Enormous effort to involve the field
• Validity as unitary concept, argued with a range of theory, designs and data
• Purpose and claims
• Guidance for assessment designers involving cognitive demands and other test attributes
© Regents of the University of California
![Page 16: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/16.jpg)
TC 16
1999 Standards
• More standards • Longer chapters to clarify concepts • Fairness chapter and standards • Weak technology chapter • Thin policy chapter • No rating scheme for Standards, choices
depended on professional judgment • Not universally acclaimed
© Regents of the University of California
![Page 17: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/17.jpg)
TC 17
“Going Public” What Happened?
• Given my hopes, not much
• Swamped by NCLB requirements
• Ambitious schedules
for developers
• Restricted testing time
• BOTA
• Cost, cost, cost
© Regents of the University of California
![Page 18: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/18.jpg)
TC 18
Making a Validity Argument • Consequences
• Meaningfulness
• Fairness
• Cognitive complexity
• Content quality
• Transfer and generalizability
• Instructional sensitivity
• Cost and efficiency
© Regents of the University of California
Linn, Baker, & Dunbar, 1991
![Page 19: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/19.jpg)
TC 19
Key Practical Barriers
• Clarity of standards
• Technical quality of complex tasks
• Evidence of transfer and generalized learning
• Time to conduct studies before use
• Scalable evidence of classroom or other learning experiences
• Confounded by accountability
© Regents of the University of California
![Page 20: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/20.jpg)
TC 20
Challenge of Multiple Purposes
• Assessments for certification, progress and improvement
• Assessments of certification and admissions
• Assessments of children used to determine adult performance (teacher evaluation)
• Research use as dependent measures
© Regents of the University of California
![Page 21: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/21.jpg)
TC 21
Looking to New CCSSs
• Carefully designed–many standards
• Consortia specifications of useful assessments
• Graphics to clarify meaning and relationships to avoid narrowness and trivial alignment • Maps, blueprints, ontologies
• Strategies for data management and validity studies
© Regents of the University of California
![Page 22: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/22.jpg)
TC 22
Ontology Design
Specifications
Relevant texts
Best Practices
Expert judgment
Principle
Procedure
Fundamental principles
Includes components
Concept
requires requires
definition
related-to
DO
MA
INS
© Regents of the University of California
![Page 23: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/23.jpg)
TC 23
My Picks • Unitary validity argument
• Process measures of learning
• Evidence that assessment scores will rise because of high quality instruction/learning
• Translational graphics
• Interrupting predictions
• Choice by students, as in badges and qualifications
© Regents of the University of California
![Page 24: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/24.jpg)
TC 24
Going Public in the Future
• Short term questions • Who is taking the test?
• Hacking
• Favorite solution: Zacharias–Give up security • Publish everything in advance, menu to guide
choice
© Regents of the University of California
![Page 25: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/25.jpg)
TC 25
Technology Now and Soon • Technology affects behaviors, brain function,
preferences
• Personalized, point-to-point communication
• Crowdsourcing Vs. expertise
• Desire to be noticed–metric: millions over weeks
• Constrained choices
• Weakened formal institutions
© Regents of the University of California
![Page 26: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/26.jpg)
TC 26
Everyone Can Know Everything About You
• Data capture from multiple sources (sensors, cameras, phone)
• Integrated profiles of individuals
• Achievement, tacit preferences or psychological states
• Reports
• Privacy, personal choice, new institutions
• Generational problem
© Regents of the University of California
![Page 27: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/27.jpg)
TC 27
To Educate for the Unpredictable
• Focus on challenging skill learning in content: Adaptive problem solving, situation awareness, risk, decisions, metacognitive skills, social and collaborative skills
• Transfer and generalization tasks–learning to learn
• Emphasize choice
© Regents of the University of California
![Page 28: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/28.jpg)
TC 28
Validity? • Pertaining to skills
• Managed by different institutions, corporations (who are just like people)
• Quality and mobility issues
• Competition
• Adaptability rather than prediction
© Regents of the University of California
![Page 29: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/29.jpg)
TC 29
Why the Chimera?
• Imagine options
• Imagine validity as a hybrid
• Chimeras are scary
© Regents of the University of California
![Page 30: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/30.jpg)
TC 30
http://www.cse.ucla.edu
Eva L. Baker
[email protected] 310-206-1530
© Regents of the University of California
![Page 31: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/31.jpg)
TC 31
Credits for Figures • FACE:
http://www.google.com/imgres?q=face+child+black&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=CXdt2WKPZy5gDM:&imgrefurl=http://newsone.com/the-education-zone/associatedpress3/education-secretary-says-minority-students-face-harsher-discipline/&docid=urHjqFA9_HP0VM&imgurl=http://newsone.com/files/2012/03/black-kids-face-harsher-treatment-in-school-discipline-.jpg&w=640&h=380&ei=rWtqT43vAoSTiAKN6NjIBQ&zoom=1&iact=hc&vpx=878&vpy=496&dur=189&hovh=173&hovw=291&tx=187&ty=109&sig=115080220995703348838&page=2&tbnh=144&tbnw=209&start=36&ndsp=47&ved=1t:429,r:4,s:36
• http://www.google.com/imgres?q=face&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsa&tbnid=W3X2sZNZGUwm0M:&imgrefurl=http://freeestuffs.com/megan-fox/megan-fox-face-758/&docid=BJzuVhbicsGP6M&imgurl=http://freeestuffs.com/wp-content/uploads/2011/12/Megan-Fox-face-758.jpg&w=1600&h=1200&ei=ZmhqT5LsJYfYiQKQlPD4BA&zoom=1
• http://www.google.com/imgres?q=face+child+asian+american&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsb&tbnid=W2zGjs11BeRUcM:&imgrefurl=http://blog.world-mysteries.com/science/how-many-major-races-are-there-in-the-world/&docid=WiG9D5oVBXB5SM&imgurl=http://blog.world-mysteries.com/wp-content/uploads/2011/02/asian-child.jpg&w=425&h=282&ei=bG1qT_CBKsHciAKk2MXSBQ&zoom=1
• http://www.google.com/imgres?q=face+latina+girl&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=emyPNGe6zfReFM:&imgrefurl=http://stockshop.wordpress.com/&docid=czgeS27bjgGMqM&imgurl=http://stockshop.files.wordpress.com/2010/03/hispanic-girl-smile.gif&w=433&h=288&ei=5G9qT9zBA6mqiQLz3vS7BQ&zoom=1&iact=hc&vpx=1279&vpy=645&dur=1833&hovh=183&hovw=275&tx=201&ty=130&sig=115080220995703348838&page=1&tbnh=151&tbnw=207&start=0&ndsp=43&ved=1t:429,r:15,s:0
© Regents of the University of California
![Page 32: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/32.jpg)
TC 32
Credits for Pictures • Glaser: http://www.fabbs.org/index.php?cID=351
• Cronbach: http://oaks.nvg.org/lee-cronbach.html
• Gordon: http://www.google.com/imgres?q=edmund+gordon&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnso&tbnid=qPa0aELm65205M:&imgrefurl=http://www.tc.columbia.edu/news.htm%3FarticleID%3D7866&docid=QIHvISuu5URgSM&imgurl=http://www.tc.columbia.edu/i/media/7866_EdmundGordon.jpg&w=250&h=364&ei=qHVqT8GjCeeWiAL-0IiBBQ&zoom=1&iact=hc&vpx=185&vpy=121&dur=6186&hovh=271&hovw=186&tx=79&ty=154&sig=115080220995703348838&page=1&tbnh=154&tbnw=116&start=0&ndsp=39&ved=1t:429,r:0,s:0
• Bloom: http://www.google.com/imgres?q=benjamin+bloom&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbol&tbnid=4KSwaC6jMQtOZM:&imgrefurl=http://www.smartkpis.com/performance_management_resources/Education_and_Learning/Bloom-Benjamin-1913-1999-&docid=R1czzBRamLn5eM&imgurl=http://www.smartkpis.com/photos/learning/Bloom-Benjamin-smartkpis-photo-12.com.jpg&w=200&h=300&ei=JHRqT-akLcWjiALChfGCBQ&zoom=1&iact=hc&vpx=522&vpy=622&dur=361&hovh=240&hovw=160&tx=59&ty=150&sig=115080220995703348838&page=1&tbnh=164&tbnw=109&start=0&ndsp=37&ved=1t:429,r:9,s:0
• Tyler: http://www.google.com/imgres?q=ralph+tyler&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbo&tbnid=u8pZGDhDp4h8GM:&imgrefurl=http://www.ecs.org/html/aboutECS/AwardsWinners.htm&docid=R7w7sBgPzz-qPM&imgurl=http://www.ecs.org/html/aboutECS/Photos/RalphTyler.jpg&w=115&h=156&ei=i3RqT-GgF8GfiALd_ozQBQ&zoom=1&iact=hc&vpx=987&vpy=228&dur=5253&hovh=124&hovw=92&tx=64&ty=137&sig=115080220995703348838&page=1&tbnh=124&tbnw=92&start=0&ndsp=45&ved=1t:429,r:6,s:0
© Regents of the University of California
![Page 33: The Chimera of Validity - Educational Testing Service Chimera of Validity Bringing the Validity Conversations Home: When Educational Measures Go Public Eva L. Baker CRESST National](https://reader031.fdocuments.us/reader031/viewer/2022030519/5ac50d247f8b9a5c558d26fe/html5/thumbnails/33.jpg)
TC 33
Credits for Figures • http://gpaper139.112.2O7.net/b/ss/gpaper139,gntbcstglobal/1/H.3--NS/0" height="1" width="1"
border="0" alt="" />
• http://www.education.com/reference/article/thorndike-edward-lee-
• Persig: http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg
• Test standards: teststandards.org
• Early test standards Amazon
• http://c2.ac-images.myspacecdn.com/images02/129/l_0157a4bec6d648a482fafb71d28befa9.jpg
• http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg
• http://www.bates-communications.com
• http://www.frontdoor.com
• http://www.bates-communications.com
• http://library.thinkquest.org/04oct/00492/Quiz_pencil.jpg
• http://crooksandliars.com/files/vfs/2010/11/gavel.jpg
• http://zeusiswatching.files.wordpress.com/2010/10/100_2816.jpg?w=510
© Regents of the University of California