Wallace Judd APA Testing Standards : RFC
description
Transcript of Wallace Judd APA Testing Standards : RFC
![Page 2: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/2.jpg)
2
The APA Testing Standards: Request for Comment
Wallace Judd, Ph.D.
President of Authentic Testing Corporation
President-Elect of the PTC
Webinar Sponsored by the PTC and byAuthentic Testing Corporation.
![Page 3: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/3.jpg)
3
The APA Testing Standards: Request for Comment
Comments due October 15, 2007
http://www.TestStandards.org
![Page 4: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/4.jpg)
4
TestStandards.org
![Page 5: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/5.jpg)
5
Webinar is organized in 2 parts:
Part I. 20-minute intro to the Standards:
• How they were formulated.
• Why they’re important.
• How they’re likely to impact your program.
• Problems with the current Standards.
![Page 6: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/6.jpg)
6
Webinar is organized in 2 parts:
Part II. Forty minutes of discussion…
• Are these issues relevant to you?
• What (if anything) do you want to do?
• What are Next Steps?
![Page 7: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/7.jpg)
7
NCME Session Title
![Page 8: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/8.jpg)
8
What The Standards Address:
Importance of the Standards –
• 30,000 copies sold.
• Represents testing practice worldwide.
• Basis for ANSI ISO 17024 enforcement
• Basis for NCCA Accreditation
![Page 9: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/9.jpg)
9
Why Standards Are Important
• ANSI – required by DOD for security certifications
• NCCA – required by Maine for stateaccreditations
• More coming…
![Page 10: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/10.jpg)
10
Who Created the Standards?
75 Organizations provided input, volunteer time, editing, ideas. They were from:
• Original Sponsors: AERA, APA, NCME
• Academic Institutions
• Non-Profit Associations
• State & Federal Government Agencies
• Test Publishers
![Page 11: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/11.jpg)
11
Not a single sponsor or reviewer was from corporate America.
![Page 12: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/12.jpg)
12
Current Standards
• Reviewed, revised 1996-1998.
• Adopted 1998.
• Published, disseminated 1999.
![Page 13: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/13.jpg)
13
Contents of the Standards
• The Standards were written
• for the professional and
• for the educated layperson
![Page 14: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/14.jpg)
14
The Standards
address professional and technical issues of test development and use in
• education,
• psychology and
• employment. (APA.org, home page)
![Page 15: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/15.jpg)
15
The Standards
are the Bible of Testing in the U.S.
![Page 16: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/16.jpg)
16
Standards = The Bible
![Page 17: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/17.jpg)
17
Contents of the StandardsThis book is a vitally important reference for • professional test developers, • sponsors,sponsors, • publishers, • users, • policymakers, • employers,employers, and • students (http://www.APA.org, home page)
![Page 18: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/18.jpg)
18
But the Standards clearly target educational & psychological
testing.
![Page 19: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/19.jpg)
19
Few Job-Related Standards
• Educational – measuring and predicting student achievement
• Psychological – measuring and evaluating psychological traits
• Sections on Performance Testing and corporate / industrial testing – minimal.
![Page 20: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/20.jpg)
20
Initial Issues
![Page 21: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/21.jpg)
21
Problem: Bylaws
Standard 2. A. The certifying program must show that the governance structure, that the governance structure, policies, and procedures that have been policies, and procedures that have been established protect against undue established protect against undue influenceinfluence that could compromise the integrity of the certification process.
![Page 22: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/22.jpg)
22
Problem: Governance Structure
Standard 2. B. The governance structureThe governance structure, policies, and procedures must provide formust provide for autonomy in decision makingautonomy in decision making regarding important aspects of the certification program, such as eligibility standards; the development, administration, and scoring of the assessment instruments; selection of personnel; and operational processes.
![Page 23: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/23.jpg)
23
Problem: No Training & Testing In the Same Organization
• Standard 2. D. To avoid conflicts of interest between certification and education functions, the certification agency must the certification agency must not also be responsible for accreditation not also be responsible for accreditation of educational or training programsof educational or training programs or courses of study leading to the certification
![Page 24: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/24.jpg)
24
Problem: Board Composition
• Standard 3. The certification board or governing committee of the certification governing committee of the certification program must include individuals from program must include individuals from the certified populationthe certified population, as well as voting representation from at least one consumer or public member.
![Page 25: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/25.jpg)
25
Problem: No Place for Gating Items
Gating Items – Items Required to Pass Test
• Pilot’s Test – Land the plane
• Veterinarian’s Test – Spay a cat
• Crane Operator’s Test – Put ball in barrel
• Word Test – Save a file
• System Admin. Test – Add a new user
![Page 26: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/26.jpg)
26
Problem: No Performance Items
• Standard 3. (p. 39) When [using IRT], the fit of the model is evaluated … by fit of the model is evaluated … by evaluating unidimensionality,evaluating unidimensionality, local independence, speededness, and equality and equality of slope parametersof slope parameters.
![Page 27: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/27.jpg)
27
Problem: Expert Judges
• Standard 3.6 The qualifications, relevant experience, and demographic characteristics of demographic characteristics of expert judges should also be documentedexpert judges should also be documented.
![Page 28: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/28.jpg)
28
Problem: Content Domain
• Standard 3.11 Test developers should document the extent to which the content document the extent to which the content domain of a test represents the defined domain of a test represents the defined domaindomain and test specifications.
![Page 29: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/29.jpg)
29
Problem: Publish Scoring Criteria
Standard 3.20 When appropriate, …criteria criteria for scoring …should be provided to the for scoring …should be provided to the test takerstest takers prior to the administration of the test.
![Page 30: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/30.jpg)
30
Why Do I Care?
• I’ve developed tests since 1981 at Xerox, Oracle, Honeywell, McGraw Hill, Kelly Services, and my tests were the foundation for Microsoft’s MOUS tests.
• I believe that skills testing and certification deserves as much recognition and validation as psychological or educational testing.
![Page 31: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/31.jpg)
31
Why Does the PTC care?
• It has many members who have testing programs, and who want them to exemplify best practices in testing.
• PTC meets October 17-19 in Scottsdale.
• PTC will formulate its own response to the RFC.
• PTC is at PerformanceTest.org
![Page 32: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/32.jpg)
32
How important is compliance with the Standards to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
![Page 33: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/33.jpg)
33
How important is NCCA accreditation to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
![Page 34: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/34.jpg)
34
How important is ANSI accreditation to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
![Page 35: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/35.jpg)
35
We have two routes to go. What should we do?
1. Formulate separate standards for corporate-sponsored testing
2. Work with AERA, APA, NCME, to revise the Standards.
3. Both
![Page 36: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/36.jpg)
36
Which would you participate in? (You can vote for more than 1.)
1. Individual corporate response
2. Industry-specific group response
3. Response representing corporations with testing programs
![Page 37: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/37.jpg)
37
Plug: The PTC
• Where: Scottsdale, Arizona
• When: October 17 – 19, 2007
• Who: – Graham Brent – Crane Operators’ Exam– Ric Luecht – Engineering Assessments– Brian Rice – How Virtualization Works
http://www.PerformanceTest.org
![Page 38: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/38.jpg)
38
Plug: Negotiating the Standards
• Workshop October 16th, PTC Summit in Phoenix.
![Page 39: Wallace Judd APA Testing Standards : RFC](https://reader035.fdocuments.us/reader035/viewer/2022081501/568148c6550346895db5e2b5/html5/thumbnails/39.jpg)
39
Plug: AuthenticTesting.com
• Automated Job Task Analysis
• Automated Exam & Item Analysis
• Performance Test Design