Using tests to predict job performance CALSWEC - 2007
description
Transcript of Using tests to predict job performance CALSWEC - 2007
Using tests to predict job performanceCALSWEC - 2007
Cynthia Parry, PhDC.F. Parry Associates
Michelle Graef, PhDCenter on Children, Families & the Law –
University of Nebraska, Lincoln
Todd Franke, PhDDepartment of Social Work –
University of California at Los Angeles
Henry Ilian, PhDNew York City Administration for Children’s Services
James Satterwhite Academy for Child Welfare Training
Using tests to predict job performanceCALSWEC - 2007
• Goals:– Define high-stakes testing– Review technical complexities and professional
standards– Discuss the risks, rewards & consequences of
high-stakes testing
• Anticipated Outcome– A phone call to one or more of the presenters prior
any venture in high-stakes testing development
Using tests to predict job performanceCALSWEC - 2007
Presentation in 4 parts:1. Overview of high stakes testing
2. Validity considerations
3. Validity issues in high stakes testing and training of child welfare workers
4. The ethics of using high-stakes testing and the ethics of not using high stakes testing
Part 1 of 4:Overview of High Stakes Testing
Cindy Parry, PhD
What do we mean by high stakes?
• Any use of a test that might affect– Your ability to get or keep a job– Your chances for promotion or raises– Your supervisor’s evaluation of your work– Testing out of training requirements or needing further
coaching or other interventions• Such as
– Licensing exams– Civil service hiring tests– Certification/credentialing tests– Any testing where individual results are reported to a
supervisor/manager or used to predict future competent performance
What is the Law and Policy Context?
• 14th Amendment Requirements– Equal protection– Due process
• Civil Rights Acts– Disparate treatment of protected group– Disparate impact on a protected group– Business necessity
• ADA– Accommodations for test takers
Taken from Mehrens, W. Legal and Professional Bases for Licensure Testing. In Impara, James (Ed.). (1995) Licensure Testing: Purposes, Procedures and Practices. Buros Institute of Mental Measurements: University of Nebraska, Lincoln.
Considerations in Developing a High Stakes Test
• Use professional standards to guide test development – The Standards for Educational and Psychological
Testing (AERA, APA, NCME) – Uniform Guidelines on Employee Selection
Procedures (EEOC)– Principles for the Validation and Use of Personnel
Selection Procedures (SIOP)
Key Considerations in Validation
• Validity is the most fundamental consideration in developing any test
• We validate the use of a test score for a specific purpose — not a test or an item
• Content validity is the most common type of evidence cited in child welfare applications– Requires full specification of the domain or construct it is
intended to measure (1.6)– Standards require complete description of procedures used
to develop test content (1.6)• Use of subscales requires validation of the inferences
made from subscale scores as well as overall score (1.12)
Standards cited from: American Educational Research Association, American Psychological Association and National Council on Measurement in Education. (2002). Standards for Educational and Psychological Testing. American Educational Research Association: Washington, DC.
Other Considerations for High Stakes Testing
• Adequate reliability for overall scores and any sub-scores that will be used in decision making
• Investigations of differential item functioning and test fairness
• Comparability of alternate forms or tests given over time
• Documentation of the method of setting the cut score and the qualifications of judges
• Notice of testing and due process procedures• Policy and procedures for accommodating persons
with disabilities • Policy and procedures for test security and
identification of test takers
Part 2 of 4:Validity Considerations
Michelle Graef, PhD
Can we use our training evaluation knowledge test to make decisions about candidates’ ability to
perform the job?Three points to consider:• Used alone, a written knowledge test is unlikely to be
adequate to make inferences about the full range of KSAOs needed to perform child welfare work.
• The target of your inferences indicates the appropriate content domain to be sampled. The typical training evaluation knowledge test (“post-test”) items have been sampled from the curriculum domain, which may not accurately represent the job domain.
• Even if you have strong evidence of content validity, evidence derived from other validation strategies (criterion-related validity) is recommended to support an employment decision. The implied inference is prediction of future job performance.
Job Domain (KSAOs): Determined through a Job Analysis
Knowledge
Skills
Other Characteristics (values, beliefs, attitudes, personality)
Abilities
Training Curriculum Content [K,S,O]
Test Item Content [K]
Sampling for development of a training evaluation knowledge test designed to assess trainees’ “mastery of training content”
Test Item Content [K]
Training Curriculum Content [K,S,O]
Test Item Content [K]
Job Domain [K,S,A,O]
Example of a training evaluation knowledge test that is inappropriate for assessing individuals’ “ability to perform the job”
Training Curriculum Content [K,S,O]
Test Item Content [K,S,O]
Job Content [K,S,A,O]
IDEAL: Appropriate sampling for training evaluation test designed to assess individuals’ “ability to perform the job”
Predictor Measure (e.g., test)
Criterion Measure (e.g., supervisor rating, case review scores)
Criterion-related validity
Predictor Construct Domain (e.g., knowledge of child maltreatment, critical thinking skill, conscientiousness)
Criterion Construct Domain (e.g., job performance)
Content validityConstruct validity
Adapted from: Standards for Educational & Psychological Testing (1999)
Conceptual basis for test validation strategies
Criterion-related validation study
• Demonstrate relationship between predictor and criterion using statistical significance testing
• Feasibility depends upon:– Availability of appropriate criterion measures– Representativeness of research sample– Adequacy of statistical power
• Variety of designs:– Predictive– Concurrent– Use of incumbents or job applicants
• Development of predictor and criterion measures that are relevant, uncontaminated, not deficient, free from bias, and demonstrate reliability
• Refer to Principles for the Validation and Use of Personnel Selection Procedures (2003)
Recommendations
Strategies:
1) retrofit: map your existing knowledge test to the job domain and supplement it with other assessments (e.g., skills) to more adequately represent the job domain
2) start over: develop a test specifically for this purpose that directly samples the job domain, such as a work sample test, simulation, or assessment center
3) gather evidence of criterion-related validity in support of the use of your existing training evaluation knowledge test for employment decisions
Part 3 of 4:Validity issues in high stakes testing and training of child
welfare workers
Todd Franke, PhD
Validity issues in high stakes testing and training of child welfare workers
Are we measuring what is important?
Los Angeles Context
Evidence based practice
Evidence based training
High stakes in Los Angeles
Minimally competent CPS workersWhat would one look like?
Translation validity -- Focuses on whether the operationalization (i.e., measure) is a good translation of the construct
Face Validity -- On its face, does the operationalization look like a good translation of the construct?
Content Validity -- Operationalization is checked against the relevant content domain for the construct
Criterion-related Validity
Predictive Validity -- Operationalization’s ability to predict something it should theoretically be able to predict (L3)
Concurrent Validity
Convergent Validity
Discriminant Validity
Construct Validity
So what constitutes the domains relevant in training child welfare workers?What should be trained/assessed?
It depends who you ask!
Knowledge of
Policies ( over 384 policies in LA-DCFS)
Procedures
submitting mileage
cellular phone reimbursement
use of logs (court calendars, visitation)
new case to-do’s
cubicle organization and systems
Examples of current topics
Appreciating multiculturalism
Family preservation/Alternative response
Adoption assessments
Home assessments
Team Decision Making
Going to court/testifying
Court report writing
SDM tools
Basic interviewing
Adult substance abuse
Child neglect
Concurrent planning
Legal foundations
Risk assessment
Sexual abuse
Legal foundations
Legal permanence
Social work values
Worker safety
Physical abuse
What else might be important?Suggestions from supervisors/trainees
Readiness to learn
Ability to multi-task
Willingness to accept supervision
Ability to make hard decisions
Ability to transfer knowledge to field
Social support
Problem solving
Are these training issues or hiring criteria?
Writing
Self-reliance
How to deal with emotional aspect of job
Critical thinking
Issues related to burn-out
Anything that links directly to case work practice
What’s next?
How should an assessment be used?
Is knowledge only one of the domains that define the construct?
Does it have any predictive validity?Is it a valid measure of future job performance?
Part 4 of 4:The Ethics of Using High-Stakes
Testing and the Ethics of Not Using High Stakes Testing
Henry Ilian, PhD
High-stakes Tests Accomplish Three Things
• Screen out trainees who cannot demonstrate a specified level of mastery
• Compel Studying
• Enforce Fidelity to the Curriculum
Compel Studying N Items Pretest Posttest
CW Common Core 74 80 39.70% 86.10%
(High Stakes)
Core Essentials for 64 32 44.30% 63.40%
Experienced CWs (Not High Stakes)
Enforce Fidelity to the CurriculumCore Essentials for Experienced CWs
Posttest 1 64 15 72.80%
Posttest 2 64 17 55.10%
A Comparison from NYC
Using and Not Using High-Stakes Tests Each Has Consequences
• Using High-Stakes Tests– The requirements for a professionally developed
testing program require significant agency resources
– Some people with the potential to be good child protective workers may not pass
• Not Using High-Stakes Tests– Risk to families, children, self and colleagues– Difficulty maintaining employee standards and
agency morale
The Major Ethical Issue is the Potential for Harm
• There is a need to balance potential harm – to caseworker trainees – to children and families served by CPS
agencies – to co-workers and the agencies
themselves
Concerns for the Evaluator
• The organization may be reluctant/unable to make available the time and resources to develop and validate instruments
• Testing decisions are often made in a political or administrative context
• Administratively imposed testing may not meet professional standards