(Even if You’re Not Calling it a “Test”) - CPS HR Consulting...2450 DEL PASO ROAD, SUITE 220...

2450 DEL PASO ROAD, SUITE 220

SACRAMENTO, CALIFORNIA 95834

(916) 263-3600

Hiring is Testing (Even if You’re Not Calling it a “Test”)

Contents

INTRODUCTION ........................................................................................................................................2

WHAT CONSTITUTES A TEST? .................................................................................................................3

COMMON TESTS AND RELATED CONCERNS ........................................................................................4

RISKS OF NEGATIVE OUTCOMES IF TESTING METHODS ARE NOT FAIR OR VALID ...........................7

HOW TO ENSURE TESTING METHODS ADHERE TO PROFESSIONAL AND LEGAL STANDARDS .......8

SPECIFIC RECOMMENDATIONS FOR COMMON TESTS .........................................................................11

CONCLUSION .........................................................................................................................................12

REFERENCES ............................................................................................................................................13

Introduction

Anyone familiar with the selection standards established to ensure compliance with civil rights laws knows that each of these more rigorous selection methods is a type of employment “test” and, as such, must comply with applicable standards for ensuring validity and fairness. However, there is a common misconception that the simpler, less rigorous selection methods do not meet the definition of an employment “test” and, therefore, do not need to be developed through methods established to ensure validity or even administered in ways that safeguard against possible discrimination. The purpose of this white paper is to explain the definition of a “test,” what common tests most agencies use, why these tests must conform to legal guidelines and how to create better selection processes that meet legal standards while still meeting the specific needs of the agency.

Hiring is Testing (Even if You’re Not Calling it a “Test”) | WHITE PAPER2

Any hiring process in which there is more than one job applicant must involve some method of determining which applicant is most suited for the job. Methods can take many forms, from simple minimum qualification (MQ) screenings, supplemental applications (often referred to as training and experience [T&E] evaluations) and hiring interviews, to much more rigorous methods such as written examinations, performance tests, structured interviews and assessment centers.

Hiring is Testing (Even if You’re Not Calling it a “Test”) | WHITE PAPER

What Constitutes a Test?

This wide-ranging determination by the EEOC of what is considered an employment test holds great weight, as the EEOC is the primary enforcement agency for the laws governing employment selection. The U.S. Department of Labor provides additional information regarding what is considered a test by stating that tests include not just the traditional knowledge and ability tests, but also “inventories, subjective procedures and projective instruments.” The fact that “subjective procedures” can be tests means that the definition of an employment test is not based on the characteristic of objectivity in the measure itself, nor on its validity or reliability, but rather on its use as an instrument in making hiring decisions. The concept that a test is anything used to make an employment decision is recognized throughout legal and professional standards for employment selection. This broad stance leaves little room for interpretation and holds employers responsible for making sure that all of their practices, whatever they are, follow the standards established for the validation of employment tests, most notably, those described in detail in the Uniform Guidelines on Employee Selection Procedures (1978); 43 FR 38295 (August 25, 1978), which were developed by the EEOC, the Civil Service Commission, the Department of Labor and the Department of Justice to assist employers in complying with civil rights laws.

3

According to Richard Tonowski of the Equal Employment Opportunity Commission (EEOC) Office of General Counsel, “An employment test is any procedure used to make an employment decision.” Therefore, an employment decision is made whenever any job applicant, candidate or incumbent is singled out for a particular action in an employment context. That action could be hiring, promotion, layoff, change in job classification or compensation, selection for a training program and so on. Therefore, in the context of the hiring process, virtually anything used to compare candidates or aid in decision-making at any point in the process should be regarded as a test.

4 Hiring is Testing (Even if You’re Not Calling it a “Test”) | WHITE PAPER

Minimum Qualifications (MQs) Screening

The MQ screening process is typically the first step in most hiring processes. Sometimes referred to as a résumé review or résumé screen, this process entails applicants submitting their résumés or applications for hiring managers or human resources staff to review. Each applicant’s education, experience or other qualification(s) is compared to the MQs listed in the job posting to narrow down the applicant pool to only those who meet the requirements and are, thus, “minimally qualified.” The listed MQs are usually drafted using the input of hiring managers and human resources professionals and are based on job-related standards and specifications.

The actual review of the applications and/or the résumés can be completed by a human being or by an automated applicant tracking system. This process is typically quick and easy while requiring minimal resources to develop and administer. The MQ screening process works on the foundation that past performance is the best predictor of future performance, to imply that an applicant’s background and experience, if related to the target job, will predict success in that job. When developed and administered correctly, the MQ screening process can be a highly-effective method for narrowing down large candidate pools to only those who match the criteria. When not properly developed and administered, however, the MQ screening process can have a negative impact on the success of the hiring process.

“

There are a wide variety of employment tests that can be used to make hiring decisions. Three of the most common tests utilized, which are often not considered “tests,” include the aforementioned MQ screens, supplemental applications or T&Es and hiring interviews. These common test types tend to be the quickest, easiest and most inexpensive to use for narrowing a candidate pool and/or comparing candidates. Although each provides value during a hiring process, these tests are susceptible to many pitfalls if not developed and administered in accordance with employment testing standards. These common missteps can lead to negative outcomes for the applicants, as well as the agency, and can lead to legal repercussions.

When developed and administered correctly, the MQ screening process can be a highly-effective

method for narrowing down large candidate pools to only those who match the criteria.

Common Tests and Related Concerns

One potential pitfall is that the criteria used for screening out candidates may not be up-to-date. Therefore, it is critical that MQs accurately match the current state of the job. Many job descriptions (and the MQs stated on them) remain unrevised for years while the job continues to change, policies change and technology advances. This creates a disconnect between the intent of the originally-developed MQs and what the current job requires. When these outdated criteria are used in an MQ screening process, there is a high likelihood that qualified applicants will be screened out unfairly and/or unqualified applicants will remain in the pool.

Another pitfall is the possibility of disparate impact occurring as a result of incorrect MQ usage. Disparate impact occurs when policies, practices, rules or other systems that appear to be neutral result in a disproportionate impact on a protected group. An example of this would be requiring a college degree for a job with no basis or job-related need for that degree. This can unintentionally result in socioeconomic discrimination because a certain group, specifically those without a college degree, were disqualified based on criteria irrelevant to success in the job.

A third pitfall of the MQ screening process is the possibility of bias, which can occur when an application is reviewed by a human. Although likely unintentional, an applicant’s name can indicate gender or nationality, which may elicit unconscious biases and lead to preference for some and discrimination against others. The potential for human error in an MQ screening process may be avoided by using an automated screening process through the use of specialized software; however, this method is also subject to pitfalls of its own. For example, using a computer to screen can lead to the elimination of qualified candidates inadvertently due to a minor error such as a qualified candidate failing to utilize the correct keyword in his/her application.

Supplemental Application

The supplemental application, or T&E evaluation, is typically seen in two formats: the point-based supplemental application and the narrative-based supplemental application. The narrative-based approach is often used as an addition to an applicant’s résumé or application and asks job-related questions that require open-ended responses. These questions require an applicant to construct a narrative response, which would ideally display his/her background and experience in more detail to give the evaluator a better understanding of his/her qualifications as they relate to the job. A well-developed narrative-based process is rated on various predetermined job-related criteria consistently across all applicants, and, when based on accurate job analytic data, has strong predictive validity.

The point-based supplemental application is the simpler, more easily developed, administered and scored version. Here, applicants are typically provided a list of task, knowledge, skill and ability statements and then asked to rate themselves on their level of proficiency or experience with these statements using a numerical scale. The scale will vary from test to test, but, in general, the greater the numerical rating applicants give themselves, the more points they will be awarded on the test. Many agencies choose these types of tests because they are easy to develop, do not have to be kept confidential and can be placed online for anyone to take at any time.

5

Common Tests and Related Concerns (cont.)


6

Research has shown, however, that point-based evaluations have very little predictive validity, meaning that these tests are usually ineffective at identifying quality applicants who would perform well on the job. Distortion of applicants’ responses to these self-ratings can result in applicants either intentionally self-inflating their ratings or simply unable to accurately rate themselves on their abilities. As a result, applicants who intentionally or unintentionally distort their self-ratings can end up at the top of an eligible list, while those applicants who may be more honest or adept at their ability to self-reflect and provide accurate self-ratings will often be at the bottom of an eligible list. Additionally, there is typically little variation in applicant performance on these types of tests, which negates the ultimate purpose of using the test since the results will not provide much differentiation between applicants. Nevertheless, it is important to recognize that this may be the only type of test agencies are able to utilize due to highly-specialized positions or limited resources.

Interview

The interview is one of the most widely used tests for selecting individuals to hire or promote. The interview allows the evaluator to ask a set of questions assessing a candidate’s experience and/or knowledge related to the job. If well-developed and properly administered, the interview can be a highly predictive test for identifying qualified candidates. Interview responses should be rated against predetermined rating criteria, either per question or per domain/competency to ensure that candidates are being assessed only on qualities that are job-related and expected upon entry to the job.

Although interviews have the potential to be an effective test, their validity and reliability are often weakened by a few common pitfalls. One of the most common pitfalls is lack of standardization. For example, not having a standard set of predetermined questions administered to all candidates equally can result in some candidates receiving more difficult questions than others. This greatly reduces the reliability of the test and can also cause candidates to perceive the process as unfair and potentially discriminatory.

Another pitfall that can occur in an interview is unintentional bias. Many forms of bias can occur in the interview process, despite good intentions, resulting in the evaluator’s skewed perceptions about the candidate that can be either favorable or unfavorable. One such bias is called the halo/horn effect, which occurs when the evaluator allows one strong positive or negative impression of a candidate’s response to an interview question to influence his/her ratings for all subsequent interview question responses. Another example of a bias that can occur is the similar-to-me effect, which is the tendency for the evaluator to compare a candidate’s perceived characteristics to how the evaluator sees himself/herself. That perceived connection can often lead to more favorable ratings than are deserved.

Being aware that the interview is an actual test that needs to be created with structure, standardization, job-related criteria and an awareness of the biases that can occur can help alleviate these pitfalls.

Common Tests and Related Concerns (cont.)


“Making a bad hire can be a costly mistake considering

the financial loss from the need to provide extra training to the individual to the cost of having to

restart the recruiting and selection process.7 Hiring is Testing (Even if You’re Not Calling it a “Test”) | WHITE PAPER

One significant, negative impact is hiring an individual not suited for the job. If the testing utilized is ineffective or is not accurately evaluating the applicants or candidates based on job-related criteria, there is a possibility that a lesser-qualified, or even unqualified, individual will be selected. Making a bad hire can be a costly mistake considering the financial loss from the need to provide extra training to the individual to the cost of having to restart the recruiting and selection process all over again. A bad hire can also have a negative impact on the morale of all employees in the agency, given that the other employees may have to make up for the bad hire’s lower productivity. This has a larger financial impact on an agency than is often considered and should be recognized when evaluating the importance of an effective selection and hiring process.

Finally, any hiring method that does not meet standards may be in violation of civil rights laws and regulations, merit rules and/or civil service rules. Not only does this hurt an agency’s reputation, but defending against litigation or arbitration is an expensive and timely process for any agency to undertake. There is an incredible financial burden placed on an agency to simply defend itself in the event of litigation for being in violation of civil right laws and the financial burden is increased further if the agency is found guilty and forced to provide an award to the plaintiff. Data reported by the EEOC in FY 2017 showed 84,254 total cases were filed against employers in the United States. Of those cases, the majority were related to discrimination based on race (33.9 percent) and sex (30.4 percent). In that report, the EEOC claims winning over $42 million for aggrieved parties during the same FY 2017 period.

As a result, it is important to recognize the severity of the implications that invalid and unfair hiring methods can have on an agency. Luckily, these negative outcomes can be prevented by adopting a few practices to improve the legal defensibility, validity and reliability of the tests utilized for hiring.

Risks of Negative Outcomes if Testing Methods Are Not Fair or Valid

If the testing methods that an agency employs are invalid or unfair, there is potential for a wide range of negative outcomes. These outcomes not only impact the applicants and candidates who may have been unfairly treated and, thus, denied a job but typically will have a much larger impact on the agency itself.

Job Analysis

A job analysis is the systematic and structured process of gathering information about a job to make informed decisions and develop defensible processes, such as those for selection and promotion. There are several methods and ways in which to go about conducting a job analysis, but, for legal defensibility, the process must meet the minimum standards established in the Uniform Guidelines.

While a job analysis process can be a lengthy and labor-intensive process, particularly when studying a job with a large number of incumbents in different work units, departments, locations and job responsibilities, it is possible to conduct a job analysis relatively quickly and without significant resources. Regardless of complexity, a proper job analysis should consist of these basic steps:

1. Conduct a Literature Review – In order to ensure a comprehensive job analysis, developing an initial understanding of the job’s core components is critical. Job-related literature includes, but is not limited to, previous job analyses, class specifications, job descriptions, standard operating procedures, policy manuals, training manuals and organizational charts.

2. Collect Additional Job Information – The job analysts need to compile as much job-relevant information as possible. The most valuable resources in the job analysis process are the people who are most familiar with the job being studied (i.e., a sample of incumbents and their supervisors). These subject matter experts (SMEs) should be utilized in the process to ensure that only the most relevant information is being included. There are recommended criteria for SMEs who participate in the process, as it is important that the SMEs represent the total population of employees performing the job under study. Information is typically collected from the SMEs through interviews, job observations and focus groups.

8

Even when considering all the potential downfalls of the common tests addressed previously, it is expected that agencies will continue to utilize them given that they are quick, easy and inexpensive to develop and administer. The good news is that, when developed and executed properly, they can bring great value to the hiring process. It is important that all hiring stakeholders take the steps necessary to enhance the validity and fairness of these tests to increase the likeliness of them producing quality hires. The first and most important step is to start with a thorough job analysis, which helps to identify the key aspects of the job and determine what should be assessed in any testing and hiring process.

How to Ensure Testing Methods Adhere to Professional and Legal Standards


9

How to Ensure Testing Methods Adhere to Professional and Legal Standards (cont.)


Develop Task and Knowledge, Skills and Abilities Lists – In this step, job analysts create a preliminary list of job tasks and knowledge, skills and abilities (KSA) statements. This list will then be reviewed and refined by carefully vetted SMEs to more accurately represent the tasks performed on the job and the KSAs required for successful job performance. Follow-up review sessions between job analysts and SMEs are often conducted to ensure the accuracy and completeness of the statements.

Plan/Assemble the Data Collection Tool – The Uniform Guidelines require that, at a minimum, the following data must be gathered:

• How frequently each task is performed on the job

• How important each task is in terms of overall job performance

• How important each KSA is for successful job performance

• If each KSA is needed upon hire or promotion, rather than learned on the job

Collect Data – The task and KSA rating process is usually done via survey completed by SMEs, typically a representative sample of job incumbents and supervisors. For jobs with few incumbents, ratings are often gathered in a meeting with an SME panel.

Analyze Data – Once the data have been collected, they should be cleaned of any erroneous or non-purposeful responses. After the data have been cleaned, statistical criteria need to be set to determine the inclusion parameters against which tasks and KSAs will be retained or excluded. It is at this stage of the job analysis process that the important tasks of the job and the KSAs that are important and needed upon entry into the job are also identified.

Link Tasks to KSAs – The retained tasks and KSAs need to be “linked” by identifying which KSAs are necessary for the completion of each job task. It is important for SMEs to identify the linkages between KSAs and job tasks to further identify the most critical characteristics necessary for the given job and to ensure that the KSAs relate directly to the work being performed.

Document the Process – This is arguably the most critical, yet often overlooked, component of the job analysis. Without proper documentation, even a thorough process would be an easy target for an opposing legal team. A well-structured technical report should include, but not be limited to the following: the user(s), location(s) and date(s) of the process; project background information; content of the job; intended uses and applications; contact information; and a declaration of accuracy and completeness.

If a thorough job analysis following these steps is completed, then a strong foundation has been set in which to develop all subsequent hiring methods.

3.

4.

5.

6.

7.

8.

10

Examination Plan

Once a job analysis has been conducted and, as a result, an agency knows which job-related KSAs should be tested, an examination plan should be developed to determine exactly how those KSAs will be tested. To determine which KSA is best measured by a certain selection test, the job analyst or stakeholder reviews each KSA and makes a logical determination based on the type of KSA and the psychometric properties of different selection tests. For example, if the KSAs are mathematical in nature, it may lend itself to a test in which candidates can be tested on math concepts, such as a multiple-choice examination. If physical abilities are critical and expected of candidates, there may be a need for a performance test in which candidates are required to demonstrate possession of those abilities.

The next considerations in the examination planning process are the resources available to the agency, including cost, time, expertise and facilities available. Once this information is factored in, you may narrow down the options to what is realistic and affordable in terms of the resources needed to develop and administer the test. Another consideration is the number of applicants expected in the testing process. If there is a large pool of applicants, consider using a narrative-based supplemental examination or a multiple-choice test to help reduce the number of applicants upfront to a more manageable number for further testing. Once the applicant pool is reduced to a more manageable size, other testing options may become more viable.

An examination plan consisting of more than one selection test will typically identify the relative weight of each test in the hiring process. This can be done using the job analysis data showing the relative importance of the KSAs being assessed by each test or the number of job tasks linked to each KSA. It is critical that the examination plan is based on a rational, thoughtful process that can be justified and documented.


How to Ensure Testing Methods Adhere to Professional and Legal Standards (cont.)

MQ Screening

• Understand that more experience is not automatically better.• Required years of experience and the specific type of experience should be determined with the help of

SMEs and based on job analysis results.• Ensure individuals reviewing applications and screening for MQs are aware of the possibility of unintentional

bias and how to eliminate it.• If using an automated system to screen applications or résumés, work with SMEs to identify key phrases

and words to input into the computerized screening tool.

Supplemental Applications

• Avoid point-based self-ratings, if possible. At a minimum, always use methods to reduce lying or response distortion, such as requiring applicants to provide contact information of a reference that can verify their self-reported information.

• If possible, choose to use the narrative supplemental applications that are scored using behaviorally-anchored rating scales or other objective criteria.

• All rating criteria should be based on tasks and KSAs collected with SME input.

Interview

• Base questions on job analysis results and SME input to ensure the interview is directly related to the job and the process is structured.

• Ask the same questions for all candidates. • Scoring should be conducted objectively using criteria derived from the job analysis.• Ensure all evaluators are trained in conducting and scoring the interview.

11

In addition to the recommendations provided for improving the foundation and planning of an effective and valid hiring process, below are some additional points and suggestions specific to the most commonly-used tests for consideration.

Specific Recommendations for Common Tests


12 Hiring is Testing (Even if You’re Not Calling it a “Test”) | WHITE PAPER

According to professional and legal standards regarding hiring, any method that is used to make an employment decision is considered a test and is required to follow the established guidelines for proper validation and administration. This includes the more commonly used tests discussed within, such as MQ screening, supplemental applications and interviews. Regardless of the options agencies currently use, they must do their due diligence to investigate other options to ensure that they are not overlooking a comparable test that is more valid and effective for their intended use. If possible, utilize multiple tests in a hiring process to collect several points of data to better evaluate candidates. The more information obtained about a candidate, the more easily an accurate decision can be made. Regardless of the test(s) being used in a hiring process, it is critical to ensure that there is a direct link to the job for which it is being used to hire while also minimizing the potential for unfair treatment of any individual or group. This can be achieved by conducting a job analysis, developing an examination plan, using SMEs and choosing the appropriate testing method.

About CPS HR Consulting

CPS HR Consulting is a self-supporting public agency providing a full range of integrated HR solutions to government and nonprofit clients across the country. Our strategic approach to increasing the effectiveness of human resources results in improved organizational performance for our clients. We have a deep expertise and unmatched perspective in guiding our clients in the areas of organizational strategy, recruitment and selection, classification and compensation and training and development.

To discuss how our services and expertise can benefit your organization, reach out to us at: CPS HR Consulting2450 Del Paso Road, Suite 220Sacramento, CA 958341-800-822-4277

Conclusion

13

Civil Rights Act of 1964 § 7, 42 U.S.C. §2000e et seq (1964). Retrieved from Equal Employment Opportunity Commission website: https://www.eeoc.gov/laws/statutes/titlevii.cfm.

Charge Statistics (Charges filed with EEOC). (n.d.). Retrieved from: https://www.eeoc.gov/eeoc/statistics/enforcement/charges.cfm.

Schmidt, F.L., Oh, I., & Shaffer, J.A. (2016). The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings (October 17, 2016). Fox School of Business Research Paper.

Significant EEOC Race/Color Cases. (n.d.). Retrieved from: https://www.eeoc.gov/eeoc/initiatives/e-race/caselist.cfm#hiring

The U.S. Department of Labor Employment and Training Administration. (2000). Testing and Assessment: An Employer’s Guide to Good Practices [PDF file]. Retrieved from: https://www.onetcenter.org/dl_files/empTestAsse.pdf.

The U.S. Equal Employment Opportunity Commission. (2007, May 16). Testing and Equal Employment Opportunity: Threats and Promises. Retrieved from: https://www.eeoc.gov/eeoc/meetings/archive/5-16-07/tonowski.html.

U.S. Equal Employment Opportunity Commission, U.S. Civil Service Commission, U.S. Department of Labor, & U.S. Department of Justice. (1978). Uniform Guidelines on Employee Selection Procedures, Federal Register, 43 C.F.R. § 38295-38309.

References


2450 DEL PASO ROAD, SUITE 220

SACRAMENTO, CALIFORNIA 95834

(916) 263-3600

(Even if You’re Not Calling it a “Test”) - CPS HR Consulting...2450 DEL PASO ROAD, SUITE 220...

Documents

Transcript of (Even if You’re Not Calling it a “Test”) - CPS HR Consulting...2450 DEL PASO ROAD, SUITE 220...