EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program...

17
Center for Research on Employment of Disadvantaged Populations EMET – Culturally Fair Testing: Summary Report Noam Fischman Tali Semani The study was initiated by JDC-Israel-TEVET and the UJA-Federation of New York, and funded with the assistance of the latter RR-699-15

Transcript of EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program...

Page 1: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Center for Research on Employment

of Disadvantaged Populations

EMET – Culturally Fair Testing:

Summary Report

Noam Fischman Tali Semani

The study was initiated by JDC-Israel-TEVET and the UJA-Federation of New York,

and funded with the assistance of the latter

RR-699-15

Page 2: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

EMET – Culturally Fair Testing:

Summary Report

Noam Fischman Tali Semani

The study was initiated by JDC-Israel-TEVET and the UJA-Federation of New York,

and funded with the assistance of the latter

Jerusalem October 2015

Page 3: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Editor: Anat Berberian

English translation, Executive Summary: Evelyn Abel

Print Design and Production: Leslie Klineman

Myers-JDC-Brookdale Institute

P.O.B. 3886

Jerusalem 91037, Israel

Tel: (02) 655-7400

Fax: (02) 561-2391

Web site: www.jdc.org.il/brookdale

Page 4: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Related Myers-JDC-Brookdale Institute Publications

Fischman, N.; King, J. 2013. EMET – Culturally Fair Testing: A Review of the International

Literature on Employment Screening Methods and an Examination of its Implications for Israel.

RR-629-13 (Hebrew)

Habib, J.; Halaban-Eilat, H.; Shatz, A.; Almog, Y. 2010. Follow-Up on Key Indicators of the

Nationwide Situation of the Ethiopian-Israeli Population. RR-560-10 (Hebrew)

King, J.; Fischman, N.; Wolde-Tsadick, A. Twenty Years Later: A Survey of Ethiopian Immigrants

who have Lived in Israel for Two Decades or More. RR-560-12 (Hebrew)

To order these publications, please contact the Myers-JDC-Brookdale Institute,

P.O.B. 3886, Jerusalem 91037;

Tel: (02) 655-7400; Fax: (02) 561-2391;

E-mail: [email protected]

Reports are also available on the Institute website www.jdc.org.il/brookdale

Page 5: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

i

Executive Summary

1. The Emet Program

The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of the difficulties of hiring Ethiopian Israelis for positions when certain tests are

part of the screening process. The main goal of the program was to facilitate the hiring and

advancement of Ethiopian Israelis, and of other minority groups, by promoting the use of culturally-

appropriate employment screening tools. The Myers-JDC-Brookdale Institute (MJB) in cooperation

with the Adam Milo Institute, a large Israeli assessment center, conducted an extensive study for

TEVET. The study consisted of the two stages described below.

2. Study Objectives and Components

The main goals of the first stage of the study were to explore methods to reduce the gaps between

different groups in employment screening, and to examine whether the existing tools prevent Ethiopian

Israelis from being hired. Some of the findings were published in Culturally Fair Testing: A Review

of the International Literature on Employment Screening Methods and an Examination of its

Implications for Israel1and were presented at various forums to human-resource personnel,

researchers, professionals, and fieldworkers actively involved with the Ethiopian-Israeli community.

After reviewing the findings from the first stage of the study, TEVET, with the help of a

multidisciplinary team of professionals, developed eight new culturally appropriate employment

screening tools that assess mathematical, verbal, interpersonal and other skills. At the request of

TEVET and in partnership with the Adam Milo Institute, we conducted a validation study in order to

identify the tools that are most effective at both predicting job performance and reducing the gaps

between the groups. This is an important study as we found no other published Israeli studies on the

employment screening tools used in Israel.

3. Literature Review

The literature review from 2013 (Fischman & King, 2013, idem) showed that minority groups (e.g.

Black Americans) do not perform as well as the majority group (e.g., White Americans) with many

employment screening tools. However, the gaps in actual job performance between these groups are

actually much smaller than those found with many popular screening tools, such as mental-ability tests.

As a result, talented candidates from minority groups are often excluded from jobs in which they could

perform well, if given the opportunity. The literature review focused mostly on research from the US

1 Fischman, N. and King, J. 2013. EMET – Culturally Fair Testing: A Review of the International Literature on

Employment Screening Methods and an Examination of its Implications for Israel. RR-629-13. Myers-JDC-Brookdale Institute, Jerusalem (Hebrew).

Page 6: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

ii

and describes how the tools that show the largest performance gaps between groups are also the most

valid predictors of job performance. This is called the Diversity-Validity Dilemma. The best way to

deal with this dilemma is to use tools that are designed to assess a wide range of skills and aptitudes,

and not to focus solely on mental abilities.

4. Do the Existing Tools Prevent Ethiopian-Israeli Job

Candidates from Finding Jobs

The primary purpose of this stage of the study was to determine if Ethiopian-Israeli job candidates

perform differently than other Israeli candidates with the existing employment screening tools. This

stage of the study included three components:

1. A comparison of the performance of Ethiopian-Israeli job candidates and other Israeli candidates,

with the existing employment screening tools.

2. In-depth interviews with Israeli employers that promoted workplace diversity in their organizations

by using alternative screening methods to facilitate the hiring of Ethiopian-Israeli job candidates.

3. Face-to-face interviews with a small group of Ethiopian-Israeli candidates after they completed a

day of testing at the Adam Milo Institute.

4.1 A Comparison of the Performance of Ethiopian-Israeli Job Candidates

and other Israeli Candidates with the Existing Employment Screening

Tools

The Adam Milo Institute provided data on 186 Ethiopian-Israeli job candidates tested between

November 2004 and July 2009. For every Ethiopian-Israeli candidate, a similar candidate from the

majority group was identified based on three criteria: gender, the position being applied for, and the

date of the testing. Overall, the two groups were very similar in these three criteria.

Large, significant differences on the existing tests of mental ability were found between the Ethiopian-

Israeli job candidates and the candidates from the majority group. Thus, using these tests as part of the

screening process is likely to have an "adverse (or disparate) impact" on the chances of Ethiopian-

Israeli candidates being hired. This, in turn, can limit cultural diversity in workplaces and

organizations.

Measures based on a task administered to candidates in a group setting (Group Dynamics Task) and a

series of other measures based on a professional tester's assessment revealed more moderate gaps.

Interestingly, no significant differences were found between the groups on measures based on a semi-

structured interview.

Page 7: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

iii

4.2 In-Depth Interviews with Ethiopian-Israeli Candidates Tested at an

Assessment Center

Personal face-to-face interviews were held with five male and four female Ethiopian-Israeli candidates

who had been tested in 2010 at the Adam Milo Institute. All were under 40, had immigrated to Israel

between the ages of 5 and 14, and most had a college degree.

The interviews showed that the candidates were very motivated to succeed on the screening tests

though they often did not know what was expected of them in order to succeed. Most said that the

screening process was "fair". Some, however, were critical, asserting, for example, that the tests

assessed skills irrelevant to the position being applied for or that they were not a valid measure of their

abilities. The employers that were interviewed voiced similar criticism.

4.3 Alternative Approaches to Recruiting and Screening Ethiopian-Israeli

Candidates – Interviews with Employers

We interviewed three managers of human resources and employee recruitment. All were interested in

promoting workplace diversity. Note that the managers recruited Ethiopian Israelis mainly for

technical and vocational positions. In addition, we interviewed the director of a leading manpower

company who was committed to placing Ethiopian-Israeli college graduates in positions in large

service companies dealing with finance, sales and communications. There was consensus that some of

the mental abilities being assessed were not relevant to many positions, and that the tests should be

changed to better assess specific job-related knowledge. The interviews confirmed that the employers

were satisfied with the work of the Ethiopian-Israelis they had hired on the basis of alternative

screening methods and that their job retention was high. The interviews also showed that beyond

specific core criteria, such as integrity and skills specific to the nature of the position, (e.g., a basic

knowledge of English in order to operate machinery, and the ability to read a manufacturer's document

in Hebrew), the employers believed that good job performance could be attained through on-the-job

training and professional support in the initial employment period.

5. Developing Alternative Screening Tools

Following in-depth discussions of the findings from the first stage of the study, TEVET decided to

develop new screening tools with the goal of reducing the gaps between the minority and majority

groups while maintaining, or even improving, their capability to predict job performance. To this end,

TEVET recruited a multicultural team of professional test developers and representatives from different communities and population groups. Eight tools were developed in this effort:

A biographical questionnaire (biodata), a semi-structured interview, and a role-playing task

testing a variety of skills, such as math ability, interpersonal skills, and persuasiveness. All three

of these tools were based on the self-reports of the people being tested. The semi-structured

interview and role-playing tasks also relied on assessments made by a professional tester.

Three tasks that use alternative methods for assessing math, verbal and analytical abilities along

with other mental abilities.

Two personality tests that assess six specific traits.

Page 8: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

iv

6. Validation of the Tools

In the second stage of the study, at the request of TEVET and in collaboration with the Adam Milo

Institute, another study was conducted to assess the quality of the new tools. Kav Mashve, a non-profit

organization that promotes equal employment opportunities for Arab college graduates, contributed to

the study by helping to recruit an additional group of Arab-Israeli college graduates.

Thirteen employers helped recruit Ethiopian, Arab, and other Israeli employees for the study.

The tools were evaluated in the following ways:

Assessment of the concurrent validity of each tool: We checked the correlation between the

scores yielded by the screening tools and actual job performance.

Assessment of the size of the gaps between the groups for each tool: We compared the average

scores of each group.

In addition, the study attempted to identify tools with the potential to minimize the gaps found between

groups while also accurately predicting job performance. In other words, we attempted to find the tools

that could best help employers overcome the Diversity-Validity Dilemma.

6.1 Study Design and Study Population

There were 144 participants in the study. We received assessments of job performance from the

managers of 142 of these participants. The testing process was meant to take about eight hours but

some of the participants, including several Ethiopian-Israelis, were exempted from certain parts in

order to stay within the time limit. As a result, about 120 participants were tested on all of the tools.

The testing sessions were held mostly at branches of the Adam Milo Institute though some participants

were tested at their place of work.

Important differences were found in the characteristics of the various population groups examined in

the study. Participants from the Arab-Israeli population were mainly young men, and most were

college graduates. Ethiopian-Israeli participants were mainly immigrants with lower levels of

education. Ethiopian-Israelis born in Israel and speaking Hebrew at the mother-tongue level were

underrepresented in the study. Moreover, 66% of the participants from the majority group, and 69%

of the Arab-Israelis were working in jobs that required a college degree compared with 41% of the

Ethiopian Israelis. The groups were similar in other aspects: about 32% of the participants from all the

groups reported having management experience, and 50% reported that they had previously been

tested at an assessment center.

6.2 Results

Figure 1 presents the position of each of the tools along two axes: concurrent validity (x-axis) and size

of the gaps between Ethiopian-Israeli participants and the majority group (y-axis). The size of the gap

between these groups on work performance is presented in the gray rectangle in the middle of the

graph. This gap was estimated to be 0.56 standard deviations. The objective was to identify the tools

Page 9: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

v

that were the best at predicting job performance with gaps between the groups that were similar to, or

smaller than, the gap in actual job performance.

The biographical questionnaire and the semi-structured interview were the most valid tools and

showed relatively small gaps between the groups. These tools are in the blue section of the figure.

Most of the new tools showed gaps that were smaller than the gap in actual job performance

between the groups.

The existing tests of mental ability and the new alternative tasks that assess mental ability show

the largest gaps between the groups. They appear at the top of the graph.

Figure 1: Estimates of Concurrent Validity and the Size of the Gaps between Groups, with the

New Screening Tools and the Existing Mental-Ability Tests

Findings on the New Tools:

The biographical questionnaire has considerable potential to improve employment-screening

processes. It was the most valid predictor of job performance. The size of the gap found between

the Ethiopian-Israeli and majority groups when using this tool was in fact a bit smaller than the

gap in job performance between these groups. In addition, Arab-Israeli participants performed

slightly better than the majority group when this tool was used.

Existing mental abilities tests

Semi-structured interview

Biographic questionnaire

Alternative math task

Alternative verbal task

Alternative analytical task

Role-playing task

Grit

Extraversion

Openness

Emotional stability

Conscientiousness

Agreeableness

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

Size

of

the

Gap

s

Concurrent Validity

Gap in job performance = 0.56 standard deviation

The most valid tools that show the

smallest gaps

Page 10: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

vi

The semi-structured interview and role-playing task also showed relatively small gaps between

the groups. Furthermore, these tools were found to have similar, or even slightly higher,

concurrent validity than the existing mental-ability tests. Note that since the participants of the

study were employed rather than looking for work, they may have been more honest in their

responses and less influenced by social-desirability effects, which can appear in screening and

testing processes.

The alternative mental-ability tasks assessing verbal, math and analytic abilities were found to

have similar, or even somewhat stronger, concurrent validity in comparison with the existing

tests. However, like the existing mental-ability tests, they showed large performance gaps

between the groups,

No significant differences between the groups were found with the personality tests. However,

these tests did not correlate with job performance.

Findings Relating to Existing Mental-Ability Tests:

The existing mental-ability tests were less valid as predictors of job performance than some of

the new tools. As expected, the existing mental-ability tests showed very large gaps between

Ethiopian-Israeli participants and the majority group. In fact, these gaps were two to three times

larger than the gaps in job performance found between these groups.

The number of Arab Israeli participants was small. Therefore, the gaps in performance found on

the existing mental-ability tests between Arab-Israeli college graduates (not shown in the figure)

and the majority group were not statistically significant. Note, however, that the Arab Israelis

scored slightly lower than the majority group with this tool, while their actual job performance

scores were a bit higher than those found in the majority group.

The findings from the first stage of the study indicated that the assessments of the psychologist,

which are based on the results from the entire screening process, may help to reduce gaps in the

measurement of mental ability. However, we did not check the concurrent validity of these

subjective measures.

We performed a series of regression analyses to determine whether the new screening tools add to the

ability of the existing mental-ability tests to predict job performance. We also attempted to determine

the extent to which the existing tests enhance the capability of the new tools to predict job performance:

The results of the regression analyses further confirm that the biographical questionnaire, semi-

structured interview, and role-playing tasks add substantial predictive ability to the existing tests.

Moreover, the existing mental-ability tests did not improve the capability of the new tools to predict

job performance in the study sample. The results indicate that the new tools have the potential to be

valid predictors of job performance for the jobs included in the study.

Study Limitations

The findings from this study should be interpreted cautiously as they are based on a relatively small

sample that is not representative of the general population of Ethiopian Israeli, Arab Israeli, and other

Page 11: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

vii

Israeli jobseekers. Nonetheless, the findings indicate that some of the newly developed tools are as

good as or better than the existing tools at predicting job performance.

Several methodological problems should be noted before drawing any conclusions based on the current

findings alone. In the second stage of the study:

The sample was relatively small (n=144). Moreover, nearly half of the participants were

Ethiopian Israeli (n=69); far fewer belonged to the majority group (43) or to the group of Arab-

Israeli college graduates (32).

As previously noted, important differences were found in the characteristics of the various

populations.

Compared with the other groups, a larger portion of Ethiopian Israelis worked at jobs that did not

require a college degree. Ethiopian Israelis also tended to earn a lower salary.

The participants in this study were not jobseekers; all were employed at the time of the study so

that the effects of social desirability may not have been a factor for them. Thus, validity estimates

from other studies that are based on job candidates (predictive validity) rather than on the current

employees within an organization (concurrent validity), may be somewhat smaller.

Despite these issues, the pattern of results provide important insights about employment-screening

processes in Israel. The estimates of concurrent validity presented in this report were similar to those

found in the international literature. Furthermore, the gaps between Ethiopian Israelis and the majority

groups that were found with the existing tests of mental-ability and with semi-structured interviews

were similar in the two stages of the study.

5. Insights, Discussion and Future Directions

The literature describes the Diversity Validity Dilemma and suggests methods for dealing with this

dilemma (see Fischman & King, 2013;2 Ployhart & Holtz, 20083). This study supports the following

claims:

Alternative screening tools can be developed to improve the chances of minority-group

candidates being hired, without sacrificing the ability to select suitable employees. Examples

from this study are the biographical questionnaire, the semi-structured interview and role-playing

tasks. Promising examples from the literature are structured interviews and integrity tests.

The development of different combinations of screening tools is a good way to devise a valid

screening process that can also minimize the size of the gaps between groups.

2 Idem 3 Ployhart, R.E.; Holtz, B.C. 2008. "The Diversity-validity Dilemna: Strategies For Reducing Racioethnic and Sex Subgroup Differences and Adverse Impact Selection." Personnel Psychology 61:153-172.

Page 12: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

viii

The methods that do not focus solely on mental ability, but instead assess a wide range of domains

(such as knowledge, skills, aptitudes and other characteristics) are the most successful in dealing

with the diversity validity dilemma. These methods succeed in maintaining valid prediction of

job performance while also helping to narrow the gaps between the groups.

The literature also points to less successful methods of narrowing the gaps between groups. This study

lends additional support to the following claims:

Choosing tests of specific mental abilities according to their relevance to the position, rather than

tests of general mental-ability does not significantly reduce gaps.

Attempts to develop alternative tests that assess mental ability do not generally lead to a

narrowing of gaps.

Tests of personality traits tend to have poor predictive validity.

There is no empirical support for the idea that practice and preparation (in a course) or orientation

for minority-group candidates significantly reduces gaps between groups4.

This study discovered large gaps in performance between Ethiopian-Israeli job candidates and

candidates from the majority group on mental-ability tests, and more moderate gaps when using newly-

developed tools. The gaps were smaller when comparing Arab Israelis with the majority group.

However, while none of the differences were significant, Arab-Israeli participants scored somewhat

lower than participants from the majority group on the existing tests even though they achieved

somewhat higher job-performance ratings.

The biographical questionnaire stands out as the most valid tool. It also shows relatively small gaps

between the groups, similar to the size of the gaps found in job performance.

We did not find other reports from Israel that provide estimates for the validity of employment-

screening tools. Considering the popularity of employment-screening tools in Israel, the lack of

published research lends added importance to this study. However, the potential of screening tools

should be examined in future studies with larger samples, with participants from more varied

professions.

4 Another pilot of TEVET's that was not described in this report tracked a small group of 15 students who had

participated in a preparatory course for employment tests of mental-ability. No significant improvements

were found in their performance on these tests.

Page 13: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

ix

Acknowledgments

I would like to thank all the people who helped conduct the study and prepare this report.

I am grateful to Judith King who coordinated the study at the start, and to Liat Basis, who provided

consultation. It was a pleasure to work with the staff of the program at TEVET: Sigal Shelach, Nurit

Lapidot, and Ina Soltantovich.

I would also like to thank my colleagues at the Myers-JDC-Brookdale Institute: Denise Naon and all

the members of my team for their ongoing advice and feedback; Jenny Rosenfeld and Anat Berberian,

for editing the report; Evelyn Abel for translating the Executive Summary into English; and Leslie

Klineman for print design and production.

Page 14: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Table of Contents

1. Introduction 1

1.1 Study Goals and Components 1

1.2 Developing Alternative Screening Tools at TEVET 1

1.3 Validation of Tools 2

1.4 Structure of Report 2

2. Review of the Literature 2

3. Do Existing Tools Hamper the Hiring of Ethiopian Israelis? 4

3.1 Investigation of Gaps between Examinees of Ethiopian Origin and the Majority Group 4

3.2 In-Depth Interviews with Candidates of Ethiopian Origin Tested at a Screening Institute 16

3.3 In-Depth Interviews with Employers 19

4. Developing New Screening Tools 21

5. Validation of Tools 24

5.1 Study Goal 24

5.2 Recruiting Employers and Participants 24

5.3 Study Design and Study Population 25

5.4 Findings 27

6. Discussion 39

Bibliography 41

Appendices

Appendix I: Itemized Screening Tools Completed by Candidates 43

Appendix II: Itemized Job Type, by Complexity 45

Appendix III: Sample Items of the New Tools and Guide for Score Definitions 46

Appendix IV: Additional Tables and Figures 52

List of Tables

Chapter 3: Do Existing Tools Hamper the Hiring of Ethiopian Israelis?

Table 1: The Size of the Performance Gap in Estimates of "General Mental Ability" 7

Table 2: Examining the Risk for Adverse Impact on Ethiopian-Israeli Job Candidates according

to the Four-Fifths Rule Based on the Estimated Skills of General Mental Ability 7

Table 3: Size of Gaps in Scores of Specific Mental-Ability Tests 8

Table 4: Size of Gaps on Measures of Suitability for a Position and Management Potential,

Based on the Semi-Structured Personal Interview 9

Page 15: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Table 5: Examining the Risk for Adverse Impact on Ethiopian-Israeli Job Candidates

according to the Four-Fifths Rule, Based on Suitability for a Position and

Management Potential, when Using the Semi-Structured Personal Interview 9

Table 6: Size of Gaps in Scores of the Suitability for a Position and Management Potential,

by the Group-Dynamics Test 10

Table 7: Examining the Risk for Adverse Impact on Ethiopian-Israeli Job Candidates,

Based on the Four-Fifths Rule when Using the Group-Dynamics Test 11

Table 8: Size of Gaps on Measures of General Mental Ability, Potential, and other

Specific Skills, Based on a Professional Evaluator's Subjective Rating 12

Table 9: Size of Gaps on Measures of Worker Efficacy and Productivity, Based on

a Professional Evaluator's Subjective Rating 13

Table 10: Size of Gaps on Measures of Personality Traits, Based on a Professional Evaluator's

Subjective Rating 14

Table 11: Size of Gaps in the Summary Scores Provided by a Professional Evaluator 14

Table 12: Examining the Risk for Adverse Impact on Ethiopian-Israeli Job Candidates,

Based on the Four-Fifths Rule, when Using the Summary Recommendation of a

Professional Evaluator 15

Table 13: Size of Gaps on Select Measures and How these Measures Correlate with Each Other 16

Chapter 5: Validation of Tools

Table 14: Socio-Demographic Characteristics of Participants, by Population Group 25

Table 15: Employment Characteristics, by Population Group 26

Table 16: Size of Gaps between Minority Groups and the Majority Group, on Overall Job

Performance 28

Table 17: Size of Gaps between Minority Groups and the Majority Group on Existing

Specific Tests of Mental Ability 29

Table 18: Comparison of the Size of the Gaps between Ethiopian Israelis and the Majority

Group in both Stages of the Study 29

Table 19: Size of Gaps between Minority Groups and the Majority Group on the

New Screening Tools 31

Table 20: Concurrent Validity and Reliability of Existing Mental-Ability Tests and of the New

Screening Tools 34

Table 21: Size of the Gaps between Population Groups, and Concurrent-Validity Estimates

for the New and Existing Screening Tools among Participants Employed

in Positions Requiring an Academic Degree 35

Page 16: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Table 22: Results from Regression Analyses Examining the Capability of Various Combinations

of Tools to Predict Overall Job Performance. The combinations of tools include

the Estimate for General Mental Ability along with Selected New Tools 38

Table 23: Results from Regression Analyses Examining the Capability of Various Combinations

of Tools to Predict Overall Job Performance. The combinations of tools include

the Tests for Specific Mental Abilities along with Selected New Tools 39

Tables in Appendices

Appendix II: Classification of Positions, by Complexity

Table II-1: Classification of Various Positions by Complexity, and the Number of

Ethiopian-Israeli Candidates, and Candidates from the Majority Group Working

in each Position in Stage I of the Study 45

Table II-2: Classification of Various Positions by Complexity, and Number of Candidates

from every Population Group working in each Position in Stage II of the Study 45

Appendix III: Examples of Items in the New Screening Tools and Sample Scoring Guides

Appendix III-1: Characteristics of Examinees Assessed by the New Tools 46

Appendix III-2: Mapping of New Tools, by the Amount of Time it Takes to Administer Them 47

Appendix III-3: Sample Items and Scoring Guides from the Semi-Structured Interview 48

Appendix III-4: Sample Items and Scoring Guides from the Role-Playing Task 49

Appendix III-5: Sample Items and Scoring Guides from the Biographical Questionnaire 50

Appendix III-6: Sample Items and Scoring Guides from the Alternative Mental-Ability

Task Assessing Math Abilities 50

Appendix III-7: Sample Items and Scoring Guides from the Alternative Mental-Ability Task

Assessing Verbal Abilities 51

Appendix III-8: Sample Items and Scoring Guides from the Alternative Mental-Ability Task

Assessing Analytic Abilities 51

Appendix IV: Additional Tables and Figures

Tables:

Table IV-1: Number of Participants, by Employer Organization and Population Groups 52

Table IV-2: Number of Planned Participants in each Population Group 52

Table IV-3: Number of Participants who Completed each Screening Test, by Population Group 52

Table IV-4: Size of the Gaps Between Groups and Concurrent Validity for Measures of

Specific Skills and of Personality Characteristics that were Assessed

by the New Screening Tools 53

Page 17: EMET – Culturally Fair Testing: Summary Report€¦ · The Emet (Culturally Fair Testing) Program was established by JDC-Israel TEVET in 2010, following reports by employers of

Table IV-5: Differential Concurrent Validity of each of the Employment Screening Tools

Included in this Study as Predictors of the Mean Overall Job Performance Score,

by Population Group 56

Table IV-6: Differential Concurrent Validity of each of the Employment Screening Tools

Included in this Study as Predictors of the Managers' Subjective Ratings of

Overall Job Performance, by Population Group 56

Table IV-7: Feedback from Participants on the Screening Tools Examined in the Study,

by Population Group 57

List of Figures:

Figure 1: Estimates of Predictive Validity and of Size of Gaps between Groups,

Using Selected Screening Tools according to the Literature in the U.S. 3

Figure 2: Employment Screening Tools used by the Israeli Assessment Center,

and the Measures used in Stage I of the Study 6

Figure 3: Estimates of Concurrent Validity and of the Size of the Gaps between Groups,

with the New Screening Tools and the Existing Mental-Ability Tests 36