What Makes a Selection System Work

8/8/2019 What Makes a Selection System Work

http://slidepdf.com/reader/full/what-makes-a-selection-system-work 1/25

What makes a selection What makes a selection

system work system work Reliability and Validity Reliability and Validity



A selection device (SD) is: A selection device (SD) is:

Anything used to pick one person over another Anything used to pick one person over anotherfor a job, including:for a job, including:

ResumeResume

Interview Interview

Intelligence testIntelligence test

Drug testDrug test

Skills testSkills test And a bunch of other things And a bunch of other things



A hiring system generally uses 2 A hiring system generally uses 2

or more selection devicesor more selection devices A typical selection system might include: A typical selection system might include:

Review resumeReview resume

Phone interview Phone interview

Interviews with HR & SupervisorInterviews with HR & Supervisor

Intelligence testIntelligence test

Drug testDrug test

Background check Background check



To use this system and these To use this system and these

selection devices they must be:selection devices they must be: LegalLegal ² ² no ¶unjustified· adverse impactno ¶unjustified· adverse impact

ReliableReliable ² ² produce consistent resultsproduce consistent results

Valid Valid ² ² measure what they are supposed tomeasure what they are supposed tomeasuremeasure

PracticalPractical ² ² cost effectivecost effective

Acceptable to the applicant Acceptable to the applicant ² ² lets not make thelets not make theapplicant madapplicant mad



CorrelationCorrelation ² ² used to ¶evaluate·used to ¶evaluate·

reliability and validityreliability and validity CorrelationCorrelation ² ² compares 2 sets of numberscompares 2 sets of numbers

(height & weight)(height & weight)

FromFrom ² ²1.00 to 0.00 to +1.001.00 to 0.00 to +1.00 Generally, we are interested in the 0.00 to +1.00Generally, we are interested in the 0.00 to +1.00



Perfectly positive correlationPerfectly positive correlation

(+1.00)(+1.00) +1.00 means that the two numbers are+1.00 means that the two numbers are perfectly perfectly

positively correlatedpositively correlated

If If X X goes up thengoes up then Y Y also goes upalso goes up If If X X goes down thengoes down then Y Y also goes downalso goes down

If If X X stays the samestays the same Y Y also stays the samealso stays the same

We We NEVER NEVER see this in the real world.see this in the real world.



Positive correlationPositive correlation

> +0.00 to +1.00> +0.00 to +1.00

If If X X goes up thengoes up then Y Y goes upgoes up

Statistical significance is strongly influenced by Statistical significance is strongly influenced by sample size:sample size:

A sample of <30 is questionable when calculating A sample of <30 is questionable when calculating

correlationcorrelation



What is reliability What is reliability

Degree of dependability, consistency or stability Degree of dependability, consistency or stability

of scoresof scores

UnreliableUnreliable ² ² I got a 32 on the ACT

in May andI got a 32 on the ACT

in May and21 in June. Something is wrong.21 in June. Something is wrong.

ReliableReliable ² ² get a 32 in May and a 31 in June.get a 32 in May and a 31 in June.



What causes errors What causes errors

The test itself The test itself

The situation (hot room) The situation (hot room)

The person (sick, bad day) The person (sick, bad day) Scoring errorsScoring errors

Administrative errors in giving a test Administrative errors in giving a test



Estimating reliabilityEstimating reliability ² ² externalexternal

methodsmethods Test Test ² ² retestretest

Take the same test at time1 and time2 Take the same test at time1 and time2

Correlate the scores at t1 and t2Correlate the scores at t1 and t2

ProblemsProblems ² ² time consuming & learning time consuming & learning

Parallel or equivalent formsParallel or equivalent forms

Take forms A and B of same test Take forms A and B of same test

Correlate the scores of A & BCorrelate the scores of A & B

ProblemsProblems ² ² time consuming & equivalency of formstime consuming & equivalency of forms



Estimating reliabilityEstimating reliability ² ² internalinternal

methodsmethods SplitSplit--half (could divide test into 2 sets of half (could divide test into 2 sets of

questions, or people into 2 sets, then correlatequestions, or people into 2 sets, then correlate

Half1 with Half2Half1 with Half2

IssuesIssues ² ² need sufficient sample size, and must splitneed sufficient sample size, and must split

truly randomly, so that halves are equivalenttruly randomly, so that halves are equivalent



Estimating reliabilityEstimating reliability ² ² internalinternal

methodsmethods K uderK uder--Richardson & Cronbach·s AlphaRichardson & Cronbach·s Alpha

For different types of measuresFor different types of measures

Basically Basically ² ² they calculate the average of all possiblethey calculate the average of all possiblesplitsplit--halveshalves

Requires a statistical package (SPSS)Requires a statistical package (SPSS)



Estimating reliabilityEstimating reliability ² ²interraterinterrater

agreementagreement Compares the agreement between (for example)Compares the agreement between (for example)

3 interviewers3 interviewers

If we rate on a scale of 1 to 10, 3 raters canIf we rate on a scale of 1 to 10, 3 raters candiffer, but we should not see a 9, a 5 and a 2 fordiffer, but we should not see a 9, a 5 and a 2 for

the same candidate.the same candidate.



R eliability and validityR eliability and validity

If a selection device is reliableIf a selection device is reliable

It may or may not be validIt may or may not be valid

If a selection device isIf a selection device is notnot reliable itreliable it cannotcannot bebe valid valid



V alidity V alidity

DefinitionDefinition ² ² the degree to which availablethe degree to which available

evidence supports inferences made from scoresevidence supports inferences made from scores

on selection deviceson selection devices

Hi ACT scores predict success in collegeHi ACT scores predict success in college

Meaning Meaning ² ² does the device measure what it isdoes the device measure what it is

supposed to measure?supposed to measure?



V alidating selection measures V alidating selection measures

CriterionCriterion--related validity related validity

PredictivePredictive

Concurrent

Concurrent

Content validity Content validity

Construct validity Construct validity



CriterionCriterion--related validityrelated validity ² ²

predictive validity predictive validity Give a test to applicants when hired andGive a test to applicants when hired and

compare to performance on the jobcompare to performance on the job

Correlate test with performance


ProblemsProblems

Subjectivity of performance measuresSubjectivity of performance measures

T

ime lag from test to performance T

ime lag from test to performance



CriterionCriterion--related validityrelated validity ² ²

concurrent validityconcurrent validity Give a test to current employees whose jobGive a test to current employees whose job

performance is already knownperformance is already known



ProblemsProblems

Are current employees similar to applicants Are current employees similar to applicants



CriterionCriterion--related validityrelated validity ² ² To use To use

this the jobs must be:this the jobs must be: Reasonably stable over timeReasonably stable over time

Must have a bias free criterionMust have a bias free criterion

Must be a representative sampleMust be a representative sample Sample must be large enough to be stableSample must be large enough to be stable



Content validityContent validity

Based on the content of the jobBased on the content of the job

Must be representative of the content of the jobMust be representative of the content of the job

Needs a complete job analysisNeeds a complete job analysis It is a miniIt is a mini--job samplejob sample

Develop using Subject Matter Experts (SME·s)Develop using Subject Matter Experts (SME·s)



Construct validityConstruct validity

Measuring an abstract concept such as:Measuring an abstract concept such as:

intelligence or sociability intelligence or sociability

We may measure intelligence with an IQ test, We may measure intelligence with an IQ test,but does it really measure the constructbut does it really measure the construct

(intelligence)?(intelligence)?

This creates a complex situation to analyze This creates a complex situation to analyze ² ² letslets

leave it at that.leave it at that.



V alidity generalization V alidity generalization

Applying a validity study from one situation (job Applying a validity study from one situation (job

or company) to another.or company) to another.

Burden of proof for applicability is on the userBurden of proof for applicability is on the userof the selection device.of the selection device.



Is the selection device practicalIs the selection device practical

Refers generally to cost effectiveness of theRefers generally to cost effectiveness of the

measuremeasure

Depends greatly on the nature of the jobDepends greatly on the nature of the job



Is the selection device acceptableIs the selection device acceptable

to job applicantsto job applicants Less crucial in a ¶tight· job marketLess crucial in a ¶tight· job market

If I find a drug test insulting will I turn downIf I find a drug test insulting will I turn down

your job if I have a second offer that does notyour job if I have a second offer that does notrequire a drug test?require a drug test?



The End The End

What Makes a Selection System Work

Documents

Transcript of What Makes a Selection System Work