Social Movement Rhetoric and the Social Psychology of Collective Action: A Case Study of...

The Psychological Impact of Assessment in a

Development Center

Jan Francis-Sm yth e1,3 an d Philip M. Sm ith2

This paper extends previous work on the psychological impact on candidates of

se lection and asse ssme nt processes, presenting results from a study of 32

participants in an assessment center (AC) program of a major U.K. financial

services organization. Both quantitative and qualitative methodologies are used

to investigate : (a) the impact of the asse ssme nt outcome on cognitive and

affective reactions to the assessme nt process and candidates’ post-assessment

psychological states, job-related attitudes, and intentions, and (b) the role of

factors such as quality of feedback from the assessor, managerial support, and

care er line position on the impact process. Results show that the assessment

outcome decision does have an effect on post-assessment psychological states

and work-related attitudes but this effect works through a person’s perception

of the impact the assessment process will have on their career. This perception

of impact is well predicted by the candidate ’s perce ived quality of feedback and

current caree r position. Good feedback helps careers more than poor feedback

and the nearer the person’s current role is to the position for which they are

being assessed the more likely they are to benefit from the process.

KEY WORDS: impact validity, procedural justice, selection, assessment.

INTRODUCTION

Although early work in selection and assessment addressed candidate s’perceptions of the validity of the use of such technology, little attention

was give n to the psychological impact of these processes on candidate s until

the early 1980s.

Hum an Relations, Vol. 50, No. 2, 1997

149

0018¯7267/97/0200 ¯0149$12.50/1 Ó 1997 The Tavistock Institute

1Worce ster College of Highe r Education, He nwick Grove, Worceste r WR2 6AJ, UK.2Opus Consulting, Edgecombe Hall, Richmond Hill, Clifton, Bristol, BS8 1AT, UK3Requests for reprints should be sent to Jan Francis-Smythe, Worcester College of Higher

Education, Henwick Grove, Worcester WR2 6AJ, UK.

In 1982, Thornton and Byham sugge sted researchers should assess not

only candidate s’ ove rall perceptions of selection processes but also their

effects on various aspe cts of their psyche such as se lf-esteem, motivation,

and job satisfaction. At a more theoretical leve l, Dreher and Sacke tt (1983)

suggested the selection or assessment decision produce s first a cognitive

response , followed by an affective response and finally a behavioral re-

sponse (Fig. 1). In the case of a selection rejection or a fail at assessment

then the cognitive response might be : a) a belief that rejection is due to

the person not posse ssing the skills or attribute s needed for the job (an

inte rnal attribution) , or b) a belief that rejection is due to a decision error

on the part of the assessors (an external attribution) . Possible affective re-

sponse s are re duce d se lf-e ste e m and job motivatio n, and be haviora l

responses might be seeking additional training or education, seeking other

jobs not requiring the same skills or even a total withdrawal from the work-

force.

Empirical work on the impact of se lection processes on candidate s has

been carried out since the late 1980s. For example , Noe and Ste ffy (1987)

showed that reactions to assessment centers (ACs) were significantly re-

late d to job involve ment and career exploration behavior 1 year after the

AC. Iles and Robertson (1989) showed candidate s expe rienced no signifi-

cant change in organizational commitment and job involve ment, but were

more like ly to be thinking of changing the ir careers and having less-definite

career goals after attending an AC. Bray, Campbe ll, and Grant (1974; cited

in Noe & Ste ffy, 1987) showed that individuals who received less-favorable

evaluations after an AC reported less career salience and less upward mo-

bility motivation as measured by proje ctive tests than people who received

favorable evaluations. More recently, Smithe r, Reilly, Millsap, Pearlman,

and Stoffe y (1993) demonstrated a re lationship between perceptions of fair-

ness of the selection process and the recommendations applicants make to

the ir friends regarding the company. This alternative perspective of selec-

tion (i.e ., the extent to which a measuring instrument, or process such as

selection, has an effect on a subje ct’s psychological characteristics) has been

referred to as ‘‘impact validity’’ (Robertson & Smith, 1989), the social side

of se lection (Herriott, 1989) , and social validity (Schule r, 1993) .

One of the first important studies (Fletcher, 1991) monitored the long-

term e ffe cts of an AC by taking me asure s of nee d for achie ve ment,

Fig. 1. The basic impact model proposed by Dre her and Sackett (1983).

150 Fran cis-Sm ythe an d Smith

organizational commitment, job involve ment, and psychological well-be ing

before , immediate ly after and 6 months after the assessment. The results

showed that the time of measurement of the psychological characteristics

was crucial; in general, self-esteem and job involve ment rose immediate ly

after the center and then fe ll sometime later. There were, however, differ-

ences between successful and unsucce ssful candidate s. Those who were

unsucce ssful exhibite d significant reductions in self-esteem, the work ethic

component of need for achie vement and job involve ment, 6 months after

the center.

While this study had monitored the affective and behavioral response

to the assessment, Robertson, Iles, Gratton, and Sharple y (1991) were the

first to attempt to monitor the cognitive response originally propose d by

Dreher et al. (1983). They provide d evidence to sugge st that beliefs about

the ade quacy of the assessment process and its perceived impact on their

career acted as mediators in the impact process (i.e ., the effects were not

produced directly as a result of the assessment process but they were pro-

duced as a result of two diffe rent cognitive response s to the process: (a)

how ade quate the person be lieved the assessment process to be, and (b)

how much the outcome decision would have an effect on their career). For

example , if a person was rejected for a new position but be lieved the psy-

chometric tests used for the selection were inappropriate , then the impact

effect would be less than if they had be lieved the tests to be highly appro-

priate .

More recent work in organizational justice theory (e.g., Greenbe rg,

1990) propose s the use of the term procedural justice for the applicant ’sperceived fairne ss of the selection process. Gilliland (1993) identified ten

procedural rule s re lated to formal characteristics of the selection process,

information offered during the process, and inte rpersonal treatment, which

combined (in varying weights according to the circumstance s) to form an

ove rall evaluation of procedural justice . The cognitive response ‘‘belief

about the adequacy of the process’’ propose d by Robertson, Iles, Gratton,

and Sharple y (1991) , and used in this study, is a composite measure of

procedural justice .

Robertson et al. (1991) made comparisons between ‘‘pass’’ and ‘‘fail’’groups on psychological health, se lf-esteem, organizational commitment,

and career withdrawal cognitions. In one type of AC studied, there were

significant differences in organizational commitment and desire to leave a

career between ‘‘passes’’ and ‘‘fails.’’ It is, however, impossible to attribute

the cause of these diffe rences to the AC process itse lf without pre-assess-

ment measure s for comparison. Gilliland (1994) also provide s evidence that

selection procedure justice impacts on both self-efficiacy and work pe rform-

ance of hired employees.

Psych ological Im pact of Assessmen t 151

Other researchers have propose d that factors such as social support

and context (Robertson & Smith, 1989) , feedback and supervisory interest

(Ilgen, Fisher, & Taylor, 1979; Robertson et al., 1991) , and career line dis-

tance (Scholl, 1983) might act as moderators in the impact process. For

example , quality of feedback might act as a moderator between the effect

of the outcome decision and a candidate ’s self-esteem. A rejected candidate

with poor feedback might suffer a lowering of se lf-esteem, whereas a re-

jected candidate with good feedback might expe rience no reduction in

self-esteem. To date, as far as the present researchers are aware, none of

these moderators have been supporte d empirically, although timely feed-

back has been shown to enhance belief in the ade quacy of the selection

process (see Bies & Moag, 1986).

This study extends earlie r work on ‘‘impact validity’’ by using both

qualitative and quantitative methods to explore the effects of an assessment

process on middle managers in a large U.K. financial services organization.

The objective s were twofold: first, to establish more firmly the causal role

of the cognitive response variable s (be lie f about adequacy and perceived

career impact) in the impact process through incorporation of ‘‘pre’’ meas-

ures in the research design, and second, to explore the possible role of

moderators (quality of feedback after assessment, extent of managerial sup-

port, and current career position) in the impact process as depicted in Fig.

2. This give s rise to two specific hypothe ses:

Hypothesis 1. Beliefs about the ade quacy of the process and its per-

ce ived care e r impact will me diate the effe ct of outcome decision on

candidate s’ se lf-esteem, organizational commitment, job involve ment, and

career planning.

Hypothesis 2. Quality of feedback after assessment, extent of manage -

rial support, and current career position will moderate the relationship

between outcome decision and self-esteem, organizational commitment, job

involve ment, and career planning.

Fig. 2. Model of psychological impact explored in this study.


METHOD

Sam ple

The study was carried out in a large U.K. financial services organiza-

tion. At the time, 167 pe ople had atte nde d the Company ‘‘in-house ’’Assessment Center. Thirty-e ight attende es were selected as participants for

the study having satisfie d the following criteria: (a) attendee still worked

for the organization, (b) attende e was male (in acknowle dgment of possible

gende r effects (Alban-Me tcalfe, 1991) , (c) attende e had a male manager

(in acknowle dgment of possible sex bias effects), (d) attende e’s line man-

age r had between two and four subordinate s who had attended an AC

(management support is like ly to vary with numbe r of subordinate s attend-

ing an AC), and (e) the final sample to include participants drawn from

at least six diffe rent ACs (to eliminate bias from any one AC). Thirty-four

of the participants agreed to take part in the study although non-availablity

of two reduced the sample size to 32.

Procedu re

Pilot measures were trialled on one participant and three people in

middle /senior management positions in other organizations through role-

play scenarios. Each of the study participants were seen at their place of

work approximate ly 6 months after the AC. They first took part in a struc-

tured inte rview with the first author in which the ir perception of perceived

impact on their career, the adequacy of the assessment process, the quality

of the assessor’s feedback, and the degree of managerial support they re-

ceived were assessed. For each of these, candidate s were first asked to an-

swer pre-defined questions on a numerical rating scale and then encourage d

to talk freely about the topic in their own words. The researcher recorded

in writing, at the time of the interview, the responses to all questions. Each

interview laste d approximate ly 45 minute s. Transcripts of the interviews were

analyze d for participant statements which might serve to provide rich illus-

trations, either in support or refute of a particular quantitative finding or

from an alte rnative , previously-unconside red perspective , on the issues under

question. The researcher remaine d in the room while the candidate then

comple ted a bookle t of questionnaire measures (self-esteem, organizational

commitment, job involve ment, career planning, change in organizational

commitment (pre-AC-post-AC), change in job involve ment (pre-AC-post-

AC), and change in career planning (pre-AC-post-AC)).


Measures

For full description of measure s see Appendix A.

Assessment outcom e (OUTCOME) was assessed on a 4-point scale :

A= Excellent, B= Good (meets standard) , C= Just Misse s Standard, and

D= Substantially Misse s Standard. This represented the final assessment

rating give n after conside ration of a participant ’s performance across six

exercises on six dimensions.

Participant ’s current career position (POSITION) represents the num-

ber of grade points he was be low the position for which he was be ing

assessed at the time of the AC.

Managerial support (SUPPO RT) was assessed as the degree of support

the participant believed he had from his current line manager with respect

to his job-re late d personal deve lopme nt.

Quality of feedback (FEEDBACK) was measure d as quality of feed-

back give n to the participan t by the assessor in terms of its perceived

usefulne ss.

The following previously-de veloped scales were used: (1) the partici-

pant’s beliefs about the adequ acy of the AC process (ADEQUACY) and

perceptions of the impact (IMPACT) the AC had had on his career (Robert-

son e t a l., 1991) ; ( 2) self-esteem (ESTEEM) (Rosenbe rg, 1965) ; (3)

organization al commitment (COMMITMENT) (Mowday, 1979) ; (4) job in-

volvement (INVOLVEMENT) (Lodahl, 1965) ; (5) The extent of future

career plan ning (PLANNING) (Gould, 1979) .

As an alternative to longitudinal studie s with self-report measure s,

Howard, Ralph, Gulanick, Maxwe ll, Nance , and Gerber (1979) propose d

the use of retrospective pretests. Five studies conducte d by Howard et al.

(1979) lend support to the fact that when actual pre¯post self-report mea-

sures are used, the results may well be confounde d by a response shift;

that is, the subje ct’s perception of the response scale would stretch to fit

the expe rience continuum. In the retrospe ctive pre¯post test, subje cts are

asked to rate themselve s after the inte rvention on both a posttest and a

retrospe ctive pretest (how they remember themselve s before the inte rven-

tion) . Postte st and retrospe ctive pretest diffe rence scores can then be

compared. Much research (e.g., Hoogstrate n, 1985; Pohl, 1982; Sprange rs

& Hoogstate n, 1989; Gutek & Winte r, 1992) has found differences of at-

titudes computed in this way to be more valid than those between actual

pre- and post-measures. Terborg and Davis (1982) found the retrospe ctive

technique was robust to proble ms of saliency, priming, and order effects

and the retrospective pretest ratings were not biased by memory distortion

or social desirability. Gutek and Winter (1992) recommend the inclusion

of both retrospe ctive and current evaluations in studies inve stigating atti-


tudinal consistency as a valid method of assessing and controlling for re-

sponse shift bias. The retrospective pretest method was utilized in this study

to measure change ( D ) (pre-AC¯post-AC) in organization al commitment, job

involvem ent, and extent of future career plann ing ( D COMMITMENT, D IN-

VOLVEMENT, D PLANNING).

RESULTS

Table I shows the means, standard deviations, and ranges of all the

variable s for the whole sample and separate outcome groups (N.B., no sub-

ject achieved Outcome A) and Table II shows the inte rcorrelations between

the variable s in the study.

Outcome groups differed significantly on IMPACT (F(2,29) = 4.7, p

< 0.05) ; participants with ove rall assessments suggesting ‘‘Good’’ or ‘‘Just

Misses Standard’’ (B or C) felt that the AC had a positive impact on their

career, whereas those in the group ‘‘Substantially Misse s Standard’’ (D)

perceived it to have had a negative impact. These quantitative differences

were supporte d by participants ’ comments at inte rview (eg. ‘‘it helped boost

my career to my current position of . . . much quicker,’’ Group B—Positive

comment; ‘‘they say the y only ke ep it for 18 months but I’m sure if I applie d

for a management job they’d look at it,’’ Group D—Negative comment).

For others, the effect of outcome decision appe ared less strong than the

quantitative results suggest and othe r variable s such as time and pre-AC

perceptions appeared to be important moderators (e.g., ‘‘A few months

Table I. Descriptive Statistics of Variables Involved in Study

Variable

Whole sample

(Bs, Cs, and Ds)

n = 32

Outcome group

B

n = 7

C

n = 18

D

n = 7

Mean SD Range Mean SD Mean SD Mean SD

Adequacy 2.88 0.86 1.3¯5 3.08 1.08 2.83 0.82 2.83 0.85

Impact 3.41 0.98 1.5¯5 3.86 1.18 3.58 0.71 2.53 0.98*

Support 2.82 1.25 1.0¯5 3.95 0.89 2.72 1.11 1.93 1.26*

Feedback 2.28 0.55 1.2¯4 2.50 0.90 2.28 0.42 2.06 0.49

Position 1.70 0.71 1¯4 1.57 0.54 1.80 0.77 1.57 0.79

Estee m 5.06 1.49 1¯6 4.71 1.25 5.39 1.29 5.14 1.57

Planning 2.30 0.72 1¯3.5 2.36 0.71 2.23 0.65 2.27 0.95

D Planning 0.94 1.58 2¯7 1.27 2.21 1.28 1.27 ¯0.01 1.34

Involvement 45.75 6.10 33¯58 43.43 5.44 46.22 6.20 45.14 5.27

D Involvement 0.73 1.48 ¯3¯4 1.00 2.00 0.78 1.21 0.43 1.81

Commitment 69.39 13.16 38¯97 68.00 19.28 69.44 12.94 71.14 8.31

D Commitme nt ¯0.10 1.87 ¯5¯5 0.27 2.98 ¯0.10 1.13 ¯0.43 2.44

*Indicates difference between outcome groups significant at alpha £ 0.05.


Ta

ble

II.

Co

rre

lati

on

s A

mo

ng

Po

ten

tia

l M

ed

iati

ng

an

d M

od

era

tin

g V

ari

ab

les

an

d O

utc

om

e M

ea

sure

s

Outcome

Adequacy

Impact

Support

Feedback

Position

Esteem

Planning

DPlanning

Involvement

DInvolvement

Commitment

Ou

tco

me

Ad

eq

ua

cy

¯0.1

0

Imp

act

¯0.4

5*

0.4

4*

Su

pp

ort

0.5

3*

0.0

70

.43

*

Fe

ed

ba

ck¯0

.24

0.5

7*

0.5

3*

0.2

4

Po

siti

on

0.0

00

.09

0.3

5*

0.1

90

.17

Est

ee

m0

.11

0.1

5¯0

.04

0.0

90

.16

¯0.2

3

Pla

nn

ing

¯0.0

40

.11

¯0.1

6¯0

.48

*¯0

.19

0.1

7¯0

.31

D Pla

nn

ing

¯0.3

00

.40

0.4

0*

0.3

6*

0.6

3*

0.4

0*

¯0.0

6¯0

.24

Invo

lve

me

nt

0.1

0¯0

.09

¯0.3

0¯0

.43

*¯0

.13

¯0.2

40

.07

0.4

5*

¯0.2

4

D In

vo

lve

me

nt

¯0.1

30

.00

0.4

1*

0.1

80

.33

0.3

6*

0.0

1¯0

.41

*0

.47

*¯0

.36

*

Co

mm

itm

en

t0

.08

0.1

10

.34

*0

.48

*0

.12

0.2

90

.08

¯0.5

9*

0.2

9¯0

.58

*0

.33

D Co

mm

itm

en

t¯0

.13

0.3

9*

0.5

1*

0.3

30

.50

*0

.58

*¯0

.21

¯0.2

30

.49

*¯0

.18

0.4

9*

0.4

5*

*p

£ 0

.05

. ¯

ago, straight after the feedback I would have felt that the AC could have

pre judiced my career—now I’m not so sure ’’ and ‘‘some people see them

as a threat—if I mess it up it could harm me’’).

Outcome groups also diffe red on SUPPO RT (F(2,29) = 6.0, p < 0.05) ;

the better the AC performance, the greater the perceived managerial sup-

port. Again, these quantitative differences were supporte d by participants ’comments at inte rview (e.g., ‘‘I have a feedback session fortnightly with

respect to proje cts and needs and get a lot of instant feedback—I definite ly

feel one needs a manage r working with you 50/50 on your action plan,’’Group B—Positive comment; ‘‘my manage r actually phone d me up in the

middle of the assessment centre!’’ Group D—Negative comment).

Interview transcripts were checked for concordance with quantitative

findings. One major anomaly existed: although self-esteem did not differ

significantly between outcome groups when measure d approximate ly 6

months after the AC, comments made at inte rview suggest that immediate ly

after the AC this may not have been the case (e.g., ‘‘my own expe rience is

that it’s not stressful—post-AC blues is nonsense,’’ Group B—Positive ; ‘‘you

go with the ‘course mentality’ and then come away with a deflated ego—there’s the potential to undermine one’s confide nce’’; ‘‘I felt totally de-

stroyed by the report as did several other people ’’; ‘‘I felt very negative after

reading my report and thought it’s blown all my credibility’’; ‘‘the written

report was most demoralizing,’’ Group D—Negative ). To what extent the

other variable s may also be affected by time is unknown and the results

should therefore be seen in this context—as a ‘‘snapshot’’ in time. Further

research using a longitudinal design is required to explore this issue.

Robertson et al. (1991) suggest that IMPACT acts as a mediator in

the impact process, i.e ., the outcome decision affects the participant ’s per-

ception of the impact the process has had on his career which then in turn

affects his reactions to the process (i.e ., there will be no direct effect of

outcome decision on participants ’ reactions) . The patte rn of correlations

obtaine d in this study supports the idea that reactions to the outcome of

the AC are mediated by beliefs about its career impact. OUTCOME is

significantly related to IMPACT (r = ¯0.45, p < 0.05) and IMPACT is in

turn significantly relate d to D COMMITMENT (r = ¯0.51, p < 0.01), D IN-

VOLVEMENT (r = ¯0.41, p < 0.05) , and D PLANNING (r = ¯0.40, p <

0.05) . In order to examine further the pote ntial mediating role of the cog-

nitive re sponse (IMPACT), multiple re gression analysis was used, as

propose d by James and Brett (1984) . This analysis confirmed the role of

perceived career impact as a mediator between the outcome decision and

subse quent affective responses thus providing partial support for Hypothe -

sis 1. For full details of this analysis, see Appendix B.


It was propose d by Robertson et al. (1991) that belief about ade quacy

might also act as a mediator in the impact process, i.e ., that an outcome

decision might evoke a cognitive response in terms of a belief about the

adequacy of the process and this would then in turn effect the participants ’affective reactions to the process. However, it was also noted that although,

in the Robertson et al. (1991) study, be liefs about ade quacy diffe red between

pass and fail groups in some of the ACs; in othe rs, which appeared to be

better designed, no differences existed. This suggests then that while ADE-

QUACY may be determined by outcome decision and hence act as a me-

diator in the impact process, it is perhaps more like ly that it is determined

by the procedure s used and hence does not act as a mediator. The results

of this study show that OUTCOME is not significantly related to ADE-

QUACY, and hence that ADEQUACY does not act as a mediator in the

impact process. Hypothesis 1 therefore remains only partially supported.

It is, however, conceivable that ADEQUACY might act as a modera-

tor in the impact process; typically, the e ffect of a ne gative outcome

decision on a participant ’s perceived career impact might vary, depende nt

on how adequate the participant perceives the process to be. If he thinks

the AC was a good one and he doe s badly at it then he is like ly to perceive

the AC as having a negative impact on his career, whereas if he thinks the

AC was a poor one and he does badly then he might perceive it to have

less of a negative impact on his career. While this hypothe sis was not sup-

porte d by statistical analysis, inte rview transcripts suggest it may well be

the case . Typically, one candidate who did poorly at the center (Outcome

D) thought the center was inade quate ‘‘too much onus is be ing put on

it—it’s not the be all and end all’’ and perceived it to have had very little

effect on his career: ‘‘I’ve actually been promote d to management in the

last 2 months despite the findings of the report! ’’ Interestingly, those who

did well perceived the center to be good (e.g., ‘‘it was an excellent test of

one ’s ability to act under pressure and time constraints—very much related

to the ‘‘real world’’ and it has he lped boost my care er to my current position

of ... much quicker’’). Several candidate s believed the assessment only pro-

vide d a small sample of behavior for assessors to monitor (e .g., ‘‘it’s a

snapshot’’; ‘‘picks up some things but doesn’t account for a bad day—tooth-

ache and so on’’; ‘‘if it’s not seen it’s not there!’’). Others be lieved it was

selecting people on the basis of the wrong criteria (e .g., ‘‘the AC is geared

toward extroverts, it funne ls out too many of the quieter type s—I be lieve

it’s funne ling more of the wrong type into management positions’’; ‘‘I think

anyone from my department would have been give n the same deve lopme nt

needs—we all have the same personality profile —all done the same de-

gree—all introve rts prepare d to work hard for exams—all the exercises

were geared to extrove rt sale speople ’’; ‘‘technical qualitie s are not given


enough importance ’’). Still others believed its ade quacy was very much de-

pendent on the individual concerned (e .g., ‘‘the center is as objective as

possible in a classroom environme nt but it’s an unfair way—it’s not suitable

or applicable to all individuals—it often brings out the worst in people—you’re in a false environme nt’’).

Regression analysis was used to test hypothe sis 2 and, hence, explore

the role of the pote ntial moderators (FEEDBACK, SUPPORT, POSI-

TION) of the relation between OUTCOME and IMPACT. The hypothe sis

was not supporte d as no statistically significant moderating effects were

found. For a full description of the analysis, see Appendix B.

Comments made at interview, however, supporte d the important roles

of FEEDBACK and POSITION in the impact process; those with good-

quality feedback felt the AC had he lped their career more than those with

poor-quality feedback, and those who were furthe r away from a middle -

management position fe lt it had he lped the ir career more than those who

were only one position away from middle manage ment. Typical positive

comments about feedback were: ‘‘I felt more positive after my feedback

chat than anything else’’; ‘‘my personal feedback was exce llent’’; and ‘‘ifyou know yourse lf well enough then the feedback shouldn’t surprise you.’’Negative comments include d: ‘‘vague , ambiguous, and not very contextual’’;‘‘long-winde d and deflating’’; and ‘‘feedback could have been much better.’’

Transcripts were analyze d for the numbe r of qualitative comments

made on each topic area. Reference was made to the ade quacy of the proc-

ess by 35% of candidate s, to managerial support by 29% , to the effects on

psychological state by 23% , and to the perceived impact on the ir career

and quality of feedback expe rience by 18% . Thus, while most comments

were made about the adequacy of the process, the least were made about

personal experience. It may be that this represents the relative importance

of these issues for candidate s or it may be that they simply fe lt more able

to expre ss their opinions on something tangible like the ade quacy of the

process than to discuss the ir own personal expe riences. This is an important

methodological point to note and supports the use of structure d closed

que stions when exploring areas in which some subje cts may feel unable to

expre ss themselves freely.

In summary, the results show that the perceived impact of the AC

process on one ’s career mediates between the effects of the AC outcome

decision and post-asse ssment states and work-relate d attitude s (partial sup-

port for H1), and that also the perceived quality of feedback and career

line distance are important predictors of perceived career impact. Hypothe -

sis 2 which states that they act as moderators was not supporte d. Good

feedback he lps attende es’ careers more than poor feedback and those at-

tende es who are 3 or 4 grade s below middle management are like ly to


benefit more than those who are just 1 grade be low. A revised mode l of

psychological impact is propose d in Fig. 3.

DISCUSSION

This study provide s further insight into the dynamics of the psycho-

logical impact of assessment processes at work. Notably, be lie fs about the

ade quacy of the process do not mediate othe r responses in this study. This

is contrary to the findings of Robertson et al. (1991) and may be, as dis-

cusse d e arlie r, due to the fact that its role is more appropriate as a

moderator. Alternative ly, it may be due to the fact that the present study

was conducte d in more of a developmental context (where candidate s were

being selected for promotion but were also given guidance as to their de-

velopment needs) rather than a pure assessment context as in Robertson

et al.’s. Overall, one might expe ct the poignance of the assessment process

to be greater in selection/promotion contexts than in strictly deve lopme ntal

one s, thus entraining more comple x psychological processes and stronger

associations between selection decisions and response s. This hypothe sis is

furthe r supporte d by the Gilliland (1994) study where in se lecting stude nts

for temporary work, selected individuals saw greater fairness in the selec-

tion proce ss than re je cte d individ uals. Pe rhaps individuals se e king

promotion with the ir current employe r are more like ly to be obje ctive about

the ade quacy of the proce ss (and thus belief about ade quacy would be more

determined by the procedures used) than those seeking a job in a new

organization. The suggestion here then is that the conte xt of the asse ssment

may well moderate procedural justice effects. Typically, selection for a job

in a new organization may show greatest justice effects, followed by selec-

tion for promotion within an organization, followed by assessment for pure

developmental reasons. While previous work (see Gilliland, 1994) has con-

sidered procedural justice effects in different selection methods, the effect

of context of the selection/asse ssment appears to remain a matter for em-

pirical clarification.

Fig. 3. Revise d model of psychological impact.


While no evidence was obtaine d for the role of FEEDBACK, POSI-

TIO N, and SUPPORT as mode rator variable s in the impact proce ss,

FEEDBACK and POSITION were shown to be important variable s in the

prediction of IMPACT, while SUPPORT and ADEQUACY were not.

Closer examination of the data, however, revealed that SUPPO RT was sig-

nificantly and negative ly relate d to OUTCOME (r = ¯0.53, df = 30, p <

0.05) . There is thus a significant difference in managerial support between

OUTCOME groups; participants who have good managerial support are

more like ly to do well at the AC. The possibility of a causal re lation be-

tween these variable s needs to be explore d. It may simply be that people

who do well at the AC have more positive outlooks on their organizational

environme nts, and that this is reflected in their ratings of manage rial sup-

port. However, it is also possible that the measures of SUPPO RT taken

here indicate real diffe rences in the level of inte rest and involve ment shown

by managers for their subordinate s, and that this in turn is re late d to suc-

cess in assessment/de ve lopme nt contexts.

There are several methodological refinements that could be suggested

in further studies of this topic. First, notwithstanding the argume nts in favor

of retrospective pre¯post tests of attitude change in some areas, it would

be beneficial to gathe r genuine pre - and post-asse ssment measure s on many

of the variable s used in this study. Additionally, given that many of the

measures are derived through self-report, the issue of common method

variance rathe r than causal links explaining some of the observed relation-

ships should be raise d. In response , both inte rvie w and que stionnaire

technique s were employe d, and well-de ve lope d measure s with sound psy-

chometric prope rties were used, factors that should help to dispe l concerns

about the role of common method variance .

The size of the present sample is also small, and the generalizability

of these findings consequently limited. However, give n the limite d number

of employe es that had undertake n the AC and the limited number of these

who satisfied the selection criteria, it was decided at the outset that the

proble ms imposed by the small sample size were ade quate ly compensated

for by the exploratory nature of the study and the need for more qualitative

data. To the extent that our earlier conjectures about the relative ly muted

poignancy of assessment in a developmental as oppose d to a selection/pro-

motion context are valid, then the emergence of any statistically significant

results here is all the more impressive, especially give n the small number

of people studie d.

Although the effects of gende r were strictly controlle d in this study,

future research in this area should explicitly addre ss the possibility of sex-

relate d effects if possible . Candidate s’ gende r could possibly affect outcome

decisions made by assessors, the reactions of candidate s to those decisions,


resultant effects on the candidate s, and the moderating effects of manage -

rial support and feedback.

The implicit message s for organizations engage d in or contemplating

using assessment or deve lopme nt center technology is clear: be aware that

the potential gains to individuals of assessment/de ve lopme nt opportunitie s

may be dependent on their career positions and ensure that high-quality

feedback is available to all. In order to minimize the potentially-ne gative

impact of such procedure s, careful feedback, counse ling, and deve lopme nt

recommendations are of vital importance . We should be seeking as best

we can to understand the ways in which assessment procedure s themselves

can enhance or unde rmine broade r organizational obje ctive s and individual

motivation.

ACKNOWLEDGMENTS

The authors wish to thank Professor Ivan Robertson for helpful com-

ments on an earlier version of this pape r.

APPENDIX A—DETAILS OF MEASURES

OUTCOME. Asse sse d on a 4-point scale : A= Excelle nt, B= Good

(mee ts standard) , C= Just Misse s Standard, or D= Substantial ly Misse s

Standard. This represented the final assessment rating give n after consid-

eration of a participant ’s performance across six exercises (presenting a

proposal, constructing a plan, running a hote l, in-tray exercise , developing

a work unit, and preparing and presenting a board report) . Each exercise

was assessed on each of six dimensions (energy and drive , results orienta-

tion, workstyle , analytical ability, emotional adjustment, and interpersonal

skills) using Grades A+ (high) through to D¯(low). A matrix of marks

across all exercises and dimensions was produced for each participant. The

final assessment rating was a subje ctive inte rpretation of ove rall perform-

ance based on this matrix of scores.

POSITION. Number of participant positions is away from the position

being assessed (maximum = 4, minimum = 1, mean = 1.7) .

SUPPORT. Three aspects of support were rated by participants in the

inte rview on 5-point scale s (5 = good support) : the participant ’s perception

of the extent of guidance give n, the extent to which an action plan was

constructed, and the extent to which the manager had facilitate d opportu-

n i t i e s for t he pa rt ic ip an t t o e ngage in oth e r work e xp e rie nc e .

Intercorre lations between the three aspects of manage ment support were

all significant at approximate ly r = 0.8 (df = 30, p < 0.05). The ir ave rage

was take n to give the measure of SUPPORT.


FEEDBACK. Quality of feedback give n to the participant by the as-

sessor in terms of its perceived usefulness. The measure was take n as the

ave rage of five aspects of assessor feedback rated during the inte rview: the

extent of guidance give n by the assessor in formulating an action plan, the

availability of relevant information in the AC report to formulate a plan,

the extent to which the feedback provide d a full synopsis of the participant ’sperformance , the extent to which it gave guidance as to how to develop

skills the participant already had, and those skills he was lacking. Items

were rated on 5-point scale s (5 = good feedback) . Item inte rcorrelations

were moderate but significant at approximate ly r = 0.5 (df = 30, p < 0.05)

and their average was take n to give the measure of FEEDBACK.

ADEQUACY. As per Robertson et al. (1991) through three items: ‘‘itaccurate ly measures important qualitie s,’’ ‘‘it is a fair way of assessing po-

tential,’’ and ‘‘it should be used in selection,’’ on 5-point scales ranging

from Strongly Disagree (1) to Strongly Agree (5). With a sample of 140

employe es going through a management development and tiering scheme

in a major U.K. company, the mean score previously obtaine d was 3.19

with SD = 1.08 and reliability (coefficient a ) of 0.86 (Robertson et al.,

1991) .

IMPACT. As per Robertson et al. (1991) through two items: ‘‘it has

helped my career,’’ and ‘‘taking it has not harmed my career,’’ again meas-

ure d on 5-point scale s (5 = positive impact) . With a sample of 140

employe es going through a management development and tiering scheme

in a major U.K. company, the mean score obtaine d was 3.39 (SD = 1.17)

and a reliability (coefficient a ) of 0.74 (Robertson et al., 1991).

ESTEEM. As per Rosenbe rg (1965) with a 10-ite m Guttman scale

(e.g., ‘‘On the whole , I am satisfie d with myself’’; ‘‘I take a positive attitude

toward myse lf’’) using a 4-point response scale from Strongly Disagree (1)

to Strongly Agree (4). Publishe d scale reliability is 0.92.

COMMITMENT. As per Mowday (1979) , through 15 items (e.g., ‘‘Iam proud to te ll othe rs I am part of this organization ’’) using a 7-point

scale (7 = high commitment). Publishe d reliability is 0.90 as measure d by

Cronbach’s alpha. Other studie s of manage ment trainees have found mean

value s of 4.7, 5.05, and 6.1 (Fletcher, 1991; Robertson et al., 1991; Mowday,

1979, respectively) .

INVOLVEMENT. As per Lodahl (1965) using a 20-ite m scale (e .g.,

‘‘I’ll stay overtime to finish a job, even if I’m not paid for it’’) with 4-point

response scale s ranging from Strongly Disagree (1) to Strongly Agree (4).

Publishe d split-half reliability coefficients are 0.8 and 0.67, respectively.

PLANNING. As per Gould (1979) using a six-ite m scale (e.g., ‘‘I have

a plan for my career’’) with a 5-point response scale ranging from Strongly


Agree (1) to Strongly Disagre e (5). High scores indicate a low degree of

planning. Reliability (coefficient a ) of 0.8.

D COMMITMENT, D INVOLVEMENT, D PLANNING . They were

measure d by asking participants first to allocate a retrospe ctive score from

1 (low) to 10 (high) to indicate how they fe lt before (pre) the AC on the

job-re lated constructs, and then to allocate a score for how they felt now

(post) on the same construct using the same scale. The pre score was then

deducted from the post score to give the change ( D ) score. Correlations

were calculate d between the post-measures of the subjective change meas-

ures of commitment, involve ment, and planning and the que stionnaire

measure s of commitment, involve ment, and planning. The questionnaire

measure s had been include d mainly to give participants some indication of

the constructs on which they would be asked to make retrospe ctive com-

ment. The corre lations between the change measures and their respective

que stionnaire measure s were r = 0.6 (p < 0.01) , r = 0.3 (p < 0.05) , r =

0.65 (p < 0.01) . This was take n to signify some degree of validity for the

change measures.

APPENDIX B—DETAILS OF STATISTICAL ANALYSES

To Test for IMPACT as a Mediator in the OUTCOME-Affective

Respon se Process

The analysis is base d on the assumption that when the cognitive re-

sponse variable is used as a predictor of affective reactions, the addition

of variable s earlie r in the chain (i.e ., outcome decision) should not improve

significantly the prediction of affective reactions (i.e ., a statistically signifi-

cant increase in R2). Three separate regression analyse s were conducted

using the post-asse ssment difference measure s as dependent variable s. IM-

PACT was entered followe d by OUTCOME. The addition of OUTCOME

to the regression mode l did not significantly increase the value of R2 for

any of the response variable s, thus confirming the role of perceived career

impact as a mediator between the outcome decision and subse quent affec-

tive response s (see Table BI).

To Test for Moderating Effects of FEEDBACK, SUPPORT, an d

POSITION in the Impact Process

According to James and Brett (1984) , a variable z is a moderator if

the relationship between two or more other variable s, say x and y, is a

function of the leve l of z, thus indicating an x by z inte raction. The roles

of the variable s O UTCOME, ADEQUACY, FEEDBACK, SUPPO RT,


and POSITION, as predictors of IMPACT, were inve stigated through step-

wise multiple regression. OUTCOME, FEEDBACK, and POSITION were

found to be the only statistically significant predictors giving an R2 of 0.47

(p < 0.05) (see Table BII).

In orde r to explore the role of FEEDBACK and POSITION as mod-

e rators in the im pact proc e ss , in te rac t ion te rm s ( O U TC O ME *FEEDBACK and OUTCOME * POSITION) were built into the regression

mode l after the single variable s had been entered. The interaction terms

were not significant thus sugge sting that FEEDBACK and POSITION were

not acting as moderators.

Table BI. Regression Analyses with Post-Assessment States and Work-Related Attitudes as

Dependent Variablesa

Independent variables

D Commitme nt D Involvement D Planning

R2

B p R2

B p R2

B p

1. Impact 0.80 0.02 0.77 0.01 0.65 0.02

2. Outcome

R2 before Outcome 0.29 0.20 0.40

R2 after Outcome 0.30 0.21 0.42

Increase in R2

0.01 0.01 0.02

aStepwise procedures were used in the regression analysis and independent variable s entered

in the regre ssion equation only if they explained a significant amount of variance in the

dependent variable .

Table BII. Multiple Regression Analysis Testing the Roles of

De cision O utcome , B e lie f About Ade quacy, Q uality of

Fee dback, Leve l of Managerial Support, and Current Career

Poisition in the Impact Proce ssa

Step

Variable

entered R R2

R2

change F value

1 Feedback 0.53 0.28 11.6*

2 Outcome 0.63 0.39 0.07 9.3**

3 Grade 0.69 0.47 0.08 7.3**

aStepwise procedures were used in the regre ssion analysis and

independent variables entered in the regression equation only

if they explained a significant (p < 0.05) amount of variance

in the dependent variable.

*p < 0.01.

**p < 0.001.


REFERENCES

ALBAN-ME TCALFE, B. Different Gende r—Different Rules? Assessment of Women in

Management. In P. Barre r and C. Cooper (Eds.), Managing organisations in 1992: Strategic

response. London: Routledge, 1991.

BIES, R. J., & MOAG, J. S. Interactional justice: Communications criteria of fairness. In R.

Le wicki, B . She ppard, and M. H. Baze rman (Eds.) , Research on negotiation s in

organisations . Gree nwich, CT: JAI Press, 1986.DREHER, G. F., & SACKETT, P. Perspectives on em ployee staffing and developm ent: Readings

and com m entary. Homewood, IL: Irwin, 1983.FLE TCHER , C. A. Candidates reactions to asse ssme nt centres and their outcomes: A

longitudinal study. Journal of Occupationa l Psychology, 1991, 64, 117-127.GILLILAND, S. W. The perceived fairness of selection systems: An organisational justice

perspective . Academ y of Managem ent Review, 1993, 18, 694-734.GILLILAND, S. W. Fairness from the applicant’s perspective: Reactions to employee selection

procedures. International Journal of Selection and Assessment, 1995, 3(1), 11-19.GILLILAND, S. W. Effects of procedural and distributive justice on reactions to a selection

system. Journal of Applied Psychology, 1994, 79(5), 691-701.GOULD, S. Characteristics of career planners in upwardly mobile occupations. Academ y of

Managem ent Journal , 1979, 22(3), 539-550.GUTEK, B. A., & WINTER, S. J. Consistency of job satisfaction across situations: Fact or

framing artifact? Journal of Vocational Behaviour, 1992, 41(1), 61-78.HERRIO TT, P. Selection as a social process. In M. Smith and I. Robertson (Eds.), Advances

in selection and assessm ent. Chichester: John Wiley and Sons, 1989.HO OGSTRATEN, J. Influence of objective measure s on se lf-reports in a re trospective

pretest-posttest design. Journal of Experimental Education, 1985, 53, 207-210.HOWARD, G. S., RALPH, K. M., GULANICK, N. A., MAXWELL, S. E., NANCE, D. W.,

& GERBER, S. K. Internal validity in pretest-posttest self-report evaluations and a

re-evaluation of re trospective pretests. Applied Psychological Measurem ent, 1979, 3(1),

1-23.

ILGEN, D. R., FISHER, C. D., & TAYLO R, S. M. Consequences of individual feedback on

behavior in organizations. Journal of Applied Psychology, 1979, 64, 349-371.

ILES, P. A., & RO BERTSO N, I. T. The impact of personnel se lection procedures on

candidates. In P. Herriot (Ed.), Assessment and selection in organisations. Chichester: John

Wiley and Sons, 1989.JAMES, L. P., & BRETT, J. M. Mediators, moderators and tests for mediation. Journal of

Applied Psychology, 1984, 69, 307-321.LO DAHL, T. M., & KEJNER, M. The definition and measure me nt of job involveme nt.

Journal of Applied Psychology, 1965, 49(1) , 24-33.MOWDAY, R. T., & STEERS, R. M. The measure me nt of organisational commitment.

Journal of Vocational Behavior, 1979, 14, 224-247.NOE, R. A., & STEFFY, B. D. The influence of individual characteristics and assessment

ce nter evaluation on care er exploration be havior and job involve me nt. Journ al of

Vocational Behavior, 1987, 30, 187-202.

POHL, N. F. Using retrospective pre -ratings to counteract response-shift confounding. Journal

of Experimental Education, 1982, 50, 211-214.

ROBERTSO N, I. T., & SMITH, J. M. Personnel selection me thods. In J. M. Smith and I.

T. Robertson (Eds.), Advances in selection and assessm ent. Chichester: John Wiley & Sons,

1989.ROBERTSO N, I., ILES, P. A., GRATTO N, L., & SHARPLEY, S. The impact of personnel

selection and assessme nt methods on candidates. Hum an Relations, 1991, 44(9), 963-982.ROSENBERG, M. Society and the adolescent self-image. Prince ton Unive rsity Press, 1965.

SCHULER, H. Social validity of selection situations: A concept and some empirical results.

In H. Schuler, J. L. Farr, and M. Smith (Eds.), Personn el selection an d assessm en t:

Individual and organisational perspectives. Hillsdale, NJ: Erlbaum., 1993.


SMITHER, J. W., REILLY, R. R., MILLSAP, R. E., PEARLMAN, K., & STOFFEY, R.

W. Applicant reactions to se lection procedures. Personnel Psychology, 1993, 46, 49-76.SPRANGERS, M., & HOOGSTRATEN, J. Pretesting effects in retrospective pre test¯posttest

design. Journal of Applied Psychology, 1989, 74(2), 265-272.TERBO RG, J. R., & DAVIS, G. A. Evaluation of a new method for assessing change to

planned job redesign as applied to Hackman and Oldham’s Job Characte ristics Model.Organisational Behaviour and Hum an Perform ance, 1982, 29, 112-128.

THO RNTO N, G. C., & BYHAM, W. C. Assessment cen ters and managerial perform ance. NewYork: Academic Press, 1982.

BIOGRAPHICAL NOTES

JAN FRANCIS-SMYTHE BSc (Hons), MSc is a Senior Lecturer in Psychology and BusinessManage ment at Worcester College of Higher Education. She is curre ntly carrying out research

for a PhD with Professor Ivan Robertson at UMIST into individual differences in attitudestoward time.

PHILIP M. SMITH BSc (Hons), PhD was, at the time the research was conducted, course

organizer for the MSc in Organisational Psychology at Bristol University. He now works forOpus Consulting in Bristol.


Social Movement Rhetoric and the Social Psychology of Collective Action: A Case Study of...

Documents

Transcript of Social Movement Rhetoric and the Social Psychology of Collective Action: A Case Study of...