A REPORT CRITICALLY EVALUATING ASSESSMENT DEVELOPMENT CENTRES

WITHIN THE THE FIRE AND RESCUE SERVICE

Janine Frankland 26 April 2010

20393518

CONTENTS PAGE

Page

1.0 Introduction

2.0 Background 1

3.0 THE FIRE AND RESCUE SERVICE Workforce Development Strategy 2

4.0 Developing Leadership/Workforce Skills and Capacity 2

5.0 Assessment Development Centres 4

5.1 Definition of an Assessment Development Centre

5.2 THE FIRE AND RESCUE SERVICE Assessment Development Centre

5.3 Structure and Context of a Typical ADC

5.4 ADC Scoring Process

5.5 Interview Process

5.6 Progression through Development to Promotion

6.0 Evaluation of Assessment Development Centres 8

7.0 Bibliography 9

Appendix 1

Appendix 2

Appendix 3

1.0 INTRODUCTION

The following report will examine the use of Assessment Development Centres as a

selection tool for promotion within THE THE FIRE AND RESCUE SERVICE and

evaluate their contribution to meeting the organisation’s strategic needs.

2.0 BACKGROUND

The role of the Fire Service has changed in line with the modernisation agenda

outlined in the June 2003 White Paper “Our The Fire and Rescue Service” (Office of

the Deputy Prime Minister) and the The Fire and Rescue Service National

Framework 2008-11 (Department of Communities and Local Government, 2008).

The The Fire and Rescue Service National Framework sets out clear objectives for

all The Fire and Rescue Services. UK Fire Services are now proactive in preventing

fires and other risks, rather than being solely reactive and responding to incidents.

There is evidence that the Fire Service is moving away from a transactional to a

transformational leadership; in 2004, the Fire Services (Discipline) Regulations 1985

were abolished. In its place was a move to introduce new disciplinary procedures

based on the Advisory Conciliation and Arbitration Services best practice guidance,

which was brought inline within the aforementioned The Fire and Rescue Service

National Framework. In the past there has been an era of ‘command and control’,

where officers issued orders to control their subordinates and any questioning of

those orders would be met with disciplinary action. The subordinates expected to be

commanded by their superiors.

In recent years, there has been a slight move away from this command and control

ethos. In 2005, there was a restructure of the management composition and the ten

ranks were reduced down to seven roles. The word ‘officer’ was replaced by the

word ‘manager’ (see Appendix 1). A change in the way THE FIRE AND RESCUE

SERVICE promotes and develop individuals has seen a move towards coaching

rather than commanding.

In July 2006, new government legislation made significant changes to the way The

Fire and Rescue Services fulfil their new and existing statutory duties and The Chief

Fire Officers Association launched a national HR strategy for the development of

people management in the UK The Fire and Rescue Service.

Page 1 of 12

3.0 THE FIRE AND RESCUE SERVICE WORKFORCE DEVELOPMENT STRATEGY

In 2009, THE FIRE AND RESCUE SERVICE carried out a review of its Human

Resources Strategy (2005) and introduced The Workforce Development Strategy

2009/11. Within this document, THE FIRE AND RESCUE SERVICE recognise that

their employees are critical to the delivery of services to the community and in

achieving the organisation’s corporate aim of making Lancashire safer. The Strategy

sets out how THE FIRE AND RESCUE SERVICE work to have the right people in

the right place, with appropriate skills, competence and motivation to achieve this.

THE FIRE AND RESCUE SERVICE strategic objectives are outlined in the Risk

Management Plan (RMP). The RMP sets out THE FIRE AND RESCUE SERVICE’s

strategic direction (Lancashire Fire and Rescue, 2008). Through this plan, THE FIRE

AND RESCUE SERVICE assess local risk and identifies how it can deploy its

resources and deliver improved public safety.

The Workforce Development Strategy works in conjunction with the RMP in order for

the organisation to achieve those objectives, and “ensure long term effective service

delivery in a flexible and responsive way” (2009, p.1).

THE FIRE AND RESCUE SERVICE list six key priorities within the Strategy (2009,

p.5), which are a reflection of the The Fire and Rescue Service CFOA National HR

Strategy (2006, p.9).

1. Developing the organisation

2. Developing leadership capacity

3. Developing workforce skills and capacity

4. Recruiting and retaining employees

5. Pay and rewards

6. Developing the capacity of Human Resources

4.0 DEVELOPING LEADERSHIP/WORKFORCE SKILLS AND CAPACITY

The individuals that work for THE FIRE AND RESCUE SERVICE provide operational

or service delivery support.

Page 2 of 12

To further expand on the key priority listed within THE FIRE AND RESCUE

SERVICE’s Workforce Development Strategy of developing leadership/workforce

skills and capacity, this report looks at the process that THE FIRE AND RESCUE

SERVICE use to promote and develop future managers within the operational sector.

The operational sector refers to individuals working in one of the seven roles from

Firefighter through to Brigade Manager (see Appendix 1).

THE THE FIRE AND RESCUE SERVICE use The Integrated Personal Development

System as a framework for how it develops its workforce.

Two firefighters lost their lives at an incident on Gillender Street, London in 1991 and

as a result a Health and Safety Improvement notice was issued to brigades requiring

the identification of performance standards and a way of measuring firefighters

against those standards. National Training Strategy Group (TSG) was established to

analyse the selection, training and development needs of the Service and as a result,

in 1995, the Integrated Personal Development System (IPDS) was developed (Skills

for Justice, 2009). This system has now been embedded in the The Fire and Rescue

Service National Framework 2008-11.

The Skills for Justice Code of Practice document explains how the IPDS provides a

structure based on agreed standards of performance, within which organisations can

identify, assess and develop people to fulfil current and future roles.

Integral to the IPDS is meeting the needs of the aforementioned Service’s Risk

Management Plan (Skills for Justice, p.6). IPDS supports the Service’s objectives of

making communities safer and through THE FIRE AND RESCUE SERVICE’s plan to

constantly reduce risk in the community, assists the organisation in making sure that

their employees have the latest skill to contribute to public safety.

The statutory fire service examinations were abolished in 2004 (Fire Services

Examination Board, nd) and in their place, the The Fire and Rescue Service National

Framework 2004/5 stated that The Fire and Rescue Services should “use the

nationally developed personal qualities and attributes for assessing the potential of

candidates for a new operational role” (p.38).

Page 3 of 12

In order to facilitate this, THE FIRE AND RESCUE SERVICE carries out Assessment

Development Centres.

5.0 ASSESSMENT DEVELOPMENT CENTRES

5.1 Definition of an Assessment Development Centre

An Assessment Development Centre (ADC) could easily conjure up an image of an

actual place or location but many authors (Ballantyne and Povah 2004, Woodruffe

2000, Hay 1997 to name a few) state the term refers an event or as described by

Chartered Occupational Psychologists, Liam Healy & Associates (nd) as an

‘approach’.

Appelbaum et al (1998, p.11) describes assessment centres and “development

assessment centres” as having a common objective, which is to observe how

individuals handle a series of situations that might occur, however they do differ in

focus. Ballantyne and Povah (2004, p.142) suggest that development centres have

evolved from assessment centres and attempt to give the reader an explanation of

the main differences between the two. They suggest that assessment centres are

used for external recruitment and internal promotion; therefore the assessment

centre is the end of the process. Development centres are used for identifying fast-

track potential and diagnosing job related strengths and weaknesses; therefore the

development centre is the start of the process.

5.2 THE FIRE AND RESCUE SERVICE Assessment Development Centre

Within THE FIRE AND RESCUE SERVICE, ADCs are carried out at three levels –

Supervisory, Middle and Strategic Management. Appendix 1 shows how the seven

roles fit into each managerial level. Individuals are to be competent in their current

role before they can apply for attend the next level ADC.

Page 4 of 12

The DCLG describe the ADCs as national tests that are designed to put individuals

under a great deal of pressure in the same way that any risk critical situation would

do (2009, p.11).

Individuals are measured against generic management skills, known as Personal

Qualities and Attributes (PQAs). These PQAs can include subjects such as working

with others, effective communication, commitment to change, openness to diversity

and integrity etc (see Appendix 2). Further information on how individuals are

measured and scored against the PQAs will be provided later in this report.

5.2 Structure and Context of a Typical ADC

The process developed within THE FIRE AND RESCUE SERVICE includes Stage 1

ADC, Stage 2 ADC and a PQA interview as explained within the organisation’s

Training Service Order (THE THE FIRE AND RESCUE SERVICE, 2008).

The Stage 1 ADC is formerly known as an ITOP which is an Initial Test of Potential. It

is a preliminary sifting exercise carried out under supervised conditions, which

examines an individual’s performance of the some of the PQAs identified as being

most crucial for success as a manager within THE FIRE AND RESCUE SERVICE.

The ITOP is divided into the following two sections - the Situational Judgement Test

(SJT) and a Simulation Exercise. The SJT is a multi choice test – it details a scenario

and then asks which of the following choices is likely to be effective. The simulation

exercise consists of three written tasks. The exercise is set in a fictitious

organisation that is non-fire service related. The exercises are usually formed

around decision making, numerical data and writing a letter/article aimed at a specific

audience (UK Fire Service Resources, nd).

The tests are designed to assess how an individual’s performance matches up to the

standards outlined in the PQAs. All individuals are given feedback on their

performance by their line managers. Successful candidates will be invited to attend

the Stage 2 ADC and unsuccessful individuals have a development plan created for

them to address their development needs.

During the Stage 2 ADC, individuals will participate in a series of exercises that

simulate the kind of tasks that they might be expected to do as a manager. The

Page 5 of 12

scenarios are non fire service related, for example, in past ADCs, individuals have

been informed that they are a manager in an airport or at a leisure centre.

When individuals are invited to an ADC by letter, they are also provided with

background information which will give them information about the organisation they

are working for, their position within the company, who they are responsible for etc.

The series of exercises are carried out in one day and involve activities such as a

group discussion, role play, multi role play, inbox exercise and briefing exercise.

Each exercise is designed to measure individuals against certain PQAs, however,

the candidates are not informed as to which PQAs they will be measured against.

The exercises are observed by a team of assessors, who are usually internal senior

managers who have been specifically trained to carry out the PQA scoring.

5.3 ADC Scoring Process

THE FIRE AND RESCUE SERVICE (2008, p.14) state that each PQA has been

mapped to particular exercises within the ADC so feedback on each PQA can be

given from these sources. Each PQA is divided into constructs and each of these

constructs has a Behaviourally Anchored Rating Scale (BARS).

After observing and recording the behaviour of an individual in one of the ADC

exercises, assessors use the exercise’s specific BARS to assess the effectiveness of

the behaviour. Within each construct, the individual’s performance is compared to

four levels and assigned the rating of the nearest match. The ratings of each

construct are then added together to provide a score for the PQA being assessed on

that exercise. For an example of how this is structured, Appendix 3 shows how the

PQA of Problem Solving is divided into constructs measured against BARS along

with a description for the levels of performance from 1 to 4. THE FIRE AND

RESCUE SERVICE (2008, p.14)

After the ADC, the assessors rate each individual’s performance on the PQAs across

the various exercises using a pre-determined scoring process. Individuals will then

be given an overall score derived from their scores on the PQA.

Page 6 of 12

5.4 Interview Process

The PQA interview has been integrated as part of the selection process, where

individuals are to provide evidence against the PQAs from either their work or from

other activities outside work.

The questions fit in to the ‘STAR’ model, where answers will be expected to fit into

Situation, Task, Action and Result (UK Fire Service Resources, nd).

The evidence provided at the interview may increase an individual’s overall PQA

score.

Only when a firefighter is deemed “successful” at both the ADC and subsequent

interview, can the individual be placed on a development programme and considered

for promotion.

5.5 Progression through Development to Promotion

THE FIRE AND RESCUE SERVICE’s Human Resources Department (HR) carry out

succession planning by determining the number of positions that will become

available within its workforce in order to determine the number of managers that will

be required in the future. This is usually determined by the number of individuals that

are due for retirement, as the firefighter pension scheme enables individuals who

have completed 30 years service and reached a minimum age of 50 to retire.

A moderation meeting is held between the HR and Training Development

Departments to discuss each individual’s scores and determine the number of

individuals that will be considered successful and therefore be placed on a

management development programme.

The ADC is not a pass/fail, which Ballantyne and Povah describe as a distinguishing

feature of a development centre. Individuals are either successful and unsuccessful.

Again, as with the ITOP stage, unsuccessful applicants will receive feedback from

their line manager and have a plan created to address their development needs

(THE FIRE AND RESCUE SERVICE, 2008).

6.0 EVALUATION OF THE ASSESSMENT DEVELOPMENT CENTRE

Page 7 of 12

There has been little written about the effectiveness of assessment development

centres, despite development centres existing since AT&T introduced them in the

1950s (Thornton and Rupp, 2006; Ballantyne and Povah, 2004).

However, Taylor (2008, p.306) describes assessment centres as ‘the Rolls Royce’ of

selection methods and reports that validity studies have consistently found

assessment centre techniques to have good predictive ability.

If THE FIRE AND RESCUE SERVICE were to use the ADC on its own, it could be

argued that this would not be sufficient to guarantee good operational managers

within the Fire Service. It is difficult to substantiate this statement as there is little or

no written evidence to suggest this, other than that endorsed by the DCLG in the

published Integrated Personal Development System Code of Practice (2009, p.11)

where they recommend that ADCs form part of the selection process and should be

not be considered as a standalone process. The DCLG recommend that other

Human Resource Management activities should be utilised.

ADCs provide organisations with an indication that individuals already display some

of the managerial qualities prior to investment of a development programme, which

will equip the individual with further managerial skills. However, it does not take into

consideration any previous qualifications, technical knowledge or practical work

related experience, unless it is specifically tied in to evidence for the interview.

Prior to the introduction of ADCs, individuals seeking promotion were required to sit a

written and practical examination. Individuals were required to be ‘time-served’ in

their current role prior to seeking promotion. In the traditional promotion interviews

held , there was a culture of ‘who you know’ rather than what you know or the ‘halo

and horns’ effect. Anderson and Shackleton (1993) cited by CIPD (2010) highlight

these judgements along with other limitations of the traditional interview, highlight

why it is a poor predictor of an individual’s performance in the job.

However, according to statistics provided by the Fire Services Examination Board,

there was a relatively low pass rate. For example, individuals taking the Leading

Firefighter and Sub Officer* examinations, between the years of 1987 and 2004, an

average of 40% passed the written paper and 60% passed the practical (Fire

Services Examination Board, 2006).

Page 8 of 12

*Now referred to as Crew and Watch Managers

A solution to this issue was brought about with the introduction of The The Fire and

Rescue Service National Framework 2008-11. The caveat of being time served was

removed due to being deemed potentially discriminatory and to allow Fire Services

more flexibility with the promotion process by introducing ADCs.

There are shortfalls that should be highlighted within the current ADC process that

THE FIRE AND RESCUE SERVICE and other Fire Services use. There were only

three sets or ‘scenarios’ that were produced by the IPDS Hub which was originally

located at the Fire Service College. The sets have been in use since 2006 since

which time, only two of the sets are now used due to an issue with Set 1 (although

there is no available written evidence of this) and therefore the same set has been

used on more than one occasion which could lead to individuals becoming ‘familiar’

with the exercises as opposed to portraying a true picture of how they would behave.

This could also lead to individuals informing others of what is involved in each

exercise therefore giving them an unfair advantage or can an individual learn the

PQAs and BARS and therefore know how they need to behave in exercises in order

to score highly?

On carrying out research of PQAs measured in the ITOP against the PQAs

measured in the ADC, there appears to be minimal correlation. If an individual

scores high in a particular PQA in the ITOP, they would be expected to score high in

the PQA in the ADC but this was not always the case. This would bring in to

question the validity of ADCs or indeed some of the processes used within an ADC.

7.0 SUMMARY

It could be argued that the validity of ADCs meeting an organisation’s strategic

needs, would depend on what an organisation introduces as part of its ADC process.

In the past, THE FIRE AND RESCUE SERVICE only carried out interviews when

there were job positions for individuals on the development programme to apply for.

However, the organisation found that this was not effective. The organisation was

heavily investing in developing individuals that were deemed unsuitable for

promotion. As a result of this, THE FIRE AND RESCUE SERVICE introduced a PQA

Page 9 of 12

interview in to the ADC process prior to individuals being placed on a development

programme.

However, other stages such as a technical/practical assessment could be introduced

to confirm an individual’s suitability within the fire service operational field. At this

moment, technical and practical skills are only developed when an individual is

already placed on a development programme.

In conclusion, it difficult to directly quantify in this report if the ADC process currently

administered within THE FIRE AND RESCUE SERVICE directly contributes to

meeting the strategic needs over any other selection process. However, THE FIRE

AND RESCUE SERVICE should constantly monitor their selection methods and

contribute to their strategic needs by delivering best value and a safe, healthy,

competent and representative workforce.

WORD COUNT: 2945

Page 10 of 12

BIBLIOGRAPHY

Anderson, N and Shackleton, V. (1993) Successful selection interviewing. Oxford: Blackwell

Appelbaum, S H; Harel, V and Shapiro, B (1998). The Developmental Assessment Centre: The Next Generation. Career Development International pp.5-12

Chartered Institute for Professional Development (2010). ‘Selection Interviewing’. Available at http://www.cipd.co.uk/subjects/recruitmen/selectn/selnintvg.htm?WBCMODE=Present-ationUnpublishedfaculty.htm Accessed 26 April 2010

Department of Communities and Local Government (2008). ‘The The Fire and Rescue Service National Framework 2008-11’. Available at http://www.communities.gov.uk/documents/fire/ pdf/nationalframework200811.pdf Accessed 14 April 2010

Department of Communities and Local Government (2009). ‘Integrated Personal Development System Code of Practice’. Available at http://www.skillsforjustice-ipds.com/documents/IPDS%20Code%20of%20Practice.pdf Accessed 16 April 2010

Fire Services Examination Board (nd). ‘About the Examinations’. Available at http://www.fseb.gov.uk/exams/index.htm Accessed 18 April 2010

Hay, J (1997) Gower Assessment and Development Centre: Volume 1 (1st Edition) Surrey : Gower

THE THE FIRE AND RESCUE SERVICE (2008). ‘Risk Management Plan 2008-11’.

THE THE FIRE AND RESCUE SERVICE (2009). ‘The Workforce Development Strategy 2009/11’.

THE THE FIRE AND RESCUE SERVICE (2008). Service Order Training – 03. ‘Development and Promotion Procedures – Assessment and Development Centre Process’ Section 1 pp.1-18

Liam Healy & Associates (nd). ‘Assessment and Development Centres’. Available at http://www.psychometrics.co.uk/adc.htm Accessed 25 April 2010

Office of the Deputy Prime Minister (2003). ‘Our The Fire and Rescue Service’. Available at http://www.communities.gov.uk/documents/fire/pdf/128947.pdf Accessed 14 April 2010

Office of the Deputy Prime Minister (2004). ‘The The Fire and Rescue Service National Framework 2004/5’. Available at http://www.communities.gov.uk/documents/fire/pdf/ 128923.pdf Accessed 14 April 2010

Page 11 of 12

Office of Public Sector Information (2004). ‘The The Fire and Rescue Services Act 2004’.

Available at http://www.opsi.gov.uk/acts/acts2004/ukpga_20040021_en_1 Accessed 14

April 2010

Taylor, S (2008) People Resourcing (4th Edition) London : CIPD

Thornton, G C and Rupp, D E (2006). Assessment Centers in Human Resource Management (1st Edition) New Jersey : Erlbaum

UK Fire Service Resources (nd). ‘Situational Judgement Test - Multiple Choice’. Available at http://www.fireservice.co.uk/development/sjt.php Accessed 25 April 2010

Woodruffe, C (2000) Development and Assessment Centres (3rd Edition) London : CIPD

Page 12 of 12

THE THE FIRE AND RESCUE SERVICE – HIERARCHICAL STRUCTURE

Chief Fire Officer

Deputy Chief Fire Officer

Asst Chief Fire Officer

Area Manager

Group Manager

Station Manager

Watch Manager

Crew Manager

Firefighter

Appendix 1

Appendix 2

The Fire and Rescue Service Overview Personal Qualities and Attributes (PQAs) Framework – Operational Roles

* = PQA is relevant to operational aspects of managerial roles only; it is assumed that those already working in operational roles within the FRS already possess this PQA.

Skills for Justice (2009)http://www.skillsforjustice-ipds.com/documents/PQA%20Framework%20-%20all%20levels.pdf

Appendix 3

The Fire and Rescue Service Supervisory Level Generic Behaviourally Anchored Rating Scales (BARS)

4 Strong Performance – meets all of the positive behaviours expected of that role with no development needs being demonstrated.3 Acceptable Performance – meets most of the positive behaviours expected of that role with very few development needs being demonstrated.2 Development Need – meets some of the positive behaviours expected of that role but also displays some development needs or displays one significant

development need that counter balances the positive behaviours.1 Significant Development Need – meets none or very few of the positive behaviours expected of that role but displays mostly development needs.