INTERNATIONAL POTENTIA ABILITY TEST MANUAL (AUG07) SERIES TEST MANUAL 2009.pdf · KCP Potentia...

Administration

Manual

Managers ♦♦♦♦♦ Graduates

KCP Potentia Series

KCPMaking Psychology Accessible

Potentia

Tests

Ability

KCP Potentia Series

KCP Potentia Series Ability Test Manual

by

Andrew Roberts BSc CPsychol

Knight Chapman Psychological Ltd

1 High Street

Lewes

East Sussex BN7 2AD

KCP Potentia Series


ACKNOWLEDGEMENTS

We have been extremely fortunate to have had considerable assistance in the developing,

trialling, validation and design of the Series.

We would like to extend our special thanks and gratitude to:

Rachel Frost of KCP who assisted in the development of this battery.

Alison Cox of KCP, who contributed extensively to the original design, development and

scoring of the tests, as well as to the necessary but exhausting task of compiling trialling

materials.

Leigh Grainger of the University of Cardiff's Department of Psychology for her creative input

into the development of the Reference Materials and test items, and for her considerable

help in trialling the tests.

Dr Chris Miles of the University of Cardiff for arranging large numbers of “guinea-pigs” for

the initial trialling.

Janice Truscott and her colleagues at Cadburys for all their help in trialling and validation.

Kyrstie Broadhead and her colleagues at Tesco for assistance in trialling.

Peter Harris and his colleagues at Llanelli Borough Council for assistance in trialling and

validation.

Peter Messer of Peter Messer Design Consultants in Lewes for the design of the Reference

Materials.

Keith Gillies of Caburn Communications for the original design of KCP test materials.

Paul Dale and John Cooper of the Venture Press, Scaynes Hill, West Sussex for all the

proofing and printing.

KCP Potentia Series


CONTENTS PAGE

BACKGROUND AND DEVELOPMENTAL OBJECTIVES 5

DESCRIPTION OF THE TESTS 6

ABILITY TEST ADMINISTRATION 8

Reference Material 10

Verbal Reasoning 11-12

Numerical Reasoning 13-14

Creative Problem Solving 15-16

ABILITY TEST SCORING 17

Verbal & Numerical Reasoning 17

Creative Problem Solving 18-28

Norm Tables 29

RELIABILITY 31-33

INTERCORRELATIONS 34

VALIDITY 35-36

KCP Potentia Series


BACKGROUND AND DEVELOPMENTAL OBJECTIVES

The Potentia Series was developed in response to the growing need for an integrated high

level assessment package.

Prior to Potentia there were several aptitude tests for graduates and managers on the

market, and in addition there were assessment exercises suitable for selection and

development at this level.

At Knight Chapman Psychological, however we became aware that assessors were rarely in

the situation where they found assessment exercises suited to their precise requirements. As

a result, many assessment centres have been designed on the basis of materials that

happened to be available “off-the-shelf”.

Candidates have therefore been presented with a disjointed and unrealistic procedure where

their abilities and working style have been assessed across discrete, unrelated scenarios -

one minute they are expected to market a cosmetic, the next they find themselves

manufacturing kitchen appliances.

In reality, of course, managers and professionals very rarely resolve problems which are so

disjointed and lack interdependence.

The Potentia package is designed to provide a complete set of associated, realistic

assessment materials to reliably measure a wide range of core management and graduate

competencies. The individual tests and exercises can be administered separately or in any

combination.

The structure of the series is such that all tasks relate to a common set of reference materials

which reflects a realistic but neutral work situation. Candidates will immediately realise the

relevance of the tests to selection or development within a graduate/managerial environment.

Gaining their ‘buy in’ is accepted as crucial to reliable assessment.

KCP Potentia Series


DESCRIPTION OF THE TESTS

Structure

The Potentia Series consists of a number of tests and exercises which measure a range of

high level problem solving abilities and skills recognised to be important for effective

managerial and graduate trainee positions.

ABILITY TESTS

The tests which are currently available to Potentia users are:

Verbal Critical Reasoning

Numerical Critical Reasoning

Creative Problem Solving

The questions in each of the tests relate to a set of documents, which candidates have 10

minutes to look through at the beginning of the Potentia administration session. There are 11

documents which refer to the proposed takeover of one small theatre by a larger group of

theatres. The documents create a general scenario, as well as providing specific verbal and

numerical information.

All of the test themselves can be administered individually or in any combination.

Verbal Critical Reasoning

Each question consists of a statement which the candidate must evaluate in the light of the

information given in the 11 documents. A three-way multiple-choice format requires the

candidate to decide whether the statement is true, or false, or whether there is insufficient

information given in the inserts to say for certain whether the statement is true or false.

Numerical Critical Reasoning

The candidate is required to use data contained in the 11 documents to answer numerical

critical reasoning questions. Although arithmetic calculations (addition, subtraction,

multiplication, division) and basic algebra are required, the emphasis is on reasoning rather

than arithmetic processing. Numerical concepts including graph reading, ratios, table

reading, percentages, percentage change, and demographic comparisons are

featured. Face and content validity are enhanced by the use of data and concepts which

candidates will recognise as relevant to real-life situations.

KCP Potentia Series


Creative Problem Solving

The candidate is required to list possible causes or reasons for given events. These events

are within the context of the information contained in the 11 documents. Candidates are

informed that this is a test of creativity and are encouraged to use their imagination in trying to

think of unlikely causes as well as more probable ones. Candidates are also asked to try to

include some causes which other people will probably not have thought of.

KCP Potentia Series


ADMINISTRATION

Research has shown that small differences in the way tests are administered have significant

effects on scores resulting from the tests. The administration of tests is therefore critical to

the provision of reliable, uncontaminated measurements. Great care must be taken to ensure

standardisation.

Preparation

The objective is to run a well coordinated testing session with no emergencies. Test

administrators should make sure they are completely familiar with the instructions and test

procedure. They should also fully understand the test objectives and background so that they

are able to answer questions raised by test takers.

Materials should be carefully counted and checked. Administrators should ensure that

pencils, rubbers, rough paper, pocket calculators, test booklets, Reference Material and

answer sheets are all collated and that a stop watch, administration booklet, test manual

and scoring key are to hand.

The test conditions also need some consideration prior to the testing session. A suitable

room should be chosen which will be free from undue noise and distraction, and will

provide adequate lighting, ventilation/heating, seating and space. The test administrator

should take steps to ensure that there are no interruptions during the session: telephones

should be disconnected or redirected and a sign should be posted on the door outside

indicating that testing is in progress. Other personnel should be briefed that interruptions

are strictly prohibited.

The administrator’s delivery of the introduction should be such that it reassures,

encourages and motivates the candidates.

The introduction should include details of the timetable, domestic arrangements (including

where the lavatories are located) and a description of the test to be taken. The purposes

of the test should be “sold” to the candidates - in particular, it should be pointed out that a

good fit between the job requirements and the job holder is in everybody’s interests.

Introduction

Prior to the formal test instructions, a less formal introduction should be delivered to the

candidates. The purpose of this is partly to provide relevant information, but also to build

rapport.

Rapport, in this context, refers to the administrator’s efforts to arouse the candidates

interest in the test, elicit their cooperation, and encourage them to give their full

concentration. The content of the introduction should eliminate the element of surprise

from the situation. Uncertainty is a major component of test anxiety and so should be

reduced as far as possible before the testing begins.

KCP Potentia Series


Test Instructions

It is imperative that these are presented in a standardised way according to the

administration booklet. In order to ensure that every candidate receives the same

instructions, these should always be read, rather than memorised.

The instructions which appear in the administration booklet are given on the following

pages.

Timing

Clear start and stop instructions should be given by the administrator. The test durations must

be timed exactly using a stopwatch. A secondary watch or clock should also be used in case

the stopwatch goes wrong or a candidate disputes the time given.

The time allowed for each test is:

* Reference Material 10 minutes

* Verbal Critical Reasoning 16 minutes

* Numerical Critical Reasoning 35 minutes

* Creative Problem Solving 14 minutes

Sequence of Tests

The Reference Material is always administered first. Subsequent tests can be administered

in any order, indeed, there may be occasions where only one test, for example, Numerical

Reasoning is being used.

However, if the entire battery is to be administered, we would generally use the sequence of

Verbal Reasoning, Numerical Reasoning and then Creative Problem Solving.

KCP Potentia Series


POTENTIA - REFERENCE MATERIAL ADMINISTRATION INSTRUCTIONS

Explain to the candidates which tests they will be asked to take and outline the timetable for the

session. Provide any other background information which is appropriate (e.g. relevance of tests to

job content).

Ensure that candidates are comfortable and answer any questions they wish to raise.

Give each candidate:

• a Reference Material booklet

• 2 sheets of rough paper

• 2 pencils

• an eraser

Say:

“Please turn to the first page of your Reference Material Booklet, where you will find the instructions. Please

follow these instructions whilst I read them.

The questions in each of the tests relate to this Reference Material which you will have 10 minutes to look

through before you begin the tests themselves.

This Reference Material consists of eleven documents relating to an ailing theatre company which is being

threatened with takeover by a large organisation, the Royal Theatre Company.

The documents have been put together for the Chairman of this group.

Before you start each test you will receive full instructions explaining what you have to do.

During the next 10 minutes you may not have time to read all of the eleven documents in detail, but try to get a

feel for the kind of information contained in each.

Remember, you will have this material to refer to throughout the tests. If you would like to make notes, you

may use the rough paper provided but please do not write on the Reference Material Booklet or Questionnaire

Booklets.

Are there any questions at this stage?”

Pause. Deal with any questions raised.

Ask the candidates to:

“Turn to the next page of your Reference Material booklet and start looking through the reference material

NOW.”

After EXACTLY 10 minutes say:

"STOP. Please close your Reference Material booklet now."

Check that all candidates have closed their Reference Material booklets.

Test administrators should now refer to the specific administration instructions for each of the tests to

be taken.

KCP Potentia Series


POTENTIA - VERBAL REASONING ADMINISTRATION INSTRUCTIONS

Say: “This is a test of your verbal critical reasoning ability - your ability to identify logic in written argument.

Each of the questions in this test consists of a statement which you must evaluate in the light of the

information given in the Reference Material about the Heritage Theatre.

Give your answers on the Answer Sheet by filling in circle A, B or C according to these rules:

Fill in Circle A if the statement is TRUE or follows logically from the information given.

Fill in Circle B if the statement is FALSE from the information given in the Reference Material.

Fill in Circle C if there is INSUFFICIENT INFORMATION given in the Reference Material to say for certain

whether the statement is true or false.

These rules are given on each page of the test.

Tell the candidates not to open their test booklets until asked to do so. Provide the candidates with

any background information about the test and its purposes. Ensure that the candidates are

comfortable and answer any questions they wish to raise.

At the beginning of the test administration session, candidates should have spent 10 minutes looking

through the Reference Material (see Potentia Reference Material Administration Instructions).

Say: “ Please write your full name and today’s date in the spaces provided on your answer sheet.” Pause.

Say: ”Open your test booklet to the first page, where you will find the test instructions. Please follow these

instructions while I read them.” Pause.

Ensure that each candidate has:

• a PVT booklet

• a PVT answer Sheet

• a Reference Material Booklet

• a sheet of rough paper

• 2 pencils

• an eraser

When giving your answers, use ONLY the information given in the Reference Material. You should assume that

this information is correct even if it contradicts what you believe to be the case in reality.

Before you start the test there are some example questions. Please do these now in your own time, giving your

answers in the Example Section on the Answer Sheet.

You WILL need to refer to the Reference Material. When you have finished the three example questions,

please close your Reference Material booklet."

Check that the candidates are answering the example questions correctly, providing help where

necessary. When all of the candidates have completed the example questions correctly, say:

KCP Potentia Series


"The answer to Example 1 is C. There is Insufficient Information to say for certain whether the statement is

true or false.

The answer to Example 2 is A. The statement is True from the information given in Document 3.

The answer to Example 3 is A. The statement is True from the information given in the Entertainments

Research Company report - Document 9.

Are there any questions about these examples?" Pause. Deal with any questions raised.

POTENTIA - VERBAL REASONING ADMINISTRATION INSTRUCTIONS (cont'd)

Remember, you are allowed to use the Reference Material throughout the test.

Are there any final questions?"



"Turn over to page 1 and BEGIN."

START THE STOPWATCH

During the test, the test administrator may answer any questions relating to procedure, but should

not help candidates with the questions themselves.

Note any significant problems or questions raised by candidates.


"STOP. Please close your Reference Material Booklet and your Test Booklet."

If you are going on to administer further tests, collect in the materials which are no longer required,

i.e. answer sheets and test booklets, but ensure that the candidates retain their Reference Material

booklets, pencils, erasers and rough paper. If you are not administering additional tests, collect all

materials and thank the candidates for doing the test.

Say: "Please note the following points.

You will have sixteen minutes for the thirty questions in this test. Work as quickly and accurately as you can.

If you are not sure of an answer, move on and return to it later rather than simply guessing. Please use the

pencils provided. Rub out and change your answers if you wish.

KCP Potentia Series


POTENTIA - NUMERICAL REASONING ADMINISTRATION INSTRUCTIONS


• a PNT booklet

• a PNT answer sheet

• a pocket calculator

• 2 sheets of rough paper


• 2 pencils

• an eraser

Tell the candidates not to open their test booklets until asked to do so. Provide the candidates with

any background information about the test and its purposes. Ensure that the candidates are comfort-

able and answer any questions they wish to raise.

At the beginning of the test administration session, candidates should have spent 10 minutes looking

through the Reference Material (see Potentia Reference Material Administration Instructions).

Say: “Please write your full name and today’s date in the spaces provided on your answer sheet.” Pause.

Say: ”Open your test booklet to the first page where you will find the test instructions. Please follow these

instructions while I read them.” Pause.

Say: “This is a test of your numerical critical reasoning ability- your ability to manipulate and integrate work-

related data. Each of the questions in this test relates to data contained in the Reference Material. For each

question you are given five answers to choose from. Only one answer is correct in each case. Decimals are

generally rounded to two decimal places.

Please note that some questions may be impossible to answer from the data that you have been given. In

these cases “Insufficient data” is the correct answer. You may use a pocket calculator and the rough paper

provided.

Give your answer by filling in the appropriate circle A,B,C,D or E on the Answer Sheet.

Before you start the test there are some example questions. Please do these now in your own time, giving your

answers in the Example Section on the Answer Sheet.

You WILL need to refer to the Reference Material. When you have finished the three example questions,

please close your Reference Material booklet.”

Check that the candidates are answering the example questions correctly, providing help where

necessary. When all of the candidates have completed the example questions correctly, say:

“The answer to Example 1 is A - 65.80%

The answer to Example 2 is D - 1,019

The answer to Example 3 is E - Insufficient data.

Are there any questions about these examples?”


KCP Potentia Series


Say: “Please note the following points:

You will have 35 minutes for the 26 questions in this test, so you will need to work quickly but accurately.

Use the pencils provided to fill in your answers on the Answer Sheet so that you can rub out and change your

answers if you wish. If you are not sure of an answer, move on and return to it later rather than simply guessing.

Remember, you may refer to the Reference Material throughout the test.

Are there any final questions?”



“Turn over to page 1 and BEGIN.”

START THE STOPWATCH

During the test, the test administrator may answer any questions relating to procedure, but should not

help candidates with the questions themselves.

Note any significant problems or questions raised by the candidates.


“STOP. Please close your Reference Material booklet and your Test booklet.”

If you are going on to administer further tests, collect in the materials which are no longer required,

i.e. answer sheets, test booklets and calculators, but ensure that the candidates retain their

Reference Material booklets, pencils, erasers and rough paper. If you are not administering

additional tests, collect all materials and thank the candidate for doing the test.

POTENTIA - NUMERICAL REASONING ADMINISTRATION INSTRUCTIONS (cont'd)

KCP Potentia Series


POTENTIA - CREATIVE PROBLEM SOLVING ADMINISTRATION INSTRUCTIONS


• a Creative Problem Solving test booklet


• 2 pencils

• a rubber

Tell the candidates not to open the test booklet until told to do so.

At the beginning of the test administration session, candidates should have spent 10

minutes looking through the Reference Material. (See Potentia Reference Material

Administration Instructions.)

Ask the candidate to :

“Please write your full name and today’s date in the spaces provide on the front cover of the

test booklet.”

Pause

“Open the test booklet to page 1 where you will find the test instructions.”

Pause

“Please follow these instructions whilst I read them.

This is a test of your creative problem solving ability - your ability to generate ideas and be

imaginative.

Each of the questions in this test consists of a possible problem which is broadly related to

the scenario contained in the Reference Material about the Heritage Theatre.

There are four problems.

Your task is to produce a list of the possible causes of each of the given problems. Try to

be imaginative. Write down as many ideas as you can. There are no right or wrong

answers, so try to come up with unusual or unlikely ideas which other people may not have

thought of, as well as more probable explanations.

KCP Potentia Series


Please remember:

Write as clearly as you can.

Use the spaces provided on the following pages to write your lists.

If you run out of space on a page, turn over and use the blank paper on the next side.

Make sure that your answers are relevant to the problem.

You will have fourteen minutes for the four problems in this test, so you will need to work

quickly.”

Say:

“Remember, you may refer to the Reference Material during the test.

Are there any final questions?”



“Turn over to page 2 and BEGIN.”

START THE STOPWATCH

During the test, the test administrator may answer any questions relating to proce-

dure, but should not help candidates with the lists themselves.

Note any significant problems or questions raised by the candidates.


“STOP. Please close your Reference Material booklet and your Test booklet.”

If you are going on to administer further tests, collect in the materials which are no

longer required, i.e. the test booklet, but ensure that the candidates retain their

Reference Material booklets, pencils and rubbers. If you are not administering

additional tests, collect all materials and thank the candidates for doing the test.

KCP Potentia Series


SCORING - VERBAL AND NUMERICAL REASONING

Verbal Reasoning and Numerical Reasoning are both multiple choice tests with systematic

scoring systems. Scoring keys are used to score the total number of correct answers to

produce a raw score which is then interpreted by comparison with a norm group. The stages

in scoring are:

1. Check the answer sheet before using the scoring key. If the candidate has filled

in more than one box for a particular question, no marks are given for this question.

2. Align the scoring key so that the circles on the answer sheet show through the

reference circles on the Key.

3. Count the number of questions answered correctly. Write the total number of

correct answers in the box marked RS on the answer sheet.

4. Count up the total number of items attempted, and write this in the box marked RS

on the answer sheet as well.

5. Choose an appropriate norm table. Make a note of the table used in the box

marked ‘comparison group.’

6. Use the norm table to convert the raw score to a percentile score or standard

score.

KCP Potentia Series


SCORING - CREATIVE PROBLEM SOLVING TEST

General Points

The Creative Problem Solving Test is designed to measure the candidate’s ability to produce

ideas creatively. This is separate and different from the candidate’s critical reasoning

abilities. Critical reasoning is a convergent thinking process - the Creative Problem Solving

Test is concerned with divergent thinking.

Assessors are often tempted to give better scores to responses which are logical or “based

on common sense”. However, since this is not a test of reasoning abilities or judgement, this

temptation should be resisted. Creative Problem Solving is purely a test of creative

production and responses should be assessed with this in mind.

Open ended tests are inevitably more difficult and less objective to score than multiple choice

tests. Assessors should aim to stick firmly to the guidelines which follow, but will need to

generalise from these guidelines on occasions. This is because the number of potential

responses to the questions is infinite and no set of scoring instructions could comprehensively

cover all eventualities.

Ideally two separate assessors should score each candidate’s responses independently and

then discuss and rectify any disparities between the generated scores.

It is strongly recommended that this approach of dual scoring is adopted when the Creative

Problem Solving Test is first implemented, as this process highlights areas where

standardisation can be tightened up.

There are three scores:-

* Fluency

* Lateral Flexibility

* Originality

The procedure for determining a score on each of these dimensions of creative thinking is

covered on the following pages.

KCP Potentia Series


Scoring Procedure

Fluency (Column One)

The fluency score is a measure of the quantity of ideas produced and is based on a simple

count of the acceptable responses given. For each item, count up the number of acceptable

responses and record the total in the box at the foot of the first column.

Responses should NOT be counted if they are:-

Totally irrelevant to the given fact or event.

Duplicates of previously given responses, i.e. a simple rewording of precisely the

same idea.

These criteria of unacceptability should be interpreted liberally. Unacceptable responses are

quite uncommon. If in doubt, assessors should allow responses as acceptable. However, if a

response is unacceptable it should be discounted from the scoring of originality and lateral

flexibility as well as fluency.

Lateral Flexibility (Column Two)

The Lateral Flexibility score concerns the range, or breadth of responses given. If one

candidate has given five responses to a particular item, but all five ideas are along similar

lines, the lateral flexibility score will be low. If another candidate has also given five responses

to the same item, but the five ideas are diverse in their content, the lateral flexibility score will

be higher.

For each test item, refer to the scoring guidelines on the following pages. These guidelines

indicate the headings into which responses may fall. Against each of the candidate’s ideas,

make a note of the heading by writing its letter, A B, C etc. in column two. If an idea is clearly

different and does not fit into one of the given categories, it should be given an “O” for “Other”.

Now add up the total number of different letters, and write this score in the box at the bottom

of column two. If there are “O” ratings, each one is awarded a mark.

Please note that for each heading, we have provided some examples. These are for

illustrative purposes only and by no means represent the total universe of responses that

might appear under each heading.

KCP Potentia Series


Originality (Column Three)

The score for originality is naturally more qualitative than quantitative. “Originality” in this

context is a measure of the rarity or unusualness of the responses given. Higher originality

scores are given for responses which are not common amongst other candidates. The

“commonness” or popularity of responses has been ascertained through content analysis of

trialling data. However, since the same kind of idea may be expressed in many different

ways, the originality of each response is assessed on the basis of the category that it falls

into.

When scoring an item for originality, also note the ‘Extra mark’ headings. Generally speaking,

an additional mark should be awarded if the response is fuller, more detailed and more

specific.

Responses which are highly unusual are placed in the “Other” category. These automatically

attract 3 marks, and no extra specificity marks should be given.

Our experience with the Supra Series suggests that “Other” categories are relatively few and

far between, and assessors should be sparing in their use of this category.

KCP Potentia Series


ITEM 1. International news reports suggest that theatres in the UK are a

thing of the past.

A. Economic/financial

e.g

People have less money to spend.

Theatres become more expensive.

Extra mark - Fuller explanation such as large scale redundancies mean

that more people are unemployed and therefore have less money to

spend.

Originality score - 0

B. Improvement/Increase in other entertainment options

e.g

Television more popular.

People prefer computer games.

Video take over.

Extra mark - Fuller explanation such as a new TV company is so popular

that no one leaves their own home for entertainment.


C. The standard of theatres is low/has declined

e.g

Theatres are dilapidated.

Props have fallen apart.

Extra mark - Fuller explanation such as the seats have all been removed

from theatres as part of the war effort.


D. Fashion trends/youth culture

e.g

Not ready/old enough to go to the theatre.

Young people are more interested in other things.

Extra mark - Fuller explanation such as schools no longer teach young

people to appreciate cultural entertainment.


KCP Potentia Series

KCP Potentia Series Ability Test Manual.

E. Advertising/marketing

e.g

Theatres not promoting themselves anymore.

Poor marketing and advertising.

Extra mark - Fuller explanation such as a particular marketing campaign

was badly handled and had the opposite effect to that intended, putting

people off from going to the theatre.


F. Incorrect information

e.g

Newspaper got figure wrong.

Journalists interpreted the statistics wrongly.

Extra mark - Fuller explanation such as the journalist is paid by the

government to distort figures to make the UK economy look better than it is.


O. Other

e.g

All UK actors and directors have fled the country and gone to Hollywood.

No extra marks.


KCP Potentia Series


ITEM 2. The Heritage Theatre wins its takeover battle and resists Inglebert.

A Aspects specifically mentioned already in the Reference Material

(excluding local community objections).

e.g

Marketing or financial ideas mentioned in the Reference Material.

Inglebert’s promotional ideas.

Extra mark - Where a candidate significantly builds upon the information

given in the Reference Material.

Originality score - O

B. Local community objections/support

e.g

This local community showed their support for the theatre by attending

productions.

Strong objections from the local community.

Extra mark - Fuller explanation such as the local community were so worried

about losing their theatre that they organised fund-raising events to save it.


C. Inglebert/RTC go off the idea and withdraw

e.g

Inglebert follows the Entertainments Research Company’s recommendation and

withdraws.

The RTC Directors put pressure on Inglebert to withdraw bid.

Extra mark - Fuller explanation such as Inglebert is charged with fraud

and the RTC have to sell off assets to pay for his bail.


D. Cash injection (other than from the local community)

e.g

Anonymous supporter bequeaths funds to the theatre.

Government gives grant aid.

Extra mark - Fuller explanation such as a local millionaire commits suicide

and has donated his whole estate to the theatre.


KCP Potentia Series


E. Entirely new marketing ideas

e.g

Mailshots are targeted better.

A celebrity visits the theatre to pledge support.

Extra mark - Fuller explanation such as a programme is sent out to all

social clubs offering a reduction in price if more than 10 individuals take

out an annual subscription.


F. The building is used for another purpose.

e.g

The Heritage is turned into a bingo hall.

The theatre is used for dances.

Extra mark - Fuller explanation such as a local company establishes a

contract to hire the theatre regularly in order to sell holiday time-shares.


G. Other

e.g

By coincidence, the theatre turns out to be the only venue suitable to stage a

play that is part of the core curriculum.

No extra marks.


KCP Potentia Series


ITEM 3. A marketing company has promised to increase the Heritage

Theatre’s sales threefold.

A. Advertising/marketing strategies already mentioned in the Reference

Material

e.g

Discounts given to old age pensioners.

Advertising space in programmes offered to local businesses.

Extra mark - Fuller explanation such as the Wednesday evening discount

is moved to Thursday as there is nothing worth watching on TV that night.


B. Improvements in the quality of productions/reduction in ticket

prices

e.g

Ticket prices are slashed.

Better quality actors are employed.

Extra mark - Fuller explanation such as they hire big name performers

for random performances so that the audience takes pot luck as to

whether they get an amateur or a professional production.


C. Advertising/marketing campaign not mentioned in the Reference

Material

e.g

Advertising space bought on Tube trains.

Direct mail campaign targeted at specific sections of the community.

Extra mark - Fuller explanation such as tokens are given at local petrol

stations - collect 10 and get a free ticket at the petrol station’s expenses.


D. Increase money available for marketing or improvements to the

theatre

e.g

Marketing company finds a sponsor to invest in the theatre.

Marketing company invests money and takes a stake in the theatre.

Extra mark - Fuller explanation such as the marketing company has been

approached by an international philanthropist who wants to invest in the

theatre.


KCP Potentia Series


E. Use theatre for other purposes

e.g

Turn into a bingo hall.

Hire a “Chippendale” - type act.

Extra mark - Fuller explanation such as hire the theatre out for hen nights

starring local “Chippendales”.


O. Other

e.g

Pretend that the takeover bid was won by the Heritage and capitalise on

advertising the theatre as better and more successful than the RTC.


KCP Potentia Series


ITEM 4. Large groups are demonstrating outside the Heritage Theatre.

A. Takeover bid (excluding refurbishment)

e.g

Local community demonstrating against (or for) takeover.

Actors lobbying for support.

Extra mark - Fuller explanation such as English National Heritage

organises a mass demonstration to show support for independent

theatres which are being forced to close.


B. Refurbishment

e.g

Heritage losing its Victorian facade.

Closure of theatre due to refurbishment.

Extra mark - Fuller explanation such as the “Save Victorian Buildings Society”

organise a mass demonstration to protest against the loss of Victorian values.


C. Contentious play/performance

e.g

Sexist play.

Script is considered racist.

Extra mark - Fuller explanation such as an anti-drug campaign against

“Needles and Opium” portraying drug use as glamorous.


D. Demonstration which has nothing to do with theatres

e.g

Animal rights activists.

Dock workers have come out on strike.

Extra mark - Fuller explanation such as a strike has been called by

dockers because a property developer has bought the site and plans to

turn it into an exclusive shopping centre.


KCP Potentia Series


E. Prices too high

e.g

People resent ticket price increases.

Discounts withdrawn.

Extra mark - Fuller explanation such as Winthorpe has doubled the ticket

prices in order to raise money for refurbishing the interior of the theatre.


O. Other

e.g

Demonstration in support of an actor who is accused of an assassination

attempt on the Queen.

No extra marks.


KCP Potentia Series


PERCENTILE NORM TABLES

Composite graduate group (N = 1354, Verbal and Numerical

N= 90, Creative Problem Solving)

%ile Verbal Numerical Fluency Lateral Flex Originality %ile

99 23-26 36+ 21+ 33+ 99

97 28-30 21-22 33-35 19-20 30-32 97

95 26-27 19-20 30-32 18 27-29 95

90 24-25 17-18 28-29 16-17 24-26 90

85 23 16 26-27 15 22-23 85

80 22 15 24-25 20-21 80

75 21 14 23 14 19 75

70 20 13 22 13 18 70

65 19 12 20-21 16-17 65

60 18 11 19 12 15 60

55 17 10 18 11 14 55

50 16 17 13 50

45 15 9 16 10 11-12 45

40 14 8 15 10 40

35 13 7 14 9 9 35

30 12 6 12-13 8 8 30

25 11 5 11 6-7 25

20 10 4 9-10 7 5 20

15 9 3 7-8 6 2-4 15

10 7-8 1-2 5-6 4-5 0-1 10

5 5-6 0 2-4 3 5

3 2-4 0-1 2 3

1 0-1 0 0-1 1

KCP Potentia Series


T-SCORE NORM TABLES

Composite graduate group (N = 1354, Verbal and Numerical)

T Verbal Numerical

80

78

76 25-26

74 24

72 23

70 29-30 22

68 28 20-21

66 27 19

64 25-26 18

62 24 17

60 22-23 16

58 21 14-15

56 20 13

54 18-19 12

52 17 11

50 16 9-10

48 14-15 8

46 13 7

44 12 6

42 10-11 5

40 9 3-4

38 8 2

36 6-7 1

34 5 0

32 4

30 2-3

28 1

26 0

KCP Potentia Series


RELIABILITY

It is critical that tests are administered and scored in a standardised way, to avoid variability

creeping into the testing process and contaminating test results. However, even when

guidelines are adhered to, psychological tests cannot measure abilities with 100% accuracy.

What is Test Reliability?

Test reliability refers to the test’s degree of precision, or accuracy of measurement. To

understand this concept, we can use the analogy of measuring distance (or “length”). If we

wanted to measure the length of a football pitch, we could pace from one end to the other

counting each pace as roughly 1 metre. This would provide us with an imprecise indication of

the football pitch’s length.

A more reliable (or precise) method would be to use a tape measure. However, a tape

measure is not the most reliable instrument for measuring a length or distance: very precise

devices such as the micrometer can be used to measure the nearest thousandth of a

millimetre. So, “pacing”, the tape measure and the micrometer are three methods of

measuring distance which differ in their reliabilities. To measure a football pitch it would be

absurd to use a micrometer, but “pacing” may be too inaccurate: the level of reliability offered

by the tape measure method may make it the most appropriate option.

Like the “pacing” and tape measure methods, testing has a degree of inherent unreliability. In

evaluating tests for use in assessment, we are concerned to ensure that the test’s reliability is

acceptable. If scores are too inaccurate, wrong decisions will be made.

The reliability of a test is expressed as a correlation coefficient and is called a reliability

coefficient.

A reliability coefficient of 0.0 means a test has zero reliability; it is completely inaccurate and

the characteristic is being measured no more precisely than a random guess. A reliability of

1.0 means that a test is 100% reliable; it measures the characteristic with perfect precision.

However, this degree of reliability is never actually found. A reliability coefficient is thus an

index with a theoretical range of 0 to 1. In practice, reliabilities generally vary between 0.5

and 0.9.

KCP Potentia Series


Different Kinds of Reliability

Test-Retest Reliability

The test-retest reliability coefficient is the correlation between the scores obtained by the

same person on two separate administrations of the test. If test results are generally

different on the second administration from the first, the test-retest reliability coefficient will

be low. If there is a good relationship between scores on the two occasions, the

coefficient will be high.

Alternate-form Reliability

This is computed by comparing scores obtained by the same individuals on two forms (or

versions) of the same test. Of course, this method is limited to tests for which more than

one form exist.

Internal Consistency

Only one administration of the test is required to compute the internal consistency coeffi-

cient. This is derived by splitting the test questions into two and then correlating the

respective totals for the two halves. One approach is to divide out the odd and even

questions and then correlate odds with evens. More sophisticated ways of dividing the

questions into two sets are also available. If all of the questions are measuring the same

characteristic, the internal consistency coefficient should be high.

KCP Potentia Series


Internal Consistencies

Test N Mean SD Internal SEm

Consistency*

Verbal Reasoning 1354 15.93 6.75 0.85 2.61

Numerical Reasoning 1354 9.58 6.10 0.86 2.28

C.P.S: Fluency 90 14.68 8.19 0.77 3.92

C.P.S: Originality 90 10.57 9.37 0.72 4.96

C.P.S: Lat.Flex. 90 8.97 4.07 0.66 2.37

* Cronbach’s alpha

These internal consistencies suggest that the tests are reliable measures. The slightly lower

alpha for lateral flexibility is not unexpected. Creativity, by definition, cannot be measured with

the same degree of precision as numerical reasoning, for example. The initial categories that

we have established are open to some interpretation by scorers, and this is likely to lead to

some variability in scores.

KCP Potentia Series


INTERCORRELATIONS

Test scores from a composite group of undergraduates, graduates and managers were used

to compute the following intercorrelation matrix for the tests Verbal Reasoning, Numerical

Reasoning and Creative Problem Solving.

Verbal Numerical C.P.S. C.P.S. C.P.S.

Reasoning Reasoning Fluency Originality Lat.Flex

Verbal

Reasoning 1.00 0.58 0.17 0.15 0.18

Numerical

Reasoning 0.58 1.00 0.38 0.28 0.37

C.P.S

Fluency 0.17 0.38 1.00 0.68 0.85

C.P.S.

Originality 0.15 0.28 0.68 1.00 0.74

C.P.S.

Lat.Flex 0.18 0.37 0.85 0.74 1.00

The intercorrelation of 0.58 between Verbal Reasoning and Numerical Reasoning indicates a

moderate relationship which suggests that there is a connection between these forms of

critical reasoning ability. However the relationship is sufficiently weak to indicate that the two

tests are still measuring different abilities.

There are stronger relationships between the Creative Problem Solving scores resulting from

the three different scoring strategies. In particular, there is a very strong relationship (0.85)

between Fluency and Lateral Flexibility in spite of a fair degree of independence in the

scoring strategies themselves. The strength of the relationships between Fluency and the

other two scores cannot be fully accounted for by the fact that a greater number of responses

will mean a greater number of different categories represented. The high intercorrelations

indicate that the capacity for producing original ideas is linked to the individual's fluency in

producing ideas generally, and that lateral flexibility is related to both of these.

It may be concluded that the scores together are providing an overall measure of creative

production, in the same way that test scores form a range of reasoning tests taken together

may be said to provide an indication of general intelligence.

KCP Potentia Series


VALIDITY

The validity of a test is the degree to which it measures what it is supposed to measure. In

occupational settings therefore, we are generally interested in whether a particular test is

capable of measuring some attribute which we know to be relevant to effective job

performance.

Face validity

This form of validity is to do with appearances. Managers and graduates often complain that

a test that they have undertaken doesn't seem to represent the real world. This is most often

the case when they have been confronted with a test which consists of discrete, unrelated

items. In these circumstances, candidates may feel less happy about the relevance of the test

results.

Potentia, like its pioneering relative the Supra Series, has been carefully designed with face

validity in mind. The tests are based on a scenario which has a realistic feel. The scenario,

however, is relatively neutral. That is, we deliberately avoided producing tests based on

information which might in some way put some candidates at an advantage or a

disadvantage.

Content validity

A test is said to have content validity if the items are representative of the attributes to be

measured. With Potentia, we drew upon our experiences of over 20 years in assessing

graduates and managers, consciously writing items were in line with job analytic studies

which revealed the kind of abilities that graduates and managers were asked to employ.

Concurrent validity

This is the form of test validity which is most frequently reported by test publishers. It is

concerned with collecting test and job performance data (often called a "criterion measure")

at the same time. The relationship between the two is calculated to see whether, for example,

people who score better on the test also gain better performance ratings.

With Potentia, we conducted a study with a sample of managers in a local authority. These

managers came from a broad range of disciplines, from personnel through to technical

functions, and ranged from junior to senior management. Each manager completed Verbal

Reasoning, Numerical Reasoning and Creative Problem Solving. Independently, ratings of

their job performance were obtained.

The results were very encouraging. Managers' overall performance was correlated at the 5%

per cent level of statistical significance (that is, we can be 95% confident that the relationship

has not occurred by chance) for Numerical Reasoning, Fluency and Lateral Flexibility, and at

the 10% level for Verbal Reasoning.

KCP Potentia Series


These relationships are likely to be an underestimate of the tests true validity given that the

sample was restricted in range. We have not corrected for this restriction since the sample

population made up a fair proportion of the initial norm group.

Additional studies of construct and concurrent validity are on-going and users are encouraged

to let KCP know if they can assist with similar studies.

Equal opportunities

KCP is committed to an on-going research programme in the area of equal opportunity.

During the trialling of Potentia we were able to collect data from bother sexes and from ethnic

minority populations. However, this data were not large enough for us to draw any firm

conclusions, and we would welcome the opportunity to collaborate with users in extending our

knowledge base on any sex or ethnic differences in the use of Potentia.

INTERNATIONAL POTENTIA ABILITY TEST MANUAL (AUG07) SERIES TEST MANUAL 2009.pdf · KCP Potentia...

Documents

Transcript of INTERNATIONAL POTENTIA ABILITY TEST MANUAL (AUG07) SERIES TEST MANUAL 2009.pdf · KCP Potentia...