Scaling Design

Chapter 8

Cooper & Schindler

Scaling Defined

• Procedure for the assignment of numbers to a property of objects in order to impart some of the characteristics of numbers to the properties in question

• We assign numbers to indicants of the properties of objects– Use mercury to measure temperature– Peer group to rate a person’s supervisory capacity

Scale Classification

• Study objective

• Response scales

• Degree of preference

• Scale properties

• Number of dimensions

• Scale construction

Study Objective

• Measure the characteristics of the respondents who complete it– Combine each respondent’s answers to form an

indicator of that respondent’s conservatism or political orientation

• Use respondents as judges of the objects or stimuli present to them– Use the same data but are interested in how

respondent view different things

Response Scales*

• Categorical scales– Used when respondents score some object without direct

reference to other objects

– Asked to rate the styling of new autos on a five-point scale

– Rating scales

• Comparative scale– The respondents are asked to choose which one of a pair

of cars has more attractive styling

– Ranking scales

Degree of Preference

• Preference measurement– Respondents are asked to choose the object eac

h favors or solution each would prefer

• Nonpreference measurement– Respondents are asked to judge which object ha

s more of some characteristic or which solution takes the most resources, without reflecting any personal preference toward objects or solutions

Scale Properties

• Nominal

• Ordinal

• Interval

• Ratio

Number of Dimensions• Unidimensional scale

– Seek to measure only one attitude of the respondent or object– One measure of employee potential is promotability

• Multidimensional scaling– Object is described in attribute space of n dimensions– The employee promotability variable is expressed by three dist

inct dimensions• Managerial performance• Technical performance• Team work

Scale Construction (I)*

• Arbitrary approach– Scale is developed on an ad hoc basis

• Consensus scales– A panel of judges evaluates the items to be included in the instrument

based on relevance to the topic area and lack of ambiguity

• Item analysis– Individual items are developed for a test that is given to a group of

respondents– After administering the tests, total scores are calculated– Individual items are then analyzed to determine which ones

discriminate between persons or objects with high total scores and low total scores

Scale Construction (II)*• Cumulative scales

– Are chosen for their conformance to a ranking of items with ascending and descending discriminating power

– The endorsement of an item representing an extreme position results in the endorsement of all items of less extreme positions

• Factor scales– Are constructed from intercorrelations of items– Common factors account for the relationships– The relationships are measured statistically through factor anal

ysis or cluster analysis

Rating Scales

• Judge properties of objects without reference to other similar objects

• Examples– Like-dislike– Approve-indifferent-disapprove

• There is little conclusive support for choosing scale point– The most widely used scales range from three to

seven points

Graphical Rating Scale• The rater checks his or her response at any point

along a continuum• The second scale (II) illustrate the graphic principle

but is not a good scale• Other graphic rating scales use pictures, icons, or

other visual to communicate with the rater• Variation 1• Variation 2• Other Variations

“How well does the employee get along with co-workers?” (Place an X at the position along the line that best reflects your judgment)

Never gets alongAlways gets along

Employee Evaluation Form

“How well does the employee get along with co-workers?” (Please check)

Always getsalong well

Sometimes hastrouble

Often hastrouble

Always at oddswith someone

(I)

(II)

Variation 1. Boxes replace the line and make it more certain that respondents will choose only one of four points

Variation 2. Two polar positions shown with a number scale used toshow degree of opinion

Gets along well Has trouble

1 2 3 4 5

Always getsalong well

Sometimes hastrouble

Often hastrouble

Always at odds withsomeone

Itemized Scale

• Presents a series of statement from which respondents select one as best reflecting their evaluation

• The judgments are ordered in a progression of a property

• This form is more difficult to develop

• It provides more information and meaning to the rater

• It increases reliability because more detailed statement help respondents develop and hold the same frame of reference while they fill them out

How well does the employee get along with co-workers?__ Almost always involved in some friction or argument with a co-worker__ Often at odds with one or more co-workers. The frequency of involvement is clearly above that of the average worker__ Sometimes gets involved in friction. The frequency of involvement is about equal to that of the average worker__ Infrequently becomes involvement in friction with others, definitely less often than most workers__ Almost never gets involved in friction situations with other workers

Problems in Using Rating Scales (I)

• Leniency– occurs when a respondent is either an easy rater or a hard

rater– Design the rating scale to anticipate it

• Central tendency– Raters are reluctant to give extreme judgments– Efforts to counteract this error

• Adjust the strength of descriptive adjectives• Space the intermediate descriptive phrases farther apart in

graphic scales

Problems in Using Rating Scales (II)

– Efforts to counteract this error• Provide smaller differences in meaning between steps near the

ends of the scale than between the steps near the center

• Use more points in the scale

• Halo effect– Is the systematic bias that the rater introduces by carrying

over a generalized impression of the subject from one rating to another

– Halo is a pervasive error– It is difficult to avoid when the property being studied is not

clearly defined

Ranking Scales• Compare two or more objects and make choices among the

m

• Vote-splitting– 40 percent choose model A; 30 percent choose model B; 30 percen

t choose model C

– 60 percent of respondents chose some model other than A; perhaps all B and C voters would place A last, preferring either B or C to it

• Ranking– Method of paired comparisons

– Method of rank order

– Method of successive intervals (X)

Pair Comparisons

• The respondent can express attitudes unambiguously by choosing between two objects

• Generally, there are more than two stimuli to judge

• The number of judgments requirement is

N=n(n-1)

2

N = Number of judgmentsn = Number of stimuli or objects to be judged

Paired-Comparison Problems• The respondent will get tired and give ill-considered answers or

refuse to continue• Reducing the number of comparisons per respondent without red

ucing the number of objects being studied can lighten this burden– Present each respondent with only a sample of this stimuli; each pair of

objects must be compared an equal number of times– Another is to choose a few objects that cover the range of attractiveness

at equal intervals; all other stimuli are then compared to these few standard objects

• Example– 36 employees are to be judge, 4 may be selected as standards and the others di

vided into four groups of eight each

Pair-Comparison Data Analysis• Transitivity

• Union bargaining committee is considering five major demand proposals– Low of Comparative Judgment– Guilford’s “Composite-standard” method

A B C D E

ABCDE

--- 164 138 50 7036 --- 54 14 3062 146 --- 32 50150 186 168 --- 118130 170 150 82 ---

Total 378 666 510 178 268

Method of Rank Order• Ask respondents to rank order their choices

• This method is faster than paired comparisons and is usually easier and more motivating to the respondent

• With 7 items, for example

• Drawbacks– respondents may grow careless in ranking 10 or more

items– The rank ordering is still an ordinal scale with all of its

limitations

Rank Order Data Analysis• There are several simple ways to combine ranking into an

overall index

• Means cannot properly be calculated, but it is possible to compute medians

• The sum of rank values will probably give the best simple indication (?)

• Translate ordinal rank data into an interval scale– Normalized-rank

– Comparative-judgment

• Complete ranking is sometimes not needed

Scale Construction Techniques

• Arbitrary scales

• Consensus scales– Differential scales

• Item analysis– Likert scales

• Cumulative scales– Guttman scalogram

• Factor Scales– Semantic Differential

Arbitrary Scales

• Collect several items that are unambiguous and appropriate to a given topic

• Company image example

• Easy to develop, inexpensive, and can be designed to be highly specific– The design approach is subjective

• Data analysisHow do you regard (name)company’s reputation1. As a place to work? Bad ___ ___ ___ ___ ___ Good2. As a sponsor of civic project? Bad ___ ___ ___ ___ ___ Good3. For ecological concern? Bad ___ ___ ___ ___ ___ Good4. As an employer of minorities? Bad ___ ___ ___ ___ ___ Good

Consensus Scaling1

• Require that the item are selected by a panel of judges who evaluate them on– relevance to the topic area– potential for ambiguity– the level of attitude it represents

• Thurstone differential scales– Known as method of Equal Appearing Intervals– Developed to create an interval rating scale for attitu

de measurement

Consensus Scaling2

• Development Procedures– Often 50 or more judges evaluating a large number of

statements (one statement per card)• Expressing different degrees of favorableness toward an object

– The judges sort each card into 1 of 11 piles• Representing their evaluation of the degree of favorableness that

the statement expresses• The judge’s agreement or disagreement with the statement is not

involved

– Three of the 11 piles are identified to the judges by labels of “favorable” “unfavorable” and “neutral” (1,6,11)

Consensus Scaling3

– The scale position for a given statement is found by calculating its median score

– A measure of dispersion is calculate for each statement (interquartile range)

• If a given statement has a large interquartile range, it is judged to be too ambiguous to be used in the final scale

– Those with median scores spread evenly from one extreme to the other and with small interquartile are included in the final attitude scale

Scale administration1

• Respondents read approximately 20 statements and select those items with which they agree

• The mean or median value of the chosen scale items is then calculated as the measure of the respondent’s attitude

• Example is part of a classic 50-item scale that was designed to reveal the attitudes of employees toward their employer– The scale value are shown here but would not be on the

instrument when it is used

Scale administration2

• Statements are arranged in random order of scale value

• The typical will choose one or several adjoining items (in terms of scale values)– If the values are valid and if it deal with only one attitude

dimension

• Divergences occur because a statement appears to tap a different attitude dimension

• Differential scales are reliable, more widely used in academic studies

• The method is costly, and time-consuming

Scale Value10.4 I think this company treats its employees better than any other company does8.9 A person can get ahead in this company if he or she tries8.5 The company is sincere in wanting to know what its employees think about it5.4 I believe accidents will happen no matter what you do about them5.1 The workers put as much over on the company as the company puts over on them4.1 Soldiering on the job is on the increase2.9 My boss gives all the breaks to his lodge and church friends2.5 I think the company goes outside to fill good jobs instead of promoting people who are here1.5 In the long run this company will “put it over” on you1.0 The pay in this company is terrible

Item Analysis• Evaluate an item base on how well it discriminates betwee

n those persons whose total score is high and those whose total score is low

• The most popular type is the summated scale– consist of statements that express either a favorable or unfavorabl

e attitude toward the object of interest– The respondent is asked to agree or disagree with each statement– Each response is given a numerical score– Total scores are total to measure the respondent’s attitude

• The most frequently used form is the Likert scales

Likert scale

I consider my job rather unpleasant

Strongly Agree Neither Agree Disagree Strongly Agree nor Disagree Disagree (1) (2) (3) (4) (5)

Likert Scale Construction1• The first is to collect a large number of statements that meet

two criteria– Each statement is believed to be relevant to the attitude being studied– Each is believed to reflect a favorable or unfavorable position on that

attitude

• Each person’s responses are then added to secure a total score• The next step is to array these total scores and select some

part of the highest and lowest total score– The top 25 percent and the bottom 25 percent are used criteria by

which to evaluate individual statement (Table 7-2)

Likert Scale Construction2

• Item analysis involves calculating the mean scores for each scale item among the low scores and high scorers

• The item means between the high-score group and the low-score group are then tested for significance by calculating t values

• Finally, the 20 to 25 items that have greatest t values are selected for inclusion in the final scale– Edwards suggests using only those statements whose

t value is 1.75 or greater , provided there are 25 or more subjects in each group

Advantages of Likert Scales• It is easy and quick to construct

• Each item has met an empirical test for discriminating ability

• More reliable than the Thurstone scale

• Provide a greater volume of data than does the Thurstone differential scale

• Easy to use both in respondent-centered and stimulus-centered studies

• It is also treated as an interval scale

• On can develop scales of these types in an arbitrary manner

Cumulative Scales

• The major scale of this type is the Guttman scalogram

• Scalogram analysis is a procedure for determining whether a set of items forms a unidimensional scale– A scale is said to be unidimensional if the respons

es fall into a pattern in which endorsement of the item reflecting the extreme position results also in endorsing all items that are less extreme

1. Style X is good looking2. I will insist on style X next time because it is great looking3. The appearance of style X is acceptable to me4. I prefer style X to other styles

Ideal Scalogram Response Pattern

Item

2 4 1 3 Respondent Score

X X X X 4-- X X X 3-- -- X X 2-- -- -- X 1-- -- -- -- 0

Guttman Scale Development1

• One first defines the universe of content– Viewer attitude toward TV advertizing

• The second step is to develop items that can be used in pretest that tell us if this topic is scaleable– Guttman suggests that a pretest include 12 or more

items, while the final may have only 4 to 6 items– Pretest respondent numbers may be small, but final

scale use should involve 100 or more respondents

Guttman Scale Development2

• Take the pretest results and order the respondents from top to bottom– Most favorable to the least favorable

• The next step is to discard those statements that fail to discriminate well between favorable and unfavorable respondents

• Finally calculate a Coefficient of Reproducibility (CR)

• CR should be 0.90 or better for a scale to be considered unidimensional

Reproducibility = 1 -

n(N)

is the number of errorsn is the number of itemsN is the number of cases

Factor Scales• Include a variety of techniques that have been develo

ped for two problems– How to deal with the universe of content that is multidime

nsional– How to uncover underlying dimensions that have not bee

n identified

• These techniques are designed to intercorrelate items so their degree of interdependence may be detected

• Semantic differential is based on factor analysis

Semantic Differential

• Developed by Osgood and his associates– Is to measure the psychological meaning of an

object to an individual

• Is based on the proposition that an object can have several dimensions of connotative meaning– The meaning are located in multidimensional pr

operty space, called semantic space

: : : : : :Good Bad

Semantic Differential2

• The scale developers produced a long list of adjective pairs useful for attitude research

• They chose 20 concepts with the psychological meaning they wished to probe

• Some concepts illustrated– Person concepts– Abstract concepts– Event concepts– Institutions– Physical concepts

• Three factors contributed most to meaningful judgment by respondents

Semantic Differential3

– Evaluation– Potency or power– Activity

• Occasionally, the potency and activity dimensions combined to form dynamism

• Lesser dimensions– Stability– Tautness– Novelty– Receptivity

Semantic Differential Development

• The first step is to select the concepts to be studied– Nouns, noun phrases, nonverbal stimuli(visual sketches)– Concepts are chosen by judgment and reflect the nature of

the problem under study

• Select the original bipolar word pairs or tailor-made scales– Three criteria should guide their selection

• The factors composition (Table 7-4)• The scale’s relevance to the concepts being judged• The scale should be stable across subjects and concepts

(E) Sociable(7)(P) Weak(1)(A) Active(7)(E) Progressive(7)(P) Yielding(1)(A) Slow(1)(E) True(7)(P) Heavy(7)(A) Hot(7)(E) Unsuccessful(1)

Unsociable(1)Strong(7)Passive(1)Regressive(1)Tenacious(7)Fast(7)False(1)Light(1)Cold(1)Successful(7)

SD Scale for Analyzing Candidates for an Industry Leadership Position

SD Illustration

Advanced Scaling Techniques

• Multidimensional scaling – Describe a collection of techniques that deal wit

h property space in a more general manner than the semantic differential

• Conjoint analysis– Used to measure complex decision making that

requires multiattribute judgements