JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR

CHOICE BETWEEN REWARDS DIFFERINGIN AMOUNT AND DELAY: TOWARD ACHOICE MODEL OF SELF CONTROLLEONARD GREEN AND MARK SNYDERMAN

WASHINGTON UNIVERSITY

A concurrent-chain procedure was used to study pigeons' choices between rewards differingin both amount and delay. The shorter delay terminated with a 2-second access to grainwhereas the longer delay terminated with a 6-second access to grain. The ratio of thedelays was constant within a given condition while their absolute values were varied.Over conditions, ratios of 6:1, 3:1, and 3:2 were studied. As the absolute values of thedelays to reinforcement increased, preference for the longer-delayed but larger rewarddecreased under both the 6:1 and 3:1 ratios, but increased under the 3:2 ratio. Theseresults are inconsistent with choice models predicting no change in preference when theratios of delays and amounts are held constant. In addition, the change in preferenceunder the 3:1 ratio is inconsistent with a simple multiplicative interaction of the trade offbetween reinforcer amount and delay, and suggests that delay is a more potent de-terminant of choice than is amount. These results have implications for models thatview choice between small immediate rewards and large but delayed rewards as under-lying the behavior commonly called self control.Key words: choice, self control, concurrent-chain, matching, amount, delay, key peck,

pigeons

This experiment is concerned with choicesbetween outcomes that differ both in rein-forcer amount and the delay to those rein-forcers. Although most studies of choice behav-ior have varied only one parameter (eitherreinforcer amount or delay) and held the otherconstant, few have addressed changes in choicewhen both variables are manipulated. Suchresearch has implications for models that viewthe choice between a small immediate rewardand a large but delayed reward as underlyingthe behavior commonly referred to as self con-trol. We will defer discussion of that topic untilthe end of the paper.In a study on the effect of delay of rein-

forcement on choice, Cliung and Herrnstein(1967) had pigeons choose between two equal,concurrently available, variable-interval sched-ules that varied in the duration of blackout(delay) imposed before reinforcer delivery.

Supported by NIMH grant 1 RO 3 MH32655-01 andSigma Xi Grant-in-Aid of Research to Leonard Green.Our thanks to the members of our lab group for theiroutstanding assistance in the running of the experi-ment. Mark Snyderman is now at the Department ofPsychology and Social Relations, Harvard University.Reprints may be obtained from Leonard Green, De-partment of Psychology, Washington University, St.Louis, Missouri 63130.

They observed matching of relative rate of re-sponse for a given alternative to the relativeimmediacy of reinforcement provided by thatalternative. That is, the pigeons distributedtheir responses in inverse proportion to therelative delays of reinforcement. The studyprovided evidence for delay of reinforcementas a parameter similar to rate of reinforce-ment in its effects on the distribution of re-sponses. That is:

R, d2R1 + R7 dI + d2

=±r2r, + r2

(1 a)

(1 b)

where R1 and R2 are responses to the twoalternatives, d, and d2 are the delays to theoutcomes, and r, and r2 are the rates of rein-forcement in the two alternatives. The match-ing law can also be represented as simple ra-tios:

R1 _ d2

r,r2

(2a)

(2b)

Results inconsistent with matching havebeen obtained in studies of delay of reinforce-

135

1980, 34.,135-147 NUMBER 2 (SEPTEMBER)

CHOICE BETWEEN DIFFERENTIAL REWARDS

ment using a concurrent-chain procedure in-volving choice between different terminal-linkfixed intervals (FI) (Herrnstein, 1964; Killeen,1970; MacEwen, 1972; Williams & Fantino,1978). In particular, MacEwen (1972) foundthat with a constant ratio between the Flschedules, preference for the key associatedwith the shorter terminal link increased withincreasing absolute Fl values. The matchinglaw based on the ratio measure predicts con-stant preference at all FI values as long as theratio of the Fl values remains constant.Many of the results from concurrent-chain

studies can be accounted for by a model pro-posed by Fantino (1969):

R_ __ T-t, (3)R1+ R2 (T-tl) + (T-t2)

(when t1 < T, t2 < T)= 0 (when t1> T, t2< T)= 1 (when t2> T, t1 < T)

where R1 and R2 are responses on the two al-ternatives during the initial link, T is theaverage time to primary reinforcement fromthe onset of the initial link, and t1 and t2 arethe two terminal-link intervals. Preference isthus determined by the reduction in delay toprimary reinforcement signaled by entry intoone terminal link relative to the reduction indelay to primary reinforcement signaled byentry into the alternative terminal link. Inthe MacEwen (1972) study, the average timefrom onset of the initial link to entry into aterminal link was constant. Therefore, therelative reduction in delay provided by theshorter terminal link increased with the abso-lute value of the Fl schedules even though theratio of terminal-link Fl's remained constant.In addition, Equation 3 appears consistentwith the data from Chung and Herrnstein(1969). A reanalysis of their data revealed thatpreference for the key associated with theshorter delay increased, the greater the abso-lute value of the delays (Williams & Fantino,1978).The matching law is better able to account

for data from studies on choice and reinforceramount. Catania (1963) found that pigeonsmatched their relative rate of response to therelative duration of reinforcement. With bothconcurrent and concurrent-chain schedules,both matching (Brownstein, 1971; Iglauer &Woods, 1974; Neuringer, 1967; Schwartz, 1969;

Ten Eyck, 1970) and undermatching (Pliskoff& Hawkins, 1967; Schneider, 1973; Todorov,1973; Walker & Hurwitz, 1971; Walker,Schnelle & Hurwitz, 1970) to reinforcer amounthave been obtained. In undermatching, prefer-ence for the alternative leading to the largerreinforcer is less than predicted by matching.

Such undermatching is handled by Baum's(1974) generalized power-ratio matching anal-ysis (applied to reinforcer amount):

(4)

where R1 and R2 are responses, a, and a2 arereinforcer amounts delivered in the two al-ternatives, and k is a measure of bias. A kvalue other than 1.0 indicates preference forone alternative when apparent equality ofreinforcement would lead to a prediction ofindifference. x is the slope of the line relatinglog response ratio to log amount ratio. x = 1represents perfect matching while x < 1 cor-responds to undermatching, a common findingin choice studies with reinforcer amount.Of greater interest, however, are the effects

that variations in both delay and reinforceramount have on choice behavior. Scant atten-tion has been paid to this area of research(Logan, 1965; Logan & Spanier, 1970; Navarick& Fantino, 1975, 1976; Rachlin & Green, 1972;Green, Fisher, Perlow, & Sherman, Note 1).Rachlin and Green (1972) adopted a matchingmodel to account for choice when both vari-ables are manipulated. Reinforcer delay andamount were assumed to contribute equallyto the "value" of the reinforcer in much thesame way as rate of reinforcement does underthe matching law. That is,

V (d2 (a, _ R1V2 'd'J a2) R2 (5)

where V is the value of an alternative, d isthe delay of reinforcement, a is reinforceramount, and R is the number of responses tothat alternative. Preference is inversely relatedto the ratio of reinforcer delays and directlyproportional to the ratio of amounts, and isunaffected by the absolute values of theseparameters.Navarick and Fantino (1976) tested Equa-

tion 5 in two experiments using pigeons re-sponding under a concurrent-chain procedurewith Fl terminal links. In both experimentsthe initial links were equal variable-interval

136

R,= . a, 9

R2 a2

L. GREEN and M. SNYDERMAN

(VI) schedules, and the amount of rewardprovided by one terminal link was three timesmore than in the other terminal link (a1/a2 =3). In their first experiment, the terminal-linkFl schedules were increased in absolute valuewhile a constant difference of 10 sec was main-tained between them. The longer-delay al-ternative was always associated with the largerreinforcer amount. d2/d1 thus approached 1.0as the absolute value of the delays increased.As predicted by Equation 5, preference for thekey associated with the larger reinforceramount tended to increase somewhat with ab-solute delay.

In their Experiment 2, the delays to rein-forcement were equal in the terminal links(d2/d1 = 1.0), but the absolute value of thesedelays was increased. Here Equation 5 pre-

dicts a constant preference, in the ratio ofa, /a2, for the key associated with the largerreinforcer amount. However, Navarick andFantino found that preference for the largeramount terminal link increased with the abso-lute value of the delays.

If delay and amount of reinforcement foreach terminal link are converted to terminal-link seconds per unit-time access to food, thesenew values can be substituted for t1 and t2 inEquation 3, on the assumption that delayand amount combine multiplicatively and are

equivalent in their effects upon choice. Afterthe appropriate conversions, Equation 3 pre-

dicts the observed increase in preference forthe key leading to the larger reinforceramount.'

Navarick and Fantino (1976) varied absolutedelays of reinforcement while keeping the de-lay ratio constant at 1.0. Equations 3 and 5make quite different predictions if the delay*ratio is other than unity. The present experi-ment utilized a concurrent-chain procedurewith equal variable-interval (VI) initial linksand varied the Fl terminal links. The ratioof reinforcer amount was held constantthrouglhout at 3:1 as in Navarick and Fan-tino (1976); the absolute delays to reinforce-ment were varied while delay ratio was heldconstant at 6:1, 3:2, or 3: 1.Under a 6:1 delay ratio, Equation 5 predicts

'It can be seen that this experiment has now becomeequivalent to MacEwen's (1972). The converted termi-nal-link delays differ by a constant ratio of 3:1 as theabsolute values of the delays increase.

that the pigeons will prefer the smaller re-ward twice as much as the more delayed,greater reward, and that this preference willremain constant as the absolute delays arevaried. Equation 3 predicts that preference forthe key associated with the longer delay willdecrease as the delays increase.Under a 3:2 delay ratio, Equation 5 now

predicts a preference for the larger rewardwhich does not change as delays are changed,while Equation 3 predicts increasing prefer-ence for the larger delay as the delays increase.Both models make identical predictions for

responding under a 3:1 delay ratio. The Rach-lin-Green formula predicts indifference (rela-tive rate = .50) between the two outcomes.After the appropriate conversions, Equation 3also predicts indifference at all delays.

METHOD

SubjectsEight male, White Carneaux pigeons were

maintained at 80% of their free-feeding bodyweights. All had previously served as subjectsin an autoshaping experiment (Green &Schweitzer, 1980). Water and grit were con-tinuously available in their home cages. Fourwere studied at 6:1 and 3:2 delay ratios and adifferent four at the 3:1 ratio.

ApparatusA sound-insulated, pigeon test chamber (Ger-

brands Co.) measured 36.2 cm long by 33.7cm wide by 40.6 cm high. The two responsekeys were 1.9 cm in diameter, 8.9 cm from eachside wall, and 21.6 cm from the grid floor. Eachkey could be rear-illuminated with white, red,or green light, and required a force of at least10 g (.10 N) to operate and produce a feedbackclick. General chamber illumination was pro-vided by two 7-W white lights mounted on theceiling. The reinforcer was limited access tomixed grain via a food hopper situated be-tween and below the response keys. Duringreinforcement, all lights in the chamber wereextinguished, and the food hopper was ele-vated and lighted by two 7-W white bulbs.White noise was continuously present and anexhaust fan provided ventilation. All sched-uling and recording were performed by elec-tromechanical equipment located in an ad-joining room.

137

CHOICE BETWEEN DIFFERENTIAL REWARDS

ProcedureA concurrent-chain procedure with variable-

interval 1-min initial-link schedules and vari-ous fixed-interval terminal-link schedules was

used. During the initial choice link, both re-

sponse keys were transilluminated with whitelight and entrance into the terminal links was

arranged by two independent VI timers on a

concurrent variable-interval 1-min variable-interval 1-min schedule (conc VI 1-min VI 1-min). The intervals were derived from thedistribution of Fleshler and Hoffman (1962).When entrance into one of the mutually ex-

clusive terminal links was scheduled by eitherVI timer, it stopped, but the other VI timercontinued to operate. The next response on

the appropriate key produced the terminal-link stimulus (either red or green illumina-tion) associated with that key, and the otherkey was darkened and inoperative. Furtherresponding on the lighted key produced thereinforcer according to the requirements of theFl schedule. During a terminal link, bothVI timers stopped. The relative rate of re-

sponding on the two initial-link keys was themeasure of preference for the terminal-linkoutcomes.Food was delivered in the terminal links ac-

cording to various fixed-interval schedules.The terminal links differed with respect to thefixed-interval schedule and the duration of ac-

cess to food. At the end of the food-deliveryperiod, the initial-link keys were both reil-luminated with white light and both VI timersresumed operation. If, however, both VI tim-ers had programmed entrance into their re-

spective terminal links, the terminal link notentered remained available upon return to theinitial link.

Sessions were conducted seven days a week,with each session lasting until 40 food rein-forcers were obtained. Each bird remained inan experimental condition until the followingstability criteria were attained: (a) the bird was

on the condition for at least 30 days; (b)median relative rates of responding for thelast three successive blocks of five sessions eachshowed neither an upward nor a downwardtrend; and (c) there was no visible trend ineither relative or absolute rates of respondingduring the final five days.The absolute delays of reinforcement (Fl

schedules in the terminal links) were varied

while the ratio of the delay values remainedconstant within an experimental condition.Two different reinforcer durations were used,2-sec and 6-sec access to food, with the longerhopper duration associated with the greaterFl delay. Tables 1, 2, and 3 summarize theconditions and their order of presentation.

Experimental ConditionsCondition 6:1: The FI values were in the

ratio 6:1. The actual pairs of FI values were:12 vs. 2 sec, 24 vs. 4 sec, 60 vs. 10 sec, and 120vs. 20 sec. Birds 23, 24, 25, and 26 were eachstudied on at least three of these pairs. ForBirds 23 and 24, the left terminal link wassignaled by a green keylight and the rightterminal link by a red keylight. The reversewas true for Birds 25 and 26.

Condition 3:1: The FI values were in theratio of 3:1, and the actual values were: 6 vs.2 sec, 30 vs. 10 sec, 60 vs. 20 sec, and 120 vs. 40sec. Birds 10, 15, 27, and 28 were each studiedon at least three of these pairs. The key wasgreen during the left terminal link and redduring the right terminal link.

Condition 3:2: The pairs of Fl delays werein a 3:2 ratio. The actual pairs of FI valueswere: 6 vs. 4 sec, 30 vs. 20 sec, 60 vs. 40 sec,and 120 vs. 80 sec. Birds 23, 24, 25, and 26were each also studied on at least three of thesepairs. For Birds 23 and 24, red and green key-lights signaled the left and right terminallinks, respectively, whereas the reverse heldfor Birds 25 and 26.

Preliminary ConditionsPrior to these experimental conditions, there

were two preliminary conditions. In the first,birds chose between equal hopper durations(2 sec) and equal FI's in the terminal links(Condition 6:1: Fl 2-sec vs. FI 2-sec; Condi-tion 3:1: FI 2-sec vs. FI 2-sec; Condition 3:2:Fl 4-sec vs. FI 4-sec). This tested key bias.

In the second preliminary condition, thelength of the Fl in the preferred terminal linkwas increased to the appropriate delay ratiowhile food duration remained constant at 2sec for each link. Thus, for Condition 6:1,the pigeons now chose between FI 12-sec andFI 2-sec; for Condition 3:1, FI 6-sec vs. FI 2-sec; and for Condition 3:2, Fl 6-sec vs. FI 4-sec. This determined whether the birds coulddiscriminate the smallest programmed differ-ence between the FI's in the terminal links.

138

L. GREEN and M. SNYDERMAN

All pigeons showed an appropriate increasein preference for the terminal link associatedwith the shorter delay.

Following this determination, hopper dura-tion was increased to 6-sec in the terminal linkassociated with the longer FI (hopper dura-tion remained at 2 sec in the other link). Al-though this was the first experimental con-dition, it also determined whether the birdscould discriminate the difference in reinforcerdurations (2 sec vs. 6 sec). All birds showed anincreased preference for the terminal link nowassociated with the 6-sec reinforcer duration.

RESULTSAbsolute and relative response rates in the

initial link as well as terminal-link responserates for each pigeon are given in Tables 1(6:1 delay ratio), 2 (3:1 delay ratio), and 3(3:2 delay ratio). These data are means fromthe last 5 sessions of each condition. Also pre-sented are the programmed Fl's and the ob-tained mean delays to reinforcement in eachterminal link (average time from entry intothe terminal link until the reinforced re-sponse). Although these values are generallygreater than the programmed FI's, the actualdelay ratios did not differ by more than 5%from the programmed ratios.

Figure 1 shows the relative rate of responseduring the initial link (proportion of re-sponses on the initial-link key leading to thelonger terminal link) as a function of the FI'sin the terminal links for each bird in the threedelay-ratio conditions.

In the first preliminary condition, the Fl'sin both terminal links were equal, as was theduration of access to grain (2 sec). The rela-tive initial-link response rate was near indiffer-ence (.50) in 9 of 12 cases. Although 3 birdsdid display bias, there was no consistency inkey preferred (Birds 26 and 27 preferred theleft key; Bird 10 preferred the right key). Un-der the second preliminary condition, an in-crease in delay to reinforcement associated withthe key preferred in the first prelinminary con-dition decreased preference for that key for allbirds. The magnitude of preference changevaried from .063 to .390. The greatest de-creases in preference occurred for birds underthe 6:1 delay ratio. There was no apparentdifference in the amount of decrease betweenbirds in the 3:1 and those in the 3:2 delay-

ratio conditions. Clearly, all subjects weresensitive to the smallest difference in delaysto reinforcement.

Sensitivity to differential reinforcer amountswas demonstrated under the first experimentalcondition. Delay to reinforcement remainedas in the previous condition, but access to grainwas increased from 2 sec to 6 sec for the key

0

-W0.48z,0OA

o 0.2

c OLc

<0

; 1.0

uJ 0.8

O 0.6

ofy 0.4

0 0.2ui

O

> 1.0

3~0.8ui

0.60.4

0.2

6:1 Delay Ratio

3:2 Deloa

. I - a

4:4 6:4 6:4PreliminaryConditions

* 230 24* 25o 26

y Ratio , '

* 230 24* 25o 26

. NNNNNNNNNNNNNNNNNNNNNNNNNNNNNINNNNNNNNNNNNNNNNNNNNNNNNNNNNNINNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm30:20 60:40 1280lExperimentalConditions

TERMINAL LINK FI LENGTHS (sec)Fig. 1. Relative rate of response for the longer termi-

nal link as a function of the duration of the terminallinks for each bird under each delay ratio. Predictionsfrom Rachlin and Green (1972) and Fantino (1969)are represented by dotted and dashed lines, respec-tively. (Both models make the same predictions underthe 3:1 delay ratio, represented by the alternating dot-dash line.)

139

o 0 m I-, - e -Ic D - 0 C

io-c U)cCo

- CDO

S04 Xm mm C48 o14.i .l .l .l el

t- O t- AD - t- r + a c",- -4 -G 1- --1 "" t I--r14 I--, C-4 '-.O-C

I . . -.e

C1 "

tosbc

t,ul)mos0 h Log - c4m en o: c14 e4 r- 04 as t-4 .4 .6 .4 .; .' Lo LeM - -- "es I-- ~oD-N-__-,_--

_ -0 LO * 0E _ 4. C4 ¢-4

C.i .i R.

CCI 4 -

r_ r_ o 01 LOCI.r s§e t- 14un -O Lo eq 0%0 -

Lo_q toCO un N LOe OCJO CrC44 CC' 0iC C' §} 0- _ u

,_

. ~. . . . . . . . . . . . . . . . . . . . . V: . . . .

es-4 ~.D-~ I.t-, I-, enI.,en CO 1- CO _-1 -- LOI- i-,_d -to -00-cO- I-- m

cn"o48 .0o C)° 4>- 0Co> - _ 4 'n on cn 0 toLo C1 _. _c, ,_t ,1 d-, inCtom*noeoeooooUnoC_qR R5!8 i__._C CD_C C- _ _I _ _ _' -00 ., 1- _, -- -1 .; ., I;I.:,I.:,

0 000C~00 O O'0I.:,IC4.:,0OC4I.:, -400I.

_

in OX I-a,en C_ _ _ _C4-- -e_ cn-cn0.4c "NCOC C4 O4t4 eno0--@m- mt 0< *fSot >o ° C14fO Owo-0 I-oM o _sel, _N.D +Xd. - D0CC- N C-Ct- C

- CCII-.O O._C00-CcO-- Oe .i 4 O

_C_tC C4

0.Uc 0_______ C__ '_C_

e0 0-C OCO 4toI-,M-q 4 ..cnu

4no

%6 o4-.: C6 o;.

C4I- - C4I-C,4e 0t- 0+4 "tt1- 0e4 -CJOt-D GD-I "D ":%-* *vf XXv@CCI CC4 Co ID

C C C C C

C4 CCI

-i

+- tC-

- c e

C4 1- CD

0 0 0

CC4 C ~4

o1 04 04 0

in 0 .Ito i

CC,

%O

m - - LO -- - - CCI t- O}

0-i CC CC -4 6 6

4C~ ~ ~~~~4 C1t - 0

C C C4 C -

_ 04

O O

C-i CCio C 0 0

C 4 O O_ eC D 4

; CC) CC,

- C" cC tC un +4

4 C.4 1-4 CC4 C4

CO tD o

C0 CCI Cl C4 CC

CCI CCI Co MD Co

COD CO I- a0 D 0

.-C,!

08 cs es Oe_cs m C4 C4 1 4

. C-. C.4 0

_-CC4 C4 i-

0 0 0

C C CCC4

0

_48 C

_ gm .4

4 C

el'! CiC4 C4 CCI Id-

.- C-4

CCI C1 CC4 4.CC4

0M 0

CCI0 C

'0

CCI

0

'C4)

'E

A.

,.O.

[

.4

lm

4.54 4

-9

L .

-4 6.SIL

140

4)

.0

0 to

. -

CIn

- )

*.5

4)4

Co

0.

0>

-_

_

C)Cu

C

1.~

C-c

so

c .Y

o 9)4

Cu,

Cu C-- W

4)0

1.. .

- 4)

C4X)

'Cn.It

_s

1.c

II

I

11

I

11-

pc

C4c

:

c

.4---

ew4

C ' 01O C

1-C ouCM- eo

Cn U. "cmCl . iM 8 .D M 4- M 04

Ln C-4

in C14 *enCli li n -i Iq Vl.0 C,4 m m m'd, 00C-4 I-' 00 GO 1-1M04

R-4

ci, 1ci, 0 ci,. ci,' '- -0

on r-10- C44t

1-111- -. -OQ

--

C'4 -E0 0Sf" 0- I--,C,C.4 -'C.4 i,'u0il0e Me n-4a l 94' Ci4

4 COSiG tG0CICM:4 ~~0 0 0 i044 -mI40Si0c ,4 0

inr si,C.4 -4 kEto- Oin0 0'O.c ,- 4

aC' C.4 0o nC 4s2.0ii-iOR

0V~..I

I--,ci c,1 0-1 I--,~4 CI

&0%0o CAoo0 or'co 4'C'- C'4 O2-.0C.4'-.08

CC,- tO '-c,M -,# .

Q4'C'~VS 0t'.U)CO Sf~C'.C.40 C.4&4 'S' 04''O 4Ci,0 ~CO) &OC-4 oCi,O-- 0 c SIO. I... an s 4'. in & C4U. 0CC 0-gm~ 04 t-C-*t.O0- 0 004 ft

C~4 04 C4 C04

C04 C~4 %O 4O

c04 C4 C4 C

o o 0 0

V..V .

4'4'

00

*Si,~

C4 C4 e4 C4 C'4

Cd! Cd! CO C

C-4 C4 C -

Cd! Ci Cd! COR 4

o4 o 0o 0

o 0 0b 00 0

CM

C4 Cd4 CO CO ID

Cd 04 CE, C

Ci,C04

o1 oD 0 00

0d 04 C4

0 4 e

C'4

C'4

C4

Cd C C Cd4

Cd 0O CO km

C--

C44'

o1 04 00

oD 0 00

Cd Ci '

iaI

t1=1.W

1-I

-4

.I2., W

-6 L.r r....24

a3: 8

.mt- Ig 2.-I*,

0 0

94r. ia -

141

A

.-a6".-11 -

C4 -a;,3.

a ?D

. L.S:g,

iI .11.1 -V:*

I

i -41.?I

zz

'1.0

2,0

S-4

40

a

IS

la

Cd4

I

4

D

p

I

"I

I

.1-

Iq Iq CR Cli ci

04 C4

en 1- eq kmOR OR r,. cl %R

C's tI- Ito 4141. Go OIto0oC, C04

-CtI,) Cco 410 -4W~ ~ 6~

Ct)0 10It l-" I'll~ I'llIf

Ch 0Ct t) 0)0C- -0 )l-

co00 ) LO.-I 14-1t)-i a!

0)c u0 41;'- ct)-Ct)o

oo 10, LO ~ ~ 1140CD t Cl. 14101- 1

0IfM- ) 000-o

'-00 '-00 If)L 014 144

U) Lo) Lo Ct)M0040 C C-e-

0C; 0)-4 Ct)1-11 f

LO 'I*-oCD O0 1-0 00( LO, ct)m0 C r- D0o0DLOCD!CDo toIf) I-.COoC)% Zlo-00 C0 %oC o i

- 00-00M 4c Dt OC44 -'.- M4 4100

L00)0)- 00410)a0a- 10 0)44 '-.) '-Cl-00)

0-4 1f) 414',10-' 0)0)4 inU)' I- 000t 0)ot ) -0) 00-C4 0~I-'LO-) -10.I--I I- % I, %C1- 0 C4 014 I-'00- d1- -

OCt)9Cl-0041 0)410.1-11000 100)~~~~~~~~~~~~~~~~~~~~~~~~0.

10r,1110040I,)410I",ct0) 01 C.C-

0C~~~C.Ct)1000--Ct)- et)41-10a11-00 D d- C- o 9- C4)C"0 c-)t 0c

0)4

C41

Ct)

0

41

0)4 0)4 0)4 0)4 0)4

0)4

0- GoC

C l-~ C

1616~~~~~~~1

10) co 0

Ct) C4) Ct) Cf)

0)4 en) 4 It,

Ct)

0)4 0)4s e4 0C4 0)4

04

0 Ct) co-

0- 01

Ct)

0)4m e

-4 0)4 LO 141

0)4

04

41

0

10)

10)0)4

0)4 0)4 0)4 0)4

0)4

c c C I410)4 0)4 Go U,)

4C44

-d

0)4 410 in

0)4 C)4 eo4 04 0)4

)14 0)4

C1 0-

LO C41

41 '4

004

142

.0CUU

Ct)~

02

CU

>.1

Si

.0

04

I-4

U

"IN

C14

If

I4

S.,.O

a

>1

Is

4I

IIj

10.11h.-S

L. GREEN and M. SNYDERMAN

associated with the longer delay. Once again,there were large changes in relative rates of re-

sponse for all subjects: preference increased forthe longer delay key, which now providedlonger access to food. The magnitude of thepreference change varied from .108 to .304.The pattern of relative response rates withinthe remaining conditions under each delayratio was highly consistent among birds. Asthe delays to reinforcement in the terminallinks increased, the relative rate of response

in the initial link on the key associated withthe longer delay decreased under both the 6:1and 3:1 ratios, but increased under the 3:2ratio condition.

Differential delay and amount of reinforce-ment appear primarily to have affected rela-tive response rates, as examination of the over-

all absolute initial-link response rates revealsno systematic variation across conditions. How-ever, response rates in each of the terminallinks tended to decrease as the delay increased.In addition, terminal-link response ratestended to be higher for the smaller of the pairof delays under the 6:1 and 3:1 ratios, whileit was higher for the larger of the pair of delaysunder the 3:2 ratio. This is consistent with thepreference functions.

DISCUSSIONRelative response rates in the initial links

agreed more with Fantino's (1969) model thanwith Rachlin and Green's (1972). The brokenlines in Figure 1 are the relative response ratesfor the longer-delay key predictedl by thesetwo formulations. Both the decrease in rela-tive rate under the 6:1 delay ratio, and theincrease under the 3:2 delay ratio with in-creasing terminal-link delays are predicted byFantino (1969). Rachlin and Green (1972)predict a constant preference with increasingterminal-link length when the ratio is heldconstant. Choice is clearly more than simplya function of the ratios of amount and delayof reinforcement.Although the Fantino (1969) predictions

fairly well describe the data on 6:1 and 3:2ratios, they are, however, only one of a num-

ber of possible predictions from Equation 3.The different predictions arise because thereare various ways to convert delay and amountof reinforcement to a single variable for use

in Equation 3. Remember that Equation 3

deals only with terminal-link delay. Navarickand Fantino (1976) proposed converting delayand amount to a single measure of delay bycalculating terminal-link seconds per some-unitaccess to food (assuming a simple multiplica-tive interaction between delay and amount).The predicted points in Figure 1 were obtainedusing estimates of terminal-link time per 6-secaccess to food. Clearly, any unit could bechosen as the standard access to food, yieldingdifferent terminal-link delay estimates and dif-ferent predictions from Equation 3. While allthe possibilities predict the same general trendin relative rates, the points presented mostclosely fit the data. Unfortunately, there is noway to determine a priori which conversionwill succeed best.

Regardless of the conversion chosen, neitherFantino (1969) nor Rachlin and Green (1972)predicts the data for the 3:1 delay ratio. Sincethe delay ratio is equal and opposite to theratio of reinforcer amounts, both models pre-dict constant indifference (relative rate = .50)at all experimental values. Clearly, this wasnot the case. There was, in fact, a marked de-crease in preference for the key associated withthe longer delay as the delays increased. Ap-parently, the effects of the 3:1 delay ratio weregreater than those of the counterbalancingchanges in reinforcer amounts.One can eliminate the possibility that a

difference between programmed and obtainedamounts of reward are responsible for thediscrepancy observed under the 3:1 ratio. Re-inforcer amount was measured in terms ofduration of hopper elevation. There is a briefperiod from the onset of hopper elevationuntil the pigeon begins eating during whichno food is being obtained. Assuming that thisperiod is reasonably constant for both hopperdurations, the obtained ratio of longer hopperduration to shorter would actually be largerthan programmed. Fantino's (1969) equationwould thus predict increased preference forthe longer delay rather than the decreasedpreference observed for the 3:1 condition.A possible problem with the present design

concerns the interaction between preferenceand rate of reinforcement. This interactionmay have exaggerated the effects observed un-der all delay ratios, particularly when prefer-ence was extreme. Because the two initial-linkVI schedules were independent, obtained rela-tive rate of reinforcement could vary with

143

CHOICE BETWEEN DIFFERENTIAL REWARDS

relative rate of response. Any increase in rela-tive rate for an already-preferred key couldthus act to increase preference further. How-ever, the interaction between response and re-inforcement rate can only serve to heightenan effect; it cannot create a preference. Pref-erences might not be as extreme if relative rateof reinforcement is held constant at .50. Inthe present experiment, however, birds enteredthe two terminal links about equally often, andso relative entries into the terminal links rarelydeviated from .50. In addition, the presentstudy suffers from a difficulty common to mostof the research in this area. The shorter delayalternative, in addition to providing more im-mediate access to food, also returns the subjectto the initial link sooner than the long delayalternative. This serves to increase the rateof reinforcement for the shorter delay key.Imposing a blackout following food deliveryfrom the shorter delay alternative so that bothalternatives lead to the same amount of timeaway from the initial link may eliminate thisconfound. When Gentry and Marr (1980)essentially replicated MacEwen's (1972) studyusing blackouts as suggested, they also founda preference for the shorter delay alternative,but no consistent pattern of preference changewas obtained as absolute delay values in-creased. If these results are supported in otherstudies utilizing postreinforcement blackouts,a critical reappraisal of the entire area of re-search involving choice and delay of rein-forcement may be in order.

Nevertheless, the general trends in the pres-ent data remain and results from the 3:1 delayratio indicate that delay of reinforcement isa more potent determinant of choice than isreinforcer amount. This conclusion is consis-tent with results from earlier studies of choicebetween various terminal-link FI schedules(Davison, 1969; Duncan & Fantino, 1970;Herrnstein, 1964; Killeen, 1970). In all cases,preference for the shorter terminal link wasgreater than predicted by matching. These re-sults, along with the frequent finding of under-matching to reinforcer amount (Schneider,1973; Todorov, 1973; Walker & Hurwitz, 1971;Walker, Schnelle & Hurwitz, 1970; Snyderman& Green, Note 2), provide a strong argumentagainst the assumption that delay and amountof reinforcement are equivalent in their effectsupon choice.The differential effects of delay and amount

of reinforcement on choice behavior may beanalyzed in Baum's (1979) terms as follows:

RI =k(a,)v(d2)V (6)

where x and y serve as measures of the poten-cies of delay and amount of reinforcement asdeterminants of choice. Equation 6 is similarto Rachlin and Green's (1972) analysis in thatit uses ratios, and would therefore be inade-quate in accounting for the present data. How-ever, the ratio of the exponents, x/y, providesa measure of the relative effects of delay andamount of reinforcement on choice.This measure may be used in converting de-

lay and amount to a single variable for use inEquation 3. Previously, the conversion couldbe made by simply multiplying the smallerdelay value, d2, by the ratio of amounts ofreinforcement, aL/a2. This value, along withthe other programmed delay values, was thenused as t, and t2 in Equation 3. With the newmeasure of relative potency, d2 is multipliedby (aj1a2)X/v. If delay of reinforcement is amore potent determinant of choice than isamount, (ai/a2)x1IY will be less than a1/a2. Thisconversion leads to a prediction of greaterpreference for the shorter delay than would anassumption of equivalent potency.Table 4 presents the mean absolute devia-

tions between the obtained data and the pre-dictions from Rachlin and Green (1972),Fantino (1969) and that of the present transfor-mation. The closest fit to the data points wasobtained using an x/y value of .67. This isequivalent to exponents of 1 for amount ofreinforcement and 1.5 for delay. Examinationof the table reveals that this transformationmore accurately predicts the present resultsthan either Equation 3 or Equation 5, par-ticularly under the 3:1 delay ratio.

Unfortunately, while the proposed analysisprovides a good ex post facto description of be-havior, too little is known about the relevantvariables to allow accurate preexperimentalpredictions. In addition to the previouslymentioned difficulty with multiple conversionmethods, there appears to be no consistentagreement on the values of the exponents xand y. Davison (1969) used a cubic transforma-tion of delay values to obtain matching, whileKilleen (1970) used an exponent of 2.5. Kil-leen noted that his exponent was probablylower than Davison's because he did not use a

144

L. GREEN and M. SNYDERMAN

Table 4

Average absolute deviations of the obtained from predicteddata points for each condition.

(Preded value) and average absolute deviation

PowerTerminal link FI Rachlin-Green Fantino transformation

(secs) (1972) (1969) (xWy =. 67)

6:1 delay ratio12:2 (.33) .12 (.45) .04 (.44) .0424:4 (.33) .05 (.40) .12 (.37) .0960:10 (.33) .19 (.25) .11 (.15) .08120:20 (.33) .26 (.00) .07 (.00) .07

X deviationfrom predicted = .155 .085 .0703:2 delay ratio

6:4 (.67) .05 (.55) .07 (.52) .1030:20 (.67) .08 (.75) .07 (.60) .1060:40 (.67) .08 (1.00) .25 (.69) .07120:80 (.67) .23 (1.00) .10 (.89) .08

X deviationfrom predicted .110 .123 .088

3:1 delay ratio6:2 (.50) .11 (.50) .11 (.49) .12

30:10 (.50) .09 (.50) .09 (.42) .0960:20 (.50) .22 (.50) .22 (.35) .07

_ 120:40 (.50) .36 (.50) .36 (.19) .10X deviation

from predicted .195 .195 .095

COD. However, Herrnstein (1964) did not usea COD, and an analysis of his data on choicebetween two unequal FI terminal links indi-cates an exponent between 1.5 and 2.0 to bean appropriate transformation. Finally, whileDuncan and Fantino (1970) clearly foundoverpreference for the shorter terminal-link Fl,no single exponential transformation couldconsistently account for their data. The po-tency of reinforcer amount as a determinantof choice is also uncertain. Results of investi-gations of the effect of differential reinforceramounts on choice range from matching toextreme undermatching.The analysis of choice between reinforcers

differing in delay and amount is tantamountto the analysis of three classes of functions:those dealing with reinforcer value as a func-tion of delay and as a function of reinforceramount; those dealing with their interaction;and a rule to predict preference between al-ternatives that vary on these two dimensions.Current best estimates indicate that rein-

forcer value, as measured in choice situations,is a power function of delay and amount ofreinforcement. For delay of reinforcement anexponent greater than one is indicated, whilethe exponent transforming reinforcer amount

is generally one or less. Thus, delay to rewardis a more potent determinant of choice thanis amount.A great deal of research has been devoted

to finding an adequate rule for two-manipulan-dum choice situations. While the matching lawprovides accurate predictions in a wide varietyof situations (see de Villiers, 1977), Fantino's(1969) equation appears better suited for thevagaries of the concurrent-chain procedure.For example, no other choice model has soaccurately accounted for changes in preferenceas initial-link values are varied (Squires &Fantino, 1971).

Perhaps the key to understanding choicebetween rewards differing in delay and amountlies in the way these two variables interact.Despite its relative importance this interactionhas received little attention. Both Rachlin andGreen (1972) and Navarick and Fantino (1976)assumed a simple multiplicative interaction.The results of the present experiment, and thefailure of either of these models to fit the data,indicate that such an assumption is incorrect.Whether some more complicated form of theinteraction is sufficient, or whether an entirelydifferent representation may be necessary,awaits further research.

Nevertheless, the results of this experimentdemonstrate that there is an orderly relation-ship between delay and amount of reinforce-ment and choice, and that delay of reinforce-ment has a greater effect upon choice thandoes amount. In addition, behavior under aconcurrent-chain schedule is a function ofmore than just the ratio of the terminal-linkschedule values.We might also note, parenthetically, that

choice experiments in which delay and amountof reinforcement are both varied have rele-vance to the study of foraging behavior. Moststudies on choice vary only one parameter:either two rewards are offered simultaneouslyor the same reward is obtainable after differentdelays. In foraging, however, the choice is be-tween one kind of prey offered immediatelyand another that may be obtained later. Thepresent procedure may thus bridge the gapbetween conditioning and foraging experi-ments.

Implications for Self ControlSeveral researchers have defined self control

as the selection of a larger, delayed reward

145;

146 CHOICE BETWEEN DIFFERENTIAL REWARDS

over a smaller, more immediate one (Ainslie,1974; Green, in press; Navarick & Fantino,1976; Rachlin & Green, 1972). At a time closeto delivery of the rewards, organisms willgenerally choose the smaller, more immediatereward. However, as the time to delivery ofboth outcomes increases, preference reversestoward the larger delayed reward and the ani-mal may be said to be exhibiting self control.

This situation is illustrated in Figure 2.The two solid lines are the reward gradientsfor the two outcomes. Reinforcer value is as-sumed to decrease hyperbolically with timefrom reinforcer delivery (Ainslie, 1974; Green,in press). Preference reversal occurs at time T,where the two gradients cross. At times closerto the outcomes than T1, preference is forthe smaller reward, while the larger rewardshould be chosen at times greater than T1.The multiplicative-interaction assumption

of the Rachlin-Green and Navarick-Fantinomodels predicts that the point of preferencereversal, T1, should occur when the ratio ofdelays of reinforcement falls to a value equaland opposite to the ratio of reinforcer amounts.At all points to the right of T1, dj/d2 is greaterthan a1/a2 (where alternative 1 is the larger,more delayed reinforcer) whereas this situationis reversed to the left of T1. However, if delayof reinforcement is a more potent determinantof choice than is amount, the delay ratio musthave a lower value before preference reverses.

LR

I ~~ /

SR

0

TIME BEFORE OUTCOMES

Fig. 2. Hypothetical values of two rewards differingin amount and delay as a function of the time beforetheir outcome. The dashed line represents the changein value over time for the larger reward (LR) whendelay is a more potent determinant of choice.

Delay of reinforcement must be transformedby a higher power than amount, and will notequal (a,/a2) until dl/d2 decreases to a valueless than a, /a2. Thus, the larger, more delayedreward will not be preferred until farther intime from the two outcomes than predicted byRachlin and Green (1972) or Navarick andFantino (1976).The effect of increased potency of delay

would be to increase the rate of decay of thevalue of the larger, more delayed alternativerelative to the smaller, less delayed alternative.This is represented by the dashed line in Fig-ure 2. The point of preference reversal isthereby moved farther back in time to T2.In terms of self control, this means that or-ganisms would be expected to exhibit moreimpulsiveness and less self control. The degreeof impulsiveness is, of course, dependent uponthe rate of decay of these two value functions.Much further research will be necessary to as-sess the precise effects of specific variables onthese functions.

REFERENCE NOTES1. Green, L., Fisher, E. B., Jr., Perlow, S., & Sherman,

L. Preference reversal and self control: Choice asa function of delay to rewards. Paper presented atthe Sixth Annual Meeting of the Association forBehavior Analysis, Dearborn, Michigan, May, 1980.

2. Snyderman, M., & Green, L. Matching in humans toamount of reinforcement under a concurrent-chainprocedure. Paper presented at the Fifth AnnualMeeting of the Association for Behavior Analysis,Dearborn, Michigan, June, 1979.

REFERENCESAinslie, G. W. Impulse control in pigeons. Journal

of the Experimental Analysis of Behavior, 1974, 21,485-489.

Baum, W. M. On two types of deviation from thematching law: Bias and undermatching. Journal ofthe Experimental Analysis of Behavior, 1974, 22,231-242.

Brownstein, A. J. Concurrent schedules of responseindependent reinforcement: Duration of a rein-forcing stimulus. Journal of the Experimental Anal-ysis of Behavior, 1971, 15, 211-214.

Catania, A. C. Concurrent performances: A baselinefor the study of reinforcement magnitude. Journalof the Experimental Analysis of Behavior, 1963,6, 299-300.

Chung, S-H., & Herrnstein, R. J. Choice and delayof reinforcement. Journal of the ExperimentalAnalysis of Behavior, 1967, 10, 67-74.

Davison, M. C. Preference for mixed-interval versusfixed-interval schedules. Journal of the Experi-mental Analysis of Behavior, 1969, 12, 247-252.

L. GREEN and M. SNYDERMAN 147

de Villiers, P. Choice in concurrent schedules and aquantitative formulation of the law of effect. InW. K. Honig and J. E. R. Staddon (Eds.), Handbookof operant behavior. Englewood Cliffs, N.J.: Pren-tice-Hall, 1977.

Duncan, B., & Fantino, E. Choice for periodic sched-ules of reinforcement. Journal of the ExperimentalAnalysis of Behavior, 1970, 14, 73-86.

Fantino, E. Choice and rate of reinforcement. Journalof the Experimental Analysis of Behavior, 1969, 12,723-730.

Fleshler, M., & Hoffman, H. S. A progression forgenerating variable-interval schedules. Journal ofthe Experimental Analysis of Behavior, 1962, 5,529-530.

Gentry, G. D., & Marr, M. J. Choice and reinforce-ment delay. Journal of the Experimental Analysisof Behavior, 1980, 33, 27-37.

Green, L. Self-control behavior in animals. In V. L.Smith (Ed.), Research in experimental economics(Vol. 2). Greenwich, Conn.: JAI Press, in press.

Green, L., & Schweitzer, L. Second-order conditioningof the pigeon's key peck using an autoshaping pro-cedure. American Journal of Psychology, 1980, 93,25-39.

Herrnstein, R. J. Aperiodicity as a factor in choice.Journal of the Experimental Analysis of Behavior,1964, 7, 179-182.

Iglauer, C., & Woods, J. H. Concurrent performances:Reinforcement by different doses of intravenouscocaine in rhesus monkeys. Journal of the Experi-mental Analysis of Behavior, 1974, 22, 179-196.

Killeen, P. Preference for fixed-interval schedulesof reinforcement. Journal of the ExperimentalAnalysis of Behavior, 1970, 14, 127-131.

Logan, F. A. Decision making by rats: Delay versusamount of reward. Journal of Comparative andPhysiological Psychology, 1965, 59, 1-12.

Logan, F. A., & Spanier, D. Chaining and nonchainingdelay of reinforcement. Journal of Comparative andPhysiological Psychology, 1970, 72, 98-101.

MacEwen, D. The effects of terminal-link fixed-interval and variable-interval schedules on re-sponding under concurrent chained schedules. Jour-nal of the Experimental Analysis of Behavior, 1972,18, 253-261.

Navarick, D. J., & Fantino, E. Stochastic transitivityand the unidimensional control of choice. Learn-ing and Motivation, 1975, 6, 179-201.

Navarick, D. J., & Fantino, E. Self-control and generalmodels of choice. Journal of Experimental Psy-chology: Animal Behavior Processes, 1976, 2, 75-87.

Neuringer, A. J. Effects of reinforcement magnitudeon choice and rate of responding. Journal of theExperimental Analysis of Behavior, 1967, 10, 417-424.

Pliskoff, S. S., & Hawkins, T. D. A method for increas-ing the reinforcement magnitude of intracranialstimulation. Journal of the Experimental Analysisof Behavior, 1967, 10, 281-289.

Rachlin, H., & Green, L. Commitment, choice andself-control. Journal of the Experimental Analysisof Behavior, 1972, 17, 15-22.

Schneider, J. W. Reinforcer effectiveness as a func-tion of reinforcer rate and magnitude: A compari-son of concurrent performances. Journal of theExperimental Analysis of Behavior, 1973, 20, 461-471.

Schwartz, B. Effects of reinforcement magnitude onpigeons' preference for different fixed-ratio sched-ules of reinforcement. Journal of the ExperimentalAnalysis of Behavior, 1969, 12, 253-259.

Squires, N., & Fantino, E. A model for choice in simpleconcurrent and concurrent-chains schedules. Journalof the Experimental Analysis of Behavior, 1971, 15,27-38.

Ten Eyck, R. L., Jr. Effects of rate of reinforcement-time upon concurrent operant performance. Journalof the Experimental Analysis of Behavior, 1970, 14,269-274.

Todorov, J. C. Interaction of frequency and magni-tude of reinforcement on concurrent performances.Journal of the Experimental Analysis of Behavior,1973, 19, 451-458.

Walker, S. F., & Hurwitz, H. M. B. Effects of relativereinforcer duration on concurrent response rates.Psychonomic Science, 1971, 22, 45-47.

Walker, S. F., Schnelle, J., & Hurwitz, H. M. B. Ratesof concurrent responses and reinforcer duration.Psychonomic Science, 1970, 21, 173-175.

Williams, B. A., & Fantino, E. Effects on choice ofreinforcement delay and conditioned reinforcement.Journal of the Experimental Analysis of Behavior,1978, 29, 77-86.

Received December 5, 1979Final acceptance April 16, 1980