1. Learning theory

Rudolf Cardinalrudolf@pobox.comUniversity Lecturer in Clinical Informatics,Department of Psychiatry, University of CambridgeHonorary consultant psychiatrist, Liaison Psychiatry Service, CPFT & CUH

Cambridge MRCPsych courseTue 23 Oct 2018

MRCPsych syllabus (2013, 2018):

Routes to action

● Unlearned behaviour: stuff you’re born with● simple, e.g. reflexes● complex, e.g. fixed action patterns

● Pavlovian conditioning (= classical conditioning): predicting the world and responding accordingly

● Instrumental conditioning (= operant conditioning): controlling the world

● Also some more complex situations in which these interact.

see Cardinal et al. 2002; Cardinal 2001; Doty & Bosma 1956; Lorenz 1939; Tinbergen 1948

Classical (Pavlovian) conditioning

● Pavlov (1927): dogs, salivation.

● The experimenter arranges a relationship between stimuli. (The subject’s responses are measured.)

● One is initially neutral: the (to-be-)conditioned stimulus (CS). One is biologically relevant: the unconditioned stimulus (US), which normally evokes an unconditioned response (UR).

● When the CS and US are paired by the experimenter, the CS comes to elicit the same response, termed a conditioned response (CR).

● Other basic phenomena:● extinction: after training, present CS without US● blocking: A → US; then A+B → US; what’s learned about B?

Pavlov (1927). Kamin (1969), blocking.

CS2

Classical conditioning: what’s being learned? (1)

Rescorla & Wagner (1972).

Simple model:

Rescorla–Wagner model:ΔV

CS = αβ(γ – V

total)

V = current associative strengthγ = degree of associative strength that the US will support

thus: “learn to the degree that an unexpected outcome occurs”[α = CS learning rate parameter; β = US learning rate parameter]

Describes learning, extinction, blocking, etc.

USCS1

responsestrong / fixedinitially weak;

plastic “synapse”

Classical conditioning: Rescorla–Wagner illustration

ΔVCS

= αβ(γ – Vtotal

)

Classical conditioning: what’s being learned? (2)

Reality is more complex, including associations formed from the CS to the US and to ‘affect’, not just the response.

Generalization

e.g. Dymond et al. (2015)

● Condition to something.e.g. 2 kHz tone (CS) → shock (US)

leads to freezing to tone (CR)

● Generalize: respond to something similar.e.g. 2.1 kHz tone → freeze (CR)

● Clearly sensible to a degree.

● Over-generalization in e.g. anxiety disorders? For example:

(1) panic attack (US?) on train (initially neutral stimulus; becomes CS?)(2) generalize to enclosed spaces, public transport, situations with mechanical noises, interoceptive stimuli vaguely like panic symptoms?

Stimulus preparedness

e.g. Seligman 1971, Öhman 1979; Öhman & Mineka 2001

● A predisposition to learn about some specific stimuli more than others.

● Easier to induce fears in the lab to e.g. snakes, spiders, heights.● Predisposition attributable to evolutionary history?

Example:● Subjects: non-fearful lab-reared monkeys● Observe wild-reared monkey displaying fear to live/toy snakes, even

on video → display fear reactions themselves to toy snakes● but not if flowers or toy rabbits replaced the snakes in the videos● even though flowers etc. work fine as CSs in appetitive experiments

● … suggests something special and innate about snakes; predisposed to activate the “fear module”?

Incubation [ALLOW MIND TO SWITCH OFF]

McAllister & McAllister (1967); Eysenck (1968 onwards); Malloy & Lewis (1990)

● Traditional definition of “fear incubation”: not a real thing?(1) CS → ‘fearful’ US(2) wait... response to CS increases

● Eysenck’s:(1) CS → US(2) more CS presentations... response to CS increases

● … if true, particularly concerning for CS exposure therapies like desensitization.

● … there is experimental evidence against this (e.g. Malloy & Lewis)

● … so: a term to be aware of (as per the syllabus), but probably not a thing.

● “Incubation” also used re creativity (answers come in the shower / during sleep, etc.). This probably is a thing.

Instrumental (operant) conditioning [MIND BACK ON]

Thorndike (1905), Grindley (1932), Tolman (1932), Skinner (1938), Dickinson (1980)...

● For controlling the world.● The experimenter arranges a contingency between the subject’s

behaviour and some (motivationally significant) outcome.● Action (‘operant’) is instrumental in producing the outcome.

● Thorndike’s Law of Effect (cats, puzzle boxes): an effective outcome strengthens the connection between stimuli and the action. A stimulus–response (S–R) habit theory.

● But animals and people can also be goal-directed.

Behaviourism: positive and negative reinforcement

● Skinner (e.g. 1932, 1938, 1953): behaviourism and reinforcement.

Positive reinforcement A → food, leads to more A

Negative reinforcement (inc. escape, avoidance) A → less shock, leads to more A

Punishment A → shock, leads to less A

Negative punishment A → less food, leads to less A

Extinction A → nothing leads to less A

● “Positive reinforcer” / “negative reinforcer” = food / shock. Defined by consequences on behaviour.

● Other useful terms: ‘appetitive’, ‘aversive’ [but not in strict behaviourist tradition as imply emotional states!].

Basic schedules of reinforcement

● Ratio schedules: outcome is delivered by the nth response● fixed ratio, e.g. n = 10

● FR-1 also called “continuous reinforcement”● like piecework● pause (e.g. to eat), then resume

● variable ratio, e.g. n = varying between 5–15● like door-to-door sales, gambling● high, quite steady rates of responding

● Interval schedules: outcome is delivered by the first response after t seconds have elapsed since the last outcome

● fixed interval, e.g. t = 60 seconds● pause after reinforcer, then ↑ towards expected reinf. time

● variable interval, e.g. t = 30–90 seconds● moderate, steady responding● … models consumption of a replenishable resource, like

grazing?

Some clinical aspects of negative reinforcement

● Successful avoidance implies lack of exposure to the US.● If avoidance action predicts “no US”, and no US comes, no reason to

alter your beliefs (or synaptic weights). So you’ll continue to avoid.

● Avoidance behaviours are very resistant to extinction.● Particularly resistant if the initial relationship involved some

uncertainty about when the US would come ( lack of ‘contrast’ ⇒when US is absent).

● Think phobias and anxiety.

An aside: another schedule of practical use

● How to “get rid of” behaviour without punishment, if extinction alone is difficult?

● Differential reinforcement of other behaviour (DRO schedule)= differential reinforcement of alternative behaviour (DRA schedule).

● Subject exhibits undesirable behaviour.● Reinforce him/her after e.g. 1 minute of not exhibiting that

behaviour.● If undesirable behaviour occurs, reset the clock.● With success, gradually increase the time.

e.g. Petscher et al. (2009)

Instrumental versus Pavlovian conditioning

Animals work for reinforcement for several reasons, including...

after Dickinson (1980)

Goal-directed action

Adams & Dickinson (1981)

Train rats to press a lever for food A. Give them food B for free.

Poison either food A (group P for poisoned) or food B (group U for unpoisoned).

Test responding in extinction (no food).

If their lever-pressing is goal-directed and they represent the value of the goal, then group P should press less than group U.

They do.

Stimulus–response habits develop after extended training

Adams (1982)

Dickinson et al. (2002) experiment 2, figure 6 redrawn

Is alcohol-seeking more ‘habitual’ than goal-directed?

Companies may have learned from rat experiments!

Samson (1986), rats; 1995 saw introduction of alcopops to UK

Sucrose ‘fading’ procedure: from e.g.• 20% sucrose• 20% sucrose, 5% ethanol• …• 5% sucrose, 10% ethanol• …• 40% ethanol

Cues paired with reinforcement can also motivate

Secondary (conditioned) reinforcement

● Associate neutral stimulus (CS) with appetitive stimulus (US), e.g. by classical conditioning.

● Then you may work for the CS – now termed a conditioned reinforcer (or sometimes a secondary reinforcer).

● Easily demonstrable in animals: tone → food, then work for tone.

● People often think of money in this context, but money is probably a lot more complicated psychologically (e.g. fungible, not just associated with a specific outcome).

● There’s an equivalent concept of conditioned punishers.

Environmental stimuli (cues and contexts) may become associated with the effects of drugs such as cocaine through Pavlovian conditioning. They become conditioned stimuli (CSs).

They may motivate an addict to seek out drugs — cue-induced (conditioned) craving.

Conditioning and addiction

Cues paired with reinforcement can also motivate

Pavlovian–instrumental transfer depends on motivational state

Dickinson (1986); Dickinson & Dawson (1987a, 1987b)

Pavlovian–instrumental transfer? Supermarkets

“Tesco TV is being established to… provid[e]… offers and value propositions from Tesco, its partners and advertisers — where it can be of most value, in-store where many purchase decisions are made… 7 ‘zones’ were identified in-store where programming could be targeted to make the best use of ‘dwell time’ to create a positive effect for the customer and advertisers… [Grocery, Beers/Wines/Spirits, etc.]…The trial began in 3 stores and its impact was comprehensively researched with Tesco customers… proposed roll-out to 300 stores.”

http://www.visagegroup.com/clients-retail-tesco.htm, 17 Feb 2004

Static advertising, of course, and advertising to children (works: e.g. Galst & White 1976 Child Dev 47:1089), but also auditory/visual stimuli:

Shopping and motivational state

Mela et al. (1996)

Summary: routes to action

● Unlearned behaviour: stuff you’re born with● simple, e.g. reflexes● complex, e.g. fixed action patterns

● Pavlovian conditioning (= classical conditioning): predicting the world and responding accordingly

● Instrumental conditioning (= operant conditioning): controlling the world

● goal-directed action● instrumental contingency● incentive value of the outcome – complex; learned; involves

hedonic value● stimulus–response habits

● Also some more complex situations in which these interact.● conditioned reinforcement● Pavlovian-to-instrumental transfer

“Reciprocal inhibition, habituation, chaining, shaping, cueing”

● Google: this combination leads you almost exclusively to MRCPsych links. This tells you something. So, definitions:

● reciprocal inhibition = you can’t be both anxious and relaxed, so evoke (in behaviour therapy) the opposing response while presenting the anxiety-provoking stimulus

● habituation = ceasing to respond after repeated presentations (e.g. startle to a loud noise → get used to it)

● shaping = reward animal for doing something close to what you want... ever closer... behaviour is “shaped” like clay

● chaining = if you want a behavioural sequence (chain) ABCDE, reinforce A, then reinforce AB, etc.

● cueing = use of a teaching signal to shape behaviour (e.g. smile, frown) (educational psychology)

Observational learning

Bandura (1961, 1971, 1972, 2004); Deguchi (1984).

● A kind of social learning.● Perceive (observe) behaviour, then perform (imitate) that behaviour.● “Transmission of aggression through imitation of

aggressive models”: Bobo doll experiment.

● Acquisition simply through observation, including in “one-trial” learning.● Delayed performance, mediated by cognitive processes (cognitive

mediation).● Vicarious, rather than direct, reinforcement. What are the consequences for

someone else when they perform a behaviour?● Bandura argues it involves:

● attentional processes (e.g. salience, functional value);● retention processes (e.g. cognitive construction, rehearsal);● production processes (e.g. monitoring of own enactment);● motivational processes (e.g. perceived benefits/costs).

Cognitivism / cognitive learning theories

● Broad group of theories.● Developed from Gestalt psychology: the way the mind acquires

meaningful perceptions in a complex environment.

● Emphasize internal mental processes (and contrasted with “radical behaviourism” that de-emphasized these), such as

● attention● working memory● computation

● You’ll come across specific examples, like the Baddeley & Hitch model of working memory [see memory lectures].