Post on 20-Jan-2016
Motor adaptation and the timescales of memory
Reza ShadmehrJohns Hopkins School of Medicine
Ali Ghazizadeh
Maurice Smith Konrad Koerding Haiyin Chen
Dave ZeeWilsaan Joiner
Jun Izawa
Tushar Rane
Duhamel et al. Science 255, 90-92 (1992)
The brain predicts the sensory consequences of motor commands
musclesMotor commandsforce
Body partState change
Sensory system
ProprioceptionVision
Audition
Measured sensory
consequences
Forward model
Predicted sensory consequences
Integration
Belief
What we sense depends on what we predicted
Wolpert et al. (1995)
5 10
5
Eye Position (deg)
Eye
Po
siti
on
(d
eg)
Saccade adaptation: gain decrease
McLaughlin 1967
5 10
5
Eye Position (deg)
Eye
Po
siti
on
(d
eg)
McLaughlin 1967
Saccade adaptation: gain decrease
Kojima et al. (2004) J Neurosci 24:7531.
_
Result 1: After changes in gain, monkeys exhibit recall despite behavioral evidence for washout.
+ +
Savings: when adaptation is followed by de-adaptation, motor system still exhibits recall
Saccade gain = Target displacement
Eye displacement
Result 2: Following changes in gain and a period of darkness, monkeys exhibit a “jump” in memory.
+ _ +
Offline learning: with passage of time and without explicit training, the motor system still appears to learn
Kojima et al. (2004) J Neurosci 24:7531.
( ) ( )( )1 2
( ) ( ) ( )
( 1) ( ) ( )1 11 1
( 1) ( ) ( )2 22 2
ˆ ˆ ˆ
ˆ
ˆ ˆ
ˆ ˆ
n nn
n n n
n n n
n n n
y y y
y y y
y a y b y
y a y b y
Adaptation as concurrent learning in multiple systems:A fast learning system that forgets quicklyA slow learning system that hardly forgets
Smith et al. PLOS Biology, 2006
prediction
Prediction error
Learning
Savings: de-adaptation may not erase adaptation
Task reversal periodre-adaptation
Trial number
Smith et al. PLOS Biology, 2006
Offline learning: Passage of time has asymmetric affects on the fast
and slow systems
Smith et al. PLOS Biology, 2006
Task reversal period
“dark” period
re-adaptation
Trial number
Slow stateFast state
-
( 1) ( ) ( )1 11 1
( 1) ( ) ( )2 22 2
ˆ ˆ
ˆ ˆ
n n n
n n n
y a y b y
y a y b y
Spontaneous recovery is also observed in reach adaptation
Trial number
Per
turb
atio
n
forc
e
Trial number
0
1
Per
form
ance
rel
ativ
e to
go
al
Task reversal period
Error clamp period
Smith et al. PLOS Biology 2006
Errors clamped to zero ( 1) ( ) ( )1 11 1
( 1) ( ) ( )2 22 2
ˆ ˆ
ˆ ˆ
n n n
n n n
y a y b y
y a y b y
1. Perturbations that can affect the motor plant have multiple time scales.Some perturbations are fast: muscles recover from fatigue quickly.Some perturbations are slow: recovery from disease may be slow.
2. Faster perturbations are more variable (have more noise).
3. The error that we observe is due to a contribution from all possible perturbations.
4. The problem of learning is one of credit assignment: when I observe an error, what is the time-scale of this perturbation?
The learner’s view about the cause of motor errors
( ) (1 1/ ) ( )disturbance t disturbance t
0, /N c
( ) ( )observation t disturbance t
Koerding, Tenenbaum, Shadmehr, unpublished
A
1w
2w
mw
t
Slow change
fast change
The Bayesian learner’s interpretation of motor error
y
xContext
perturbation y
x
1w
State of the variouspotential causes of error 2w
mw
tDisease state
Fatigue state
Savings: de-adaptation does not washout the
adapted system
Simulation
Koerding, Tenenbaum, Shadmehr, unpublished
Spontaneous recovery
Characteristics of long-term motor memoryData from Robinson et al. J Neurophysiol 2006
Bayesian Learner
Koerding, Tenenbaum, Shadmehr, unpublished
Motor system is disturbed by processes that have various timescale (fatigue vs. disease). Credit assignment of error depends on uncertainty regarding what is the timescale of the disturbance.
Prediction: When there are actions but the sensory consequences cannot be observed, states decay at various rates, but uncertainty grows. Increased uncertainty encourages learning.Bayesian learner
Adapting with uncertainty
Adapting with uncertainty: two predictions
Sensory deprivation Faster subsequent rate of learning.
Example: A subject that spends a bit of time in the dark will subsequently learn faster than a subject that spends that time with the lights on.
Why: In the dark, uncertainty about state of the motor system increases.
Longer inter-stimulus interval Better retention.
Example: A subject that trains on n trials with long ITI will show less forgetting than one that trains on the same n trials with short ITI.
Why: events that take place spaced in time will be interpreted as having a long timescale.
Ali Ghazizadeh Maurice Smith
Konrad Koerding
Fast and slow adaptive processes arose because disturbances to the motor system have various timescales (fatigue vs. disease). When faced with error, the brain faces a credit assignment problem: what is the timescale of the disturbance? To solve this problem, the brain likely keeps a measure of uncertainty about the timescales.
A prediction error causes changes in multiple adaptive systems. Some are highly responsive to error, but rapidly forget. Others are poorly responsive to error but have high retention. This explains savings and spontaneous recovery.
Summary
1. Internal models are supposed to help us control our movements in real-time. What are these fast and slow systems learning and how does that learning affect real-time control of movements?
2. Can we say anything about the neural structures that might be responsible for computing internal models?
What are some of the holes in these ideas?
Body +environment
State change
Sensory system
ProprioceptionVision
Audition
Measured sensory consequences
Forward model
Predicted sensory consequences
Integration
Belief about state of body
and world
Goalselector
Motor commandgenerator
Emo Todorov: Motor command generator as an optimal controller
( 1) ( ) ( ) ( )
( ) ( ) ( )
1(0) ( ) ( ) ( ) ( 1) ( 1) ( 1)
0
( ) ( ) ( )
( 1) ( ) ( ) ( ) ( ) ( )
ˆ
ˆ ˆ ˆ
k k k ku
k k ky
pk T k k k T k k
k
k k k
k k k k k k
A C
B
J L T
G
A AK B C
x x u ε
y x ε
u u y y
u x
x x y x u
Signal dependent motor noise
Signal dependent sensory noise
Actual state of the system(eye state, target state, etc.)
What we can observe about the state of the system
Motor command generator as a stochastic optimal controller
Todorov (2005)
Cost to minimize
Feedback control policy
Body +environment
State change
Sensory systemMeasured sensory
consequences
Forward modelPredicted sensory consequencesIntegration
Belief about state of body
and world
Goalspecification
Motor commandgenerator
Belief about state
eye velocity
deg
/sec
0 0.05 0.1 0.15 0.2 0.25
0
100
200
300
400
500
Time (sec)
Body +environment
State change
Sensory systemMeasured sensory
consequences
Forward modelPredicted sensory consequencesIntegration
Belief about state of body
and world
Goalspecification
Motor commandgenerator
5 10 15 30 40 50 Saccade size
The mathematical framework allows one to produce detailed trajectory of movements.
In the target jump paradigm, error is a difference between predicted and actual sensory consequences of oculomotor commands.
Therefore, the forward model must adapt.
But if that adaptation is not precisely matched by the motor command generator, the result will be sub-optimal saccades.
Prediction error
The direct and indirect output pathways from the superior colliculus (SC)
• Direct pathway
SCbrainstem
• Indirect pathway
SCcerebellumbrainstem
Cross-axis saccade adaptation
Equal rates of learning in the controller and the forward model
saccades remain straight
Learning in the forward model only
saccades become curved
Body +environment
State change
Forward modelPredicted sensory consequences
Belief about state of body
and world
Goalspecification
Motor commandgenerator
Incre
ased
tra
inin
g
T1
T2
fixation
Cross-axis saccade adaptation: Experiment design
(In complete darkness, with search coil lenses on the eyes)
Chen, Joiner, Zee, Shadmehr (unpublished)
Characteristics of primary saccades during adaptation
T1
T2
15o5o
Chen, Joiner, Zee, Shadmehr (unpublished)
Curvature of primary saccades quantified through chord slopes
Chen, Joiner, Zee, Shadmehr (unpublished)
The observation that saccades become curved, and therefore sub-optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced.
The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway).
Saccade curvature suggests that errors cause rapid adaptation in the forward
model
Body +environment
State change
Sensory system
Forward modelPredicted sensory consequencesIntegration
Belief about state of body
and world
Goalspecification
Motor commandgenerator
Prediction error
Haiyin Chen
Dave Zee
In saccades and reaching, performance is guided by internal models that adapt at multiple timescales:
A fast learning system that has poor retention.
A slow learning system that hardly forgets.
The observation that saccades become curved, and therefore sub-optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced.
The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway).
Summary:
Wilsaan Joiner
1. If learning of forward models (indirect pathway) is faster than the controller (direct pathway), the result is a sub-optimal system. Most of our movements appear optimal. What guides learning in the direct pathway so that we eventually become optimal?
2. If we learn as a Bayesian, we keep a measure of uncertainty about what we know. Does the uncertainty in the internal model affect our control policies (direct pathway)?
What are some of the holes in these ideas?
Raymond Clarence Ewry (USA)Gold Medal, 1908 Olympics
Cornelius Johnson (USA)Gold Medal, 1936 Olympics Dick Fosbury (USA)
Gold Medal, 1968 Olympics
Body +environment
State change
Sensory system
Forward modelPredicted sensory consequences
Integration
Belief about state of body and world
Goalspecification
Motor commandgenerator (control policy)
Prediction error
Learning in the direct pathway:
finding a better control policy in the high jump task
N=6
The optimal control policy
To maximize probability of arriving at target in time, I should minimize my motor commands near the end of the movement.
Over compensate for the forces early, let the robot bring you back.
Predicted trajectories under the optimal control policy
Accuracy of model
Izaw
a,
Ran
e,
Don
ch
in,
Sh
ad
meh
r (u
np
ub
lish
ed
)
Null field
Izawa, Rane, Donchin, Shadmehr (unpublished)
In performing an action, the motor commands that we generate should depend on our confidence (uncertainty) in
our models.
( )
( )
0
0
n
n
Bf x
B
9cm
FP
Screen
( 1) ( ) ( ) ( )
( ) ( ) ( )
1(0) ( ) ( ) ( ) ( 1) ( 1) ( 1)
0
( ) ( ) ( )ˆ
k k k ku
k k ky
pk T k k k T k k
k
k k k
A C
B
J L T
G
x x u ε
y x ε
u u y y
u x
Stochastic optimal control with model uncertainty
Jun Izawa
Tushar Rane
Traditional stochastic optimal control
( 1) ( ) ( ) ( )ˆ
ˆ ,
k k k kuA C
A N M V
x x u ε
Izawa, Rane, Donchin, Shadmehr (unpublished work)
Stochastic optimal control with model uncertainty: Predictions
Izawa, Rane, Donchin, Shadmehr (unpublished work)
People learn policies that depend on their model uncertainty:
Overcompensate only if you are certain of the world
N=6
High certainty
Low certainty
High certainty
Low certainty
Jun Izawa
Motor control is about solving two distinct problems:
Learning a control policy (direct pathway).Learning a forward model (indirect
pathway).
Motor learning is at multiple timescales:A fast learning system that has poor
retention.A slow learning system that hardly forgets.
The forward model (indirect pathway) adapts much more quickly than the controller (direct pathway).
Overview: Computational problem of motor control
Maurice Smith
Haiyin Chen
1. In saccade adaptation, nothing happened to the body; it was the target that was behaving strangely. When there is error, how does the brain distinguish between changes in the body vs. changes in the world? This is a second credit assignment problem.
2. What is the error signal that guides learning of control policies?
3. Are the direct and indirect pathways computational pathways or neural pathways?
What are some of the holes in these ideas?
thalamus
Motor cortex
Deep cerebellar nuclei
Pons
DBS: deep brain stimulation
Inf. Olive
Reversible disruption of cerebellar pathways in humans
Cerebellar cortex
Co
rtic
osp
inal
tra
ct
Sherwin Hua
Deep Brain Stimulation
1.5 mm electrode is implanted in the thalamus and connected via subcutaneous wires to a stimulator.
The subcutaneous stimulator and battery.
Parameter settings can be adjusted via an external device.
Fred Lenz
Stimulation of VL thalamus improves tremor but impairs adaptation
Ch
en
et
al.
Cere
bra
l C
ort
ex,
2006
Stimulation voltage
Bipolar stimUnipolar stim
Tremor during reaching
Movement onset
Thoroughman & Shadmehr, J Neurosci, 1999
EMG patterns during reach adaptation
Neural correlates of motor learning in the VL thalamus
Adaptation level was low
Behavioral performance
• Sites attempted recording ……………….. 105• Sites successfully recorded units ………. 58 (55%)• Units with more than 60 trials …………… 61
–Vim………………….35–Vim-Vop border……12–Voa/Vop…………… 14
• Single units ……………………………….. 16 (26%)• Movement related units …………………. 36 (59%)
–Vim………………….21–Vim-Vop border……5–Voa/Vop……………10
• Units showed direction selectivity ………. 18 (50%)–Vim………………….11–Vim-Vop border……1–Voa/Vop…………….6
Recording sites and neural responses
target Vmax stop hold/wait
Adaptation induces change in firing pattern before movement onset
1. The cerebellum appears to be a critical structure for motor adaptation. Is this the place where forward models are formed?
2. Speculation: cerebellar cortex may represent the “fast system”, with the cerebellar nuclei representing the “slow system”. Prediction: cerebellar patients may learn slowly, but they will also forget slowly.
3. Learning control policies depends on reward prediction errors.Is the basal ganglia the structure crucial for learning control policies?
4. Challenge ahead: To look for behavior and neural signatures of control policies and forward models in healthy individuals and patients with motor disorders.
Conclusion and speculations
The neural basis of motor adaptation
Cerebellar degeneration impaired adaptation of reaching
Huntington’s disease (HD) patients showed no deficit in adaptation
Smith and Shadmehr, J Neurophysiology 2005
early null3
Visual rotation adaptation