Big Data and the Brain...We need big data! •V4 receptive fields are moderately large •Potential...
Transcript of Big Data and the Brain...We need big data! •V4 receptive fields are moderately large •Potential...
Big Data and the Brain MICHA E L O L IVE R G A L L A NT L A B UC BER KELEY
The brain is gigantic
• The human brain has ~100 billion neurons connected by ~100 trillion synapses
• Multiple levels of organization
But our data is only “Big”
• Electrophysiology experiments can record from ~100 neurons simultaneously
• fMRI experiments we can record from ~90,000 voxels of about 20 mm3
• There are over 2,000,000 neurons per voxel
The visual brain
System identification and neuroscience
• Model each neuron based on relationship between stimulus and response
• Evaluate models based on their ability to predict responses to novel stimuli
Why system identification in visual cortex is hard
• Non-linear
• High dimensional
• Interpretability is important!
Car Eiffel Tower
Linearized regression
p1
p2
p3 Pixel Space
f1
f2
f3 Feature Space
a1
a2
a3 Brain Activity Space
Nonlinear Feature Mapping
Encoding Model
Decoding Model
Feature spaces
Feature Mapping
Responses Stimuli
W1
W2
Wn
No
nlin
eari
ty
+
Model neuron/voxel
Linear Weighting
Movie reconstruction from fMRI data
Nishimoto S, et al. Curr Biol. 2011
Oct 11;21(19):1641-6
Van Essen DC, Gallant JL. Neuron. 1994 Jul;13(1):1-10.
Tuning in V4
We need big data!
• V4 receptive fields are moderately large
• Potential stimulus space is very large
• Natural images span the relevant space
• Response is highly nonlinear
How to get big data
• Implantable electrodes allow us to record from the same cell over many days
• We used over 1 million frames of natural movies, the largest ever stimulus set in V4
General nonlinear modeling
• The Volterra Series:
• Can control model flexibility by order choice
• Parameter space grows quickly with order
x1 x2
h11 h12
h21 h22 h111 h121
h211 h221
h111 h121
h111 h121
The scale of the problem
• If we have 1000 pixels
• 2nd Order: 500,000+ coefficients
• 3rd Order: 160 million+ coefficients
• 4th Order: 40 billion+ coefficients
• But we actually have about 196,608 pixels…
• 2nd Order: 19 billion+ coefficients
• 3rd Order: 1.2 quadrillion+ coefficients
• 4th Order: 62 quintillion+ coefficients
Pixels 1.2x1015
Taming dimensionality
Pixels 1.2x1015
PCA 1.6x108
Kernel regression
For all i,j =1:n in training data
Calculate weights for kernel regression model
Use weights to make predictions for new x
Kernel function equivalent to dot product in feature space
Pixels 1.2x1015
PCA 1.6x108
Kernel 1x106
The Inhomogeneous Polynomial Kernel
2nd Order IHP:
Implicitly Maps to feature space containing all first and second order terms
Neural networks as kernel machines
Input Hidden Units
Output In a standard NN:
From NN perspective, kernel regression with a tanh kernel function is equivalent to a NN with hidden units = training samples
Pixels 1.2x1015
PCA 1.6x108
Kernel 1x106
IPKN 1.5x104
Stochastic gradient boosting
Volterra Space Error Surface Data
Prediction performance by model order
Color constancy in V4
V4 Tuning for Color
Kusunoki M et al. J Neurophysiol 2006;95:3047-3059
Color Tuning of V4 Cells
First Order Color Coefficients
V4 tuning to curvature
Pasupathy A , and Connor C E J
Neurophysiol 2001;86:2505-2519
V4 tuning to Non-Cartesian Gratings
Gallant JL, Connor CE, Rakshit S, Lewis JW, Van Essen DC.
J Neurophysiol. 1996 Oct;76(4):2718-39.
Eigenvectors of second order V4 receptive field model
Eigenvectors of second order V4 receptive field model
Shape tuning of a V4 cell’s Volterra model
Shape tuning of a V4 cell’s Volterra model
Embracing the Complexity
• Demonstrated a way to make this big problem tractable
• Shown many reported features of V4 tuning can exist in a single cell
• Interpretation of large models is still a major problem
• Need tensor libraries that exploit symmetry to decompose large models
Thank you!
V4 High Response Movie Frames
Stochastic Gradient Boosted IPKNs
• Use IPKNs as the weak learners
• Fit to sample of data using backprop w/ a stopping set
• Perform line search to determine step size that minimizes error on sample
• Multiply step size by learning rate and update function
• Ensemble is equivalent to a single Volterra model!
Some Important Unanswered Questions
• What information are our models missing in V2, V4 and beyond?
• Do we need nonlinear combinations of basis functions?
• Can we derive new basis functions?
Extracting Coefficients from Model
• Create a design matrix or the desired order of interactions from the support vectors/input weights
• Multiply by the output weights and weight by correction factor
• Extract coefficients from each iteration’s network, weight by step size and sum to get final set of coefficients
Feature Spaces
x =
Response to each
category was found
using regularized
linear regression
Movies were shown
to subjects
Wom
an
Talkin
g
Text
Car
Build
ing
Category labels
2.8 -0.4
-0.9
1.7 -0.8
-1.0
-2.1
0.7
0.1 -2.70.1
-3.7
-3.2-1.5
-1.6
BOLD responsesCategory modelweights
Natu
ral m
ovie
s
Movies were labeled
with 1705 nouns
and verbs
BOLD responses were
recorded from the whole
brain using fMRI
120 minutes 120 minutes
Constructing a Semantic Space
person
athlete
plantorganism
animal
arthropod
reptile
bird fish
mammal
carnivore talk
communicate
text
group
herd
attribute
dirt
measure
communication
event
wave
rodeo atmospheric
phenomenon
mist
bodypart
eye
leg
matter
food
underwatersky
material bamboo
location
city
grassland
geol. formation
hill
plant organ
artifactway
clothing
road
structure
room
shop
door
building
furniture
container
bottle
device
laptop
gas pump
weapon
tool
kettle
ball
vehicleboat
wheeled
vehiclecar
travel
breathe
fastenmove
(transitive)
touch
hit
jump
change
walk
gallop
rappel
drag
turn
spin
bloom
lean
rodent
ungulate
consume
rub
move
C
equipment
Coefficient on 1st PC– +
person
mammal
bodypart
vehicle structure
location
plant organ
athlete
2nd PC
+
–+
+3rd PC
4th PCB
(Red channel)
(Green channel)
(Blue channel)
city
RGB colormap
for the group
semantic
space
sky
atmos.
phenom.
A
Semantic Decoding
Flattening the Brain
Visualizing Semantic Space
CoS
ITS
MTS
STS
IPS
CiSmr
PoCeS
Sylvian
Fissure
CeS
IFS
SFS
CiS
STSMTS
IPS
CiSmr
PoCeS
CeS
SFS
IFS
CiS
ITS
CoS
Sylvian
Fissure
V1
V2
V3
V4
V3b
V3a
V7IPS
MT+
EBA
OFA
FFA
PPA
RSC
Speech
FO
FEF
M1M
S1M
M1H
S1H
M1F S1F
SMHA
SMFA
AC
SpeechVper
Vper
pSTS
A1
V1
V2V3 V4
PPA
FFA
EBA
MT+LO
V3b
V3a
V7
IPS
RSC
FO
FEF
SpeechSEF
SMFA
S1M
M1MS1H
M1H
M1F
S1F
AC
Vper
Vper pSTS
A1
Speech
PrCu
TPJTPJ
PrCu
LO
TOS
SEF
TOS
SMHA
!
!!
!
!
!
!
!
!!LHanterior
su
pe
rio
r
!
!!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!
Correlation with 1st PC- +
RHsu
pe
rio
r
anterior
WordNet LSA
LabelsCoS
ITSMTS
STS
IPS
CiSmr
PoCeS
Sylvian
Fissure
CeS
IFS
SFS
CiS
STS
MTS
IPS
CiSmr
PoCeS
CeS
SFS
IFS
CiS
ITS
CoS
Sylvian
Fissure
V1
V2
V3V4
V3b
V3aV7IPS
MT+
EBA
OFA
FFA
PPA
RSC
IFSFP
Speech
FO
FEF
M1M
S1M
M1H
S1H
M1FS1F
SMHASMFA
AC
Speech
Vper
VperpSTS
A1
V1
V2
V3
V4
PPA
FFA
OFA
EBA
MT+
LO
V3b
V3a
V7
IPS
RSC
FBA
IFSFP
FO
FEF
Speech
Speech
SEF
SMFA
S1M
M1M
S1H
M1H
M1FS1F
AC
Vper
Vper
pSTS
A1 Speech
PrCu
TPJTPJ
PrCu
!
!
!
!
!
!
!
!
!
!
EBA
LO
LHanterior
su
pe
rio
r
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
B
Correlation with 1st PC- +
A
RHsu
pe
rio
r
anterior
C D