RDM mixtures for predicting visual cortices...
Transcript of RDM mixtures for predicting visual cortices...
![Page 1: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/1.jpg)
RDM mixtures for predicting visual cortices responses
Agustin Lage Castellanos1,2 and Federico De Martino2
1-Cuban Neuroscience Center, 2-Maastricht University
Algonauts Challenge 2019
![Page 2: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/2.jpg)
Intuition behind our method
Perceptual
Cat
ego
rica
l
fMRI-EVC
MEG-early
fMRI-ITC
MEG-Late
DNN-L1
DNN-L3DNN-L2
DNN-L5DNN-L4
![Page 3: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/3.jpg)
Combining RDMs to improve predictions
Predicted RDM Perceptual Categorical DNN
= 𝑤1 + 𝑤2 + 𝑤3
![Page 4: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/4.jpg)
Perceptual RDMs
![Page 5: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/5.jpg)
Perceptual RDMs
Only uses image information
Extract Edges and Smooth
Perceptual-RDMPixel Overlap
![Page 6: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/6.jpg)
Categorical RDMs
![Page 7: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/7.jpg)
Categorical Structure of the 92 image set
Objects-Scenes
animals
Human
Fruits-vegetables
Faces
Hands
Monkey faces
Animal Faces
![Page 8: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/8.jpg)
Within category RDM based on fMRI/MEG data similarity
92 x 92 8 x 8
mean
Between image fMRI/MEG similarity Between category fMRI/MEG similarity
fMRI-ITC
![Page 9: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/9.jpg)
Training a GNB classifier as predicting category
GNB
Class Labels
Last fully connected layer (defines category membership)
Leave one out CV on the 92 image training set
![Page 10: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/10.jpg)
Classification of the 78 test set images
𝑊𝐺𝑁𝐵
Predicted Labels
x
Predicted as Human Faces
Predicted as Animal Faces in the 78 set
Objects-Scenes
Animal Faces
animalsHumanFruits-vegetablesFacesHands
![Page 11: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/11.jpg)
Assigning distances between new test images based on categorical RDM and predicted labels
Test Set Image 1
Test Set Image 2
human face
animal face
Assigned distance0.37
![Page 12: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/12.jpg)
Predicted categorical RDM for the 78 images test data
Same distance for all the images within the same category
![Page 13: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/13.jpg)
Mixing perceptual and categorical components
Large impact on fMRI-ITC and MEG-Late.
𝑅 = 1 − 𝑤2 𝑅𝑝𝑒𝑟 + 𝑤2𝑅𝑐𝑎𝑡
Training data: 92 image set
![Page 14: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/14.jpg)
Results Test set: Perceptual + Categorical RDMs
![Page 15: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/15.jpg)
DNN based RDMs
![Page 16: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/16.jpg)
RDM based on DNN features at one layer
117 𝑥 117
mean 0.12
corrDNN
1
64
1
64
2
63
2
63
Vgg L-1
![Page 17: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/17.jpg)
Model Improvement including DNN Based RDMs
𝑅 = 1 − 𝑤3 𝑅(𝑝𝑒𝑟+𝑐𝑎𝑡) +𝑤3𝑅𝑑𝑛𝑛
Improvement of 𝑅2 (explained variance) in EVCfor the 92 image set
𝑤3 𝑤3𝑤3
![Page 18: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/18.jpg)
Results Test set including DNNs
![Page 19: RDM mixtures for predicting visual cortices responsesalgonauts.csail.mit.edu/slides/Algonauts2019_Agustin_Lage_Castellanos.pdfRDM mixtures for predicting visual cortices responses](https://reader030.fdocuments.us/reader030/viewer/2022040901/5e72654e0ea5fd5c2f7a8461/html5/thumbnails/19.jpg)
Conclusions
• A mixture of perceptual and categorical RDMs made the largest contribution to the prediction accuracy in fMRI-ITC/MEG-Late.
• VGG was the DNN that produced the largest improvement on the model performance.
• However, it is necessary to evaluate the perceptual-categorical vs DNN contribution in the inverse order.