Sparse coding for machine learning, image processing and ...
AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning...
Transcript of AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning...
![Page 1: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/1.jpg)
Sparse Coding Trees with Application to Emotion Classification
Kevin H.C. ChenMarcus Z. ComiterH. T. KungBradley McDanel
AMFG 2015
Harvard University
![Page 2: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/2.jpg)
Business Applications
User feedback systems, advertising, security systems
Politics
Automatic derivation of voter preferences, focus group testing
Medicine
Additional metrics for patient care, helping children with autism
Application Motivation
Emotion Classification for IoT and Beyond
![Page 3: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/3.jpg)
Methodology Motivation
Machine Learning and Unsupervised Feature Extraction
Feature 1
Feat
ure
2
- Sparse coding makes data more linearly separable
- Labels are not required
![Page 4: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/4.jpg)
Sparse Coding Pipeline for Classification
Sparse Coding
Classifier (e.g., SVM)
Transform data into feature representation
Prediction with simple classifiers such as SVM
x
z
label
Unsupervised
Supervised
![Page 5: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/5.jpg)
Representation by Sparse Coding
argminz || x - Dz|| 2 + λ|| z|| 02
Express the input signal (x) as the weighted (z) sum of a few features (D)
Note: we can also penalize L1 norm instead of L0 norm
![Page 6: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/6.jpg)
Dictionary Learning
argminD,Z || X - DZ|| 2 + λ|| Z || 0
Sparse Coefficients
- Finds common patterns in training data- Solved by alternating updates of D and Z
2
![Page 7: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/7.jpg)
Our Enhancement to SC
Sparse Coding tree (SC-tree)to learn features with hierarchy
Non-negative constraintsto mitigate over-fitting in SC
Mirroringto increase variation tolerance
![Page 8: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/8.jpg)
Sparse Coding TreeLearning Features for Hard Cases
- Some discriminating features can be subtle- Finding clusters within clusters, similar to how
hierarchical k-means works
Fear can be confused with happiness because they both display teeth
![Page 9: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/9.jpg)
Constructing Sparse Coding Tree
Input
Sparse Coding
Classifier (e.g., SVM)
Group/Label Assignment
label
label label
If certain classes get confused consistently, put them through another layer of feature extraction
![Page 10: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/10.jpg)
anger
contempt
sadness
disgust
fear
happiness
surprise
Branching in Sparse Coding TreeBased on the confusion matrix from the coarse predictor
![Page 11: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/11.jpg)
Features Learned in SC-tree
Features learned in the root node
Features learned in a happy v.s. fear node
happy
Could be happy or fear
fear
Input
label label
label
![Page 12: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/12.jpg)
maxpoolingsplit
Sparse coding(LASSO/OMP)
flip
Sparse coding(LASSO/OMP)
Mirroring for Reflection InvarianceUsing max pooling to capture the horizontal symmetry inherent in emotion classification
A reflected image would get the exact same representation
![Page 13: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/13.jpg)
Improved Robustness with MirroringWith max pooling, we always pick up response from the side of face with stronger features
![Page 14: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/14.jpg)
Nonnegative Sparse Coding
argminz || x - Dz|| 2 + λ|| z|| 02
s.t. D ≥ 0, z ≥ 0D with NN-constraintD without NN-constraint
Tends to learn regional components
Nonnegativity prevents cancelation of components, and therefore mitigates over-fitting
![Page 15: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/15.jpg)
Datasets
Cohn-Kanade Extended Dataset (CK+)Emotions in the Wild Dataset (EitW)GENKI-4K DatasetAM-FED Dataset
CK+
GENKI
AM-FED
Multi-classMulti-class
BinaryBinary
after pre-processingoriginal data
![Page 16: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/16.jpg)
Performance on Emotion Classification
Results reported in average recall
The sparse coding tree improves the performance of our pipeline consistently.
79.9
SC NNSC MNNSC
75.1
70.1
73.6
71.5
76.8
w/SC Tree
w/o SC Tree
CK+ dataset
![Page 17: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/17.jpg)
Performance on Emotion Classification
Results reported in average recall
The sparse coding tree improves the performance of our pipeline consistently.
33.0
SC NNSC MNNSC
29.4
26.5
28.628.1
29.7
w/SC Tree
w/o SC Tree
EitW dataset
![Page 18: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/18.jpg)
MNNSC Performance
Results reported in area under curve
with Mirroring and the non-negativity constraint, even greedy methods like OMP (L0) can be competitive
90.0
L0-min L1-min
best reported96.1
L0-min L1-min
best reported95.1
96.792.396.2
95.7
93.1
91.2
89.7
92.1
88.8
86.0
97.0
sparse coding
Non-negativity
Mirroring
GENKI-4K AM-FED
![Page 19: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/19.jpg)
Applying Sparse Coding Tree to Action Recognition
Tested on KTH dataset
with SC Tree 92.13 %without: 86.57 %
![Page 20: AMFG 2015 Sparse Coding Trees with Application to Emotion ... · Sparse Coding Tree. Learning Features for Hard Cases - Some discriminating features can be subtle - Finding clusters](https://reader033.fdocuments.us/reader033/viewer/2022041818/5e5c77db2699f52e4969cfa0/html5/thumbnails/20.jpg)
ConclusionSparse coding, as an effective feature extraction method, can be enhanced by these techniques:
Sparse Coding tree (SC-tree)to learn features with hierarchy
Non-negative constraintsto mitigate over-fitting in SC
Mirroringto increase variation tolerance