Image Synthesis with a Single (Robust)...
1
Image Synthesis with a Single (Robust) Classifier Shibani Santurkar*, Dimitris Tsipras*, Brandon Tran*, Andrew Ilyas*, Logan Engstrom*, Aleksander Madry Massachusetts Institute of Technology madry-lab.ml Deep Learning revolutionized Computer Vision Super-resolution 0 5 10 15 20 25 30 2010 2011 2012 2013 2014 Human 2015 2016 2017 Image classification Initial success Prompted application in many domain-specific tools Generation Style transfer and many more! *approximate timeline Can we perform complex image synthesis tasks using just image classifiers? How can we use classifiers for input manipulation? Most natural approach: Class maximization [Erhan et al. 2009] cat: 95% dog: 3% … frog: 0.5% Goal: Introduce class features by increasing class score Problem: Standard ML models are brittle = + 0.005 x cat: 95% dog: 99% perturbation Imperceptible perturbations completely change model predictions Key ingredient: Robustness Classifiers need to be invariant to small input changes “dog” maximization for robust classifier [Tsipras et al. 2019] Robustness is all you need Goal: Develop a toolkit for image synthesis using robust classifiers → Just gradient descent on simple loss functions → No domain-specific priors and regularizers → Minimal tuning → Single classifier for all tasks Image generation Class maximization starting from random noise (sample seed from multivariate Gaussian to ensure diversity) Image-to-image translation Train a (robust) classifier to distinguish between domains horse → zebra apple → orange Super-resolution Maximize underlying class to enhance input features In-painting Maximize underlying class while matching uncorrupted image Interactive image manipulation Sketch-to-image: Turn crude sketches into “art” Feature painting: Add features to specific parts of the image Takeaways → Robustness is be important beyond security → Robust classifiers can be powerful primitives Full paper, blog post, robustness library: gradsci.org arXiv:1906.09453 pip install robustness
Transcript of Image Synthesis with a Single (Robust)...
Image Synthesis with a Single (Robust) ClassifierShibani Santurkar*, Dimitris Tsipras*, Brandon Tran*, Andrew Ilyas*, Logan Engstrom*, Aleksander Madry
Massachusetts Institute of Technology madry-lab.ml
Deep Learning revolutionized Computer Vision
Super-resolution
0
5
10
15
20
25
30
2010 2011 2012 2013 2014 Human 2015 2016 2017
ILSVRC top-5 Error on ImageNet
Image classification
Initial success
Prompted application in many domain-specific tools
Generation Style transfer
and many more!
*approximate timeline
Can we perform complex image synthesistasks using just image classifiers?
How can we use classifiers for input manipulation?Most natural approach: Class maximization [Erhan et al. 2009]
cat: 95%
dog: 3%
…
frog: 0.5%
Goal: Introduce class features by increasing class score
Problem: Standard ML models are brittle
=+ 0.005 x
cat: 95% dog: 99%perturbation
Imperceptible perturbations completely change model predictions
Key ingredient: RobustnessClassifiers need to be invariant to small input changes
“dog” maximizationfor robust classifier
[Tsipras et al. 2019]
Robustness is all you needGoal: Develop a toolkit for image synthesis using robust classifiers→ Just gradient descent on simple loss functions→ No domain-specific priors and regularizers→ Minimal tuning→ Single classifier for all tasks
Image generationClass maximization starting from random noise(sample seed from multivariate Gaussian to ensure diversity)
Image-to-image translationTrain a (robust) classifier to distinguish between domains
horse → zebra apple → orange
Super-resolutionMaximize underlying class to enhance input features
In-paintingMaximize underlying class while matching uncorrupted image
Interactive image manipulationSketch-to-image: Turn crude sketches into “art”
Feature painting: Add features to specific parts of the image
Takeaways→ Robustness is be important beyond security→ Robust classifiers can be powerful primitivesFull paper, blog post, robustness library:
gradsci.orgarXiv:1906.09453 pip install robustness
