Image Synthesis with a Single (Robust) ClassifierShibani Santurkar*, Dimitris Tsipras*, Brandon Tran*, Andrew Ilyas*, Logan Engstrom*, Aleksander Madry
Massachusetts Institute of Technology madry-lab.ml
Deep Learning revolutionized Computer Vision
Super-resolution
0
5
10
15
20
25
30
2010 2011 2012 2013 2014 Human 2015 2016 2017
ILSVRC top-5 Error on ImageNet
Image classification
Initial success
Prompted application in many domain-specific tools
Generation Style transfer
and many more!
*approximate timeline
Can we perform complex image synthesistasks using just image classifiers?
How can we use classifiers for input manipulation?Most natural approach: Class maximization [Erhan et al. 2009]
cat: 95%
dog: 3%
…
frog: 0.5%
Goal: Introduce class features by increasing class score
Problem: Standard ML models are brittle
=+ 0.005 x
cat: 95% dog: 99%perturbation
Imperceptible perturbations completely change model predictions
Key ingredient: RobustnessClassifiers need to be invariant to small input changes
“dog” maximizationfor robust classifier
[Tsipras et al. 2019]
Robustness is all you needGoal: Develop a toolkit for image synthesis using robust classifiers→ Just gradient descent on simple loss functions→ No domain-specific priors and regularizers→ Minimal tuning→ Single classifier for all tasks
Image generationClass maximization starting from random noise(sample seed from multivariate Gaussian to ensure diversity)
Image-to-image translationTrain a (robust) classifier to distinguish between domains
horse → zebra apple → orange
Super-resolutionMaximize underlying class to enhance input features
In-paintingMaximize underlying class while matching uncorrupted image
Interactive image manipulationSketch-to-image: Turn crude sketches into “art”
Feature painting: Add features to specific parts of the image
Takeaways→ Robustness is be important beyond security→ Robust classifiers can be powerful primitivesFull paper, blog post, robustness library:
gradsci.orgarXiv:1906.09453 pip install robustness
Top Related