A Closer Look at Adversarial Examples for Separated...
Transcript of A Closer Look at Adversarial Examples for Separated...
![Page 1: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/1.jpg)
Kamalika Chaudhuri
A Closer Look at Adversarial Examples for Separated Data
University of California, San Diego
![Page 2: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/2.jpg)
Adversarial Examples
Small perturbation to legitimate inputs causing misclassification
Panda Gibbon
![Page 3: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/3.jpg)
Adversarial Examples
Can potentially lead to serious safety issues
![Page 4: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/4.jpg)
Adversarial Examples: State of the Art
A large number of attacks
A few defenses
Not much understanding on why adversarial examples arise
![Page 5: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/5.jpg)
Adversarial Examples: State of the Art
A large number of attacks
A few defenses
Not much understanding on why adversarial examples arise
This talk: A closer look
![Page 6: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/6.jpg)
Background: Classification
Given: ( xi, yi )
Vector of features
Discrete Labels
Find: Prediction rule in a class to predict y from x
![Page 7: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/7.jpg)
Background: The Statistical Learning Framework
Training and test data drawn from an underlying distribution D
+
-
![Page 8: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/8.jpg)
Background: The Statistical Learning Framework
Training and test data drawn from an underlying distribution D
Goal: Find classifier f to maximize accuracy
Pr(x,y)⇠D
(f(x) = y)
+
-
![Page 9: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/9.jpg)
Measure of Robustness: Lp norm
A classifier f is robust with radius r at x if it predicts f(x) for all x’ in
kx� x0kp r
![Page 10: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/10.jpg)
Why do we have adversarial examples?
![Page 11: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/11.jpg)
Data distribution +
- Distributional Robustness
Why do we have adversarial examples?
![Page 12: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/12.jpg)
Data distribution
Too few samples
+
-
+
-
Distributional Robustness
Finite SampleRobustness
Why do we have adversarial examples?
![Page 13: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/13.jpg)
Data distribution
Too few samples
+
-
+
-
Bad algorithm
+
-
Distributional Robustness
Finite SampleRobustness
AlgorithmicRobustness
Why do we have adversarial examples?
![Page 14: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/14.jpg)
Data distribution +
- Distributional Robustness
Why do we have adversarial examples?
Are classes separated in real data?
![Page 15: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/15.jpg)
r-Separation
Data distribution D is r-separated if for any (x, y) and (x’, y’) drawn from D
y 6= y0 =) kx� x0k � 2r
2r 2r
![Page 16: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/16.jpg)
r-Separation
Data distribution D is r-separated if for any (x, y) and (x’, y’) drawn from D
y 6= y0 =) kx� x0k � 2r
2r 2r
r-separation means accurate and robust at radius r classifier possible!
2r
![Page 17: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/17.jpg)
Real Data is r-Separated
Separation Typical rDataset
MNIST 0.74 0.1
CIFAR10 0.21 0.03
SVHN* 0.09 0.03
ResImgnet* 0.18 0.005
Separation = min distance between any two points in different classes
![Page 18: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/18.jpg)
Robustness for r-separated data: Two Settings
Non-parametric Methods
![Page 19: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/19.jpg)
Non-Parametric Methods
k-Nearest Neighbors Decision Trees
Others: Random Forests, Kernel classifiers, etc
![Page 20: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/20.jpg)
What is known about Nonparametric Methods?
![Page 21: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/21.jpg)
The Bayes Optimal Classifier
Classifier with maximum accuracy on data distribution
Only reachable in the large sample limit
![Page 22: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/22.jpg)
What is known about Non-Parametrics?
With growing training data, accuracy of non-parametric methods
converge to accuracy of the Bayes Optimal
![Page 23: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/23.jpg)
What about Robustness?
Prior work: Attacks and defenses for specific classifiers
Our work: General conditions when we can get robustness
![Page 24: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/24.jpg)
What is the goal of robust classification?
![Page 25: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/25.jpg)
What is the goal of robust classification?
Bayes optimal undefined outside distribution
Bayes optimal
![Page 26: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/26.jpg)
The r-optimal [YRZC20]
Bayes optimal undefined outside distribution
r-optimal = classifier that maximizes accuracy at points that have robustness radius at least r
Bayes optimalr-optimal
x
![Page 27: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/27.jpg)
Convergence Result [BC20]
2r
Theorem: For r-separated data, condi-tions when non-parametrics converge to r-optimal in large n limit
![Page 28: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/28.jpg)
Convergence Result [BC20]
2r
r-optimal: Nearest neighbor, Kernel classifiers
Convergence limit:
Theorem: For r-separated data, condi-tions when non-parametrics converge to r-optimal in large n limit
![Page 29: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/29.jpg)
Convergence Result [BC20]
2r
r-optimal: Nearest neighbor, Kernel classifiers
Bayes-optimal but not r-optimal: Histograms, Decision trees
Convergence limit:
Theorem: For r-separated data, condi-tions when non-parametrics converge to r-optimal in large n limit
![Page 30: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/30.jpg)
Convergence Result [BC20]
2r
r-optimal: Nearest neighbor, Kernel classifiers
Bayes-optimal but not r-optimal: Histograms, Decision trees
Convergence limit:
Theorem: For r-separated data, condi-tions when non-parametrics converge to r-optimal in large n limit
Robustness depends on training algorithm!
![Page 31: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/31.jpg)
Robustness for r-separated data: Two Settings
Non-parametric Methods
Neural Networks
![Page 32: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/32.jpg)
Robustness in Neural Networks
A large number of attacks
A few defenses
All defenses show a robustness-accuracy tradeoff
Is this tradeoff necessary?
![Page 33: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/33.jpg)
The Setting: Neural Networks
Neural network computes function f(x)
Classifier output sign(f(x))
x
f(x)
0
f
![Page 34: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/34.jpg)
The Setting: Neural Networks
Neural network computes function f(x)
Classifier output sign(f(x))
x
f(x)
0
If f is locally Lipschitz around x, and f(x) is away from 0,then f is robust at x
f
Robustness comes from Local Smoothness:
![Page 35: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/35.jpg)
Robustness and Accuracy Possible through Local Lipschitzness
x
f(x)
0
Theorem [YRZSC20] If distribution is r-separated, then there exists an f s.t. f is locally smooth and sign(f) has accuracy 1 and robustness radius r
Neural network computes function f(x)
Classifier output sign(f(x))
f
![Page 36: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/36.jpg)
In principle, no robustness-accuracy tradeoff
In practice there is one
What accounts for this gap?
![Page 37: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/37.jpg)
Empirical Study
4 standard image datasets
7 models
6 different training methods - Natural, AT, Trades, LLR, GR
Measure local Lipschitzness, accuracy and adversarial accuracy
![Page 38: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/38.jpg)
Result: CIFAR 10
![Page 39: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/39.jpg)
Observations
Trades and Adversarial training have best local Lipschitzness
Overall, local Lipschitzness correlated with robustness and accuracy - until underfitting begins
Generalization gap is quite large - possibly a sign of overfitting
Overall: robustness/accuracy tradeoff due to imperfect training methods
![Page 40: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/40.jpg)
Data distribution
Too few samples
+
-
+
-
Bad algorithm
+
-
Distributional Robustness
Finite SampleRobustness
AlgorithmicRobustness
Conclusion: Why do we have adversarial examples?
![Page 41: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/41.jpg)
References
Robustness for Non-parametric Methods: A generic defense and an attack, Y. Yang, C. Rashtchian, Y. Wang, and K. Chaudhuri, AISTATS 2020
When are Non-parametric Methods Robust?R. Bhattacharjee and K. Chaudhuri, Arxiv 2003.06121
Adversarial Robustness through Local LipschitznessY. Yang, C. Rashtchian, H. Zhang, R. Salakhutdinov and K. Chaudhuri, Arxiv 2003.02460
![Page 42: A Closer Look at Adversarial Examples for Separated Datacseweb.ucsd.edu/~kamalika/slides/NFMAISafetyMay20.pdfExamples for Separated Data University of California, San Diego. Adversarial](https://reader033.fdocuments.us/reader033/viewer/2022042408/5f23c3b7f46d6e774b3d2e65/html5/thumbnails/42.jpg)
Acknowledgements
Cyrus
Rashtchian
Yaoyuan
Yang
Yizhen
Wang
Hongyang
Zhang
Robi
Bhattacharjee
Ruslan
Salakhutdinov