What Does It Mean for Machine Learning to Be Trustworthy?
Transcript of What Does It Mean for Machine Learning to Be Trustworthy?
![Page 1: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/1.jpg)
@NicolasPapernot
What Does It Mean for Machine Learning to Be
Trustworthy?Nicolas Papernot
University of Toronto & Vector Institute
January 2020 – USENIX Enigma
![Page 2: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/2.jpg)
Why is the trustworthiness of ML important? security
2
YouTube filtering
Content evades detection at inference
Microsoft’s Tay chatbot
Training data poisoning
![Page 3: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/3.jpg)
Why is the trustworthiness of ML important? safety
3Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Testing the NVIDIA DAVE-2 self-driving car platform.
![Page 4: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/4.jpg)
Why is the trustworthiness of ML important? privacy
4
Model inversion attack
Adversary learns about the training data from model predictions
Fredrikson et al. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures.
![Page 5: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/5.jpg)
Why is the trustworthiness of ML important? fairness & ethics
5
Percentage of subjects by (skin color, gender) pairs
Classification confidence scores from IBM
Buolamwini and Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
![Page 6: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/6.jpg)
Why is the trustworthiness of ML important? fairness & ethics
6
Percentage of subjects by (skin color, gender) pairs
Classification confidence scores from IBM
Buolamwini and Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
Boneh et al. How Relevant is the Turing Test in the Age of Sophisbots?
DeepfakesImage from Suwajanakorn et al.
Synthesizing Obama: Learning Lip Sync from Audio
![Page 7: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/7.jpg)
How do we design training algorithms that support trust and escape the arms race?
![Page 8: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/8.jpg)
How to define our security policy? A failed attempt
Jacobsen et al. Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
![Page 9: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/9.jpg)
How to define our security policy? A failed attempt
Jacobsen et al. Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
![Page 10: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/10.jpg)
10Is attacking machine learning easier than defending it? [Blog post at www.cleverhans.io]Ian Goodfellow and Nicolas Papernot
Training robust models creates an arms race because we don’t have a
good security policy
![Page 11: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/11.jpg)
Is achieving trustworthy ML any different from real-worldcomputer security?
“Practical security balances the cost of protection and the risk of loss, which is the cost of recovering from a loss times its probability” (Butler Lampson, 2004)
Is the ML paradigm fundamentally different in a way that
enables systematic approaches to security and privacy?
![Page 12: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/12.jpg)
How to define a security privacy policy? A successful attempt
Randomized Algorithm
Randomized Algorithm
Answer 1Answer 2
...Answer n
Answer 1Answer 2
...Answer n
??? ?
Dwork et al. Calibrating noise to sensitivity in private data analysis.
}Differential Privacy:
![Page 13: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/13.jpg)
How to train a model?
Initialize parameters θ
For t = 1..T do
Sample batch B of training examples
Compute average loss L on batch B
Compute average gradient of loss L wrt parameters θ
Update parameters θ by a multiple of gradient average
![Page 14: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/14.jpg)
How to train a model with differential privacy?
Initialize parameters θ
For t = 1..T do
Sample batch B of training examples
Compute per-example loss L on batch B
Compute per-example gradients of loss L wrt parameters θ
Ensure L2 norm of gradients < C by clipping
Add Gaussian noise to average gradients (as a function of C)
Update parameters θ by a multiple of noisy gradient average
Deep Learning with Differential Privacy (CCS, 2016) Abadi, Chu, Goodfellow, McMahan, Mironov, Talwar, Zhang
![Page 15: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/15.jpg)
15Slides adapted from Ulfar Erlingsson
![Page 16: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/16.jpg)
16Slides adapted from Ulfar Erlingsson
![Page 17: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/17.jpg)
17Slides adapted from Ulfar Erlingsson [arxiv:1910.13427]
![Page 18: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/18.jpg)
Architectures, initializations, hyperparameters for DP-SGD learning
Papernot et al. Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy (in submission)
More capacity is not always
helpful
![Page 19: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/19.jpg)
Why is differential privacy in ML successful?• Definition of robustness to adversarial examples using simplistic
distances like L_p norms directly conflicts with generalization• Instead differential privacy encourages generalization
1. No necessary trade-off between privacy and ML objective
2. Degrades smoothly to not learning when it cannot be done privately
![Page 20: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/20.jpg)
What does it mean for ML to be trustworthy?
![Page 21: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/21.jpg)
Whatabouttesttime?
Admissioncontrolmayaddresslackofassurance.
How can sandboxing, input-output validation and compromise recordinghelp secure ML systems when data provenance and assurance is hard?
Auditingmayhelpwithmodelgovernance.
How can compromise recording help secure ML systems throughout their lifetime?
![Page 22: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/22.jpg)
Admission control at test timeWeak authentication (similar to search engines) calls for admission control:
Do we admit a sandboxed model’s output into our pool of answers?
Example: define a well-calibrated estimate of uncertainty to reject outliers (hard when distribution is unknown) through conformal prediction
Deep k-Nearest Neighbors (2018)Papernot and McDaniel
Soft Nearest Neighbor Loss (2019)Frosst, Papernot and Hinton
![Page 23: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/23.jpg)
Machine Unlearning… towards model governance.
Machine Unlearning. Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot (in submission)
![Page 24: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/24.jpg)
Towards trustworthy ML• Policies are needed to align ML with societal norms:
• Security: integrity, confidentiality…• Privacy: differential privacy, confidentiality, …• Ethics: fairness criteria, …
• Technology needs to:• At train time: propose algorithms that satisfy these policies• At test time: can perform admission control and model governance
• Beyond technology, complement with legal frameworks and education
• Trustworthy ML is an opportunity to make ML better
![Page 25: What Does It Mean for Machine Learning to Be Trustworthy?](https://reader033.fdocuments.us/reader033/viewer/2022060718/6297c6f2b133627c347b1de5/html5/thumbnails/25.jpg)
?
Resources: cleverhans.iogithub.com/tensorflow/cleverhansgithub.com/tensorflow/privacy
Contact information:[email protected]@NicolasPapernot
I’m hiring at UofT & Vector:- Graduate students- Postdocs- Faculty positions at all ranks