Open Problems in the Security of Learning
-
Upload
jayme-nguyen -
Category
Documents
-
view
18 -
download
0
description
Transcript of Open Problems in the Security of Learning
![Page 1: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/1.jpg)
![Page 2: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/2.jpg)
Can we learn securely?Learning: key to security (intrusion/spam
detection, etc)
How to learn in a malicious environment? Bad training data
Research Questions: What attacks are feasible? Measurement Defenses
![Page 3: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/3.jpg)
Secure Learning as a Game
Adversarial capability
Adversarial influence
Secure learning
Example: Spam Detection Adversary knows
lexicon Controls fraction of e-
mail Degrades performance (Poisons classifier)
Information
Actions
Actions Actions Actions
![Page 4: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/4.jpg)
Adversarial CapabilityThreat model:
Adversary objective Learning algorithm Typical training data What adversary
controls
Research questions: Formal threat
models Measure threat
impact
Information
Actions
Add words to spam
Poison filter
Probe filter
Find weaknesses
Typical training data
Learning algorithm
Strategies
![Page 5: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/5.jpg)
Attacker
Email Distributio
n
Filter
Contamination
Attacker’s Information
INBOX
Poisoning the Training Set
Learner
Spam
Ham
AttackCorpus
Spam Folder
![Page 6: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/6.jpg)
Stealthy AttacksGoal: Undetectable, covert attacks
Covert attacks are more effective Untraceable
Research Questions: Measure visibility of attack Provably undetectable attacks Defenses
![Page 7: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/7.jpg)
Effort-Impact Trade-offsHow much adversary
effort? Depends on attack
type Data contamination Probing learner
Research Questions: Bound on adversarial
effort Bound # of learner
mistakes
Actions
Actions Actions
Best forAdversar
y
Best forLearner
no-op poison
Naive spam filter
![Page 8: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/8.jpg)
Dictionary AttackMake spam filter unusable
misclassify ham as spam
Spammer
![Page 9: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/9.jpg)
Dictionary AttackInitial Inbox: 10K
messages
Attacks Black: Optimal
Red: English dictionary
Blue: 90K most common words in Usenet
![Page 10: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/10.jpg)
ProbingGoal: Adversary wants to reverse engineer
classifier Adversary probes learner state through queries Adversary finds classifier vulnerabilities
Research Questions: Complexity of classifier reverse engineering Develop classifiers resistant to reverse
engineering
![Page 11: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/11.jpg)
DefensesGoal: secure learning
Malicious environments Range of threats
Research Questions: Build secure learning Robust against
contamination Measure robustness
Actions
Actions Actions
no-op poison
Naive spam filterRobust spam filter
![Page 12: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/12.jpg)
DefensesReject on Negative Impact (RONI)
Method Assess impact of query message on training Exclude messages with large negative impact
Preliminary Results Perfectly identifies dictionary attacks Unable to differentiate focused attacks
SpamBayes Filter
SpamBayes Learner
![Page 13: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/13.jpg)
Designing Security Sensitive Learners
Generally difficult to design defenses for a particular adversary
Robust Learning – learners are resilient to limited amount of arbitrary contamination. Robustness measures provide comparison between
procedures
QUESTION: What are useful measures of a procedure’s robustness for designing security sensitive learners? Which learners perform well w.r.t. these measures?
![Page 14: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/14.jpg)
Orthogonal Learners Idea: use parallel independent learners
Orthogonal learners have different strengths & vulnerabilities
Attack must succeed against entire group
Research Questions: Design orthogonal learners Automatic generation of orthogonal learners
![Page 15: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/15.jpg)
Future of Secure MLTraditional machine learning: no malicious
environment
Contrast with Secure ML Open research area Vital for use of machine learning in security
Strong preliminary results Showed attack feasibility on Bayesian spam detection Showed attack feasibility on principal component
analysis Evidence of activity in the wild
Prediction: area is ready to move
![Page 16: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/16.jpg)
![Page 17: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/17.jpg)
![Page 18: Open Problems in the Security of Learning](https://reader036.fdocuments.us/reader036/viewer/2022062422/56812ca2550346895d914871/html5/thumbnails/18.jpg)
Characterizing Reasonable Capabilities
What are common adversarial actions? Actions limited by capabilities & information
adversary has
QUESTION: Which forms of adversarial control are tolerable? What sorts of information? Controls?
QUESTION: What are the trade-offs between the learner’s generalization capacity & the adversary’s capabilities?