ANOMALY DETECTION Scholar: Andrew Emmott Focus: Machine Learning Advisors: Tom Dietterich, Prasad...
1
ANOMALY DETECTION Scholar: Andrew Emmott Focus: Machine Learning Advisors: Tom Dietterich, Prasad Tadepalli Donors: Leslie and Mark Workman Acknowledgements: Funding for my research is provided by the U.S. Army Research Office (ARO) and Defense Advanced Research Projects Agency (DARPA) under Contract Number W911NF-11-C-0088. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Further Acknowledgements: Additional funding for my research is provided by an ARCS Scholar Award from the Portland Chapter of the ARCS Foundation. Thanks! What is Anomaly Detection? Anomaly Detection is the practice of identifying the strangest data points (items, people, events, etc.) from a large selection of normal ones. Sometimes this is also called statistical outlier detection, (where, given a large collection of data, we ask which items are the least likely), but in our work we prefer the more general idea of anomalies because most applications are concerned with identifying some rare class of events that are influenced by some external process. For example, consider a fluke football game where the underdog team won by a very large score. This event was unlikely, but we might only consider it a true anomaly if we learned that the favored team had thrown the match. If it was an honest game, then it was still generated by the normal football game process – it was simply an unlikely outcome. A thrown match is a true anomaly because it is caused by forces outside normal football. Anomaly Normal Normal Normal A more serious example: Consider that a cyber criminal might go to great lengths to make sure all of their actions appear normal. They are a rare class of person, and are not following the same rules that normal people follow, but they might not seem weird at first glance. Thus, it is not easy to establish an all- purpose definition of anomalies. You might be familiar with some applications of Artificial Intelligence such as face recognition. These problems are known as supervised learning problems because the machine is supervised when it is told which images contain faces and which don’t. It simply finds rules that distinguish faces from not- faces. Anomaly detection is an unsupervised learning problem. We want to develop algorithms that can distinguish normal from not-normal without being told which is which. This is more difficult but also more practical; we are asking the machine to find things we are already having trouble finding on our own! NOT A FACE FACE What Are the Applications? There are many real-world applications of anomaly detection. In general, whenever you might wish to identify things that are not normal, anomaly detection might be able to help! Some examples: • Cyber Security & Insider Threat Detection • Normal: Regular Employees • Not Normal: Data Thief • Machine Failure Prevention • Normal: Functioning Jet Engine • Not Normal: Failing Jet Engine • (Also: Elevators, Hard Drives, etc.) • Medical Prognosis • Normal: Healthy Cells • Not Normal: Cancer Cells • Surveillance • Normal: Bank teller is approached with a deposit slip. • Not Normal: Bank teller is approached with a gun. Methodologies Work Anomaly detection is still a nascent field and there is a lack of consistency in how algorithms are evaluated. The bulk of my work has been exploring the efficacy of various methodologies. Different applications might have different characteristics and might allow for different algorithms to succeed. We have urged for a more careful and precise quantification of the qualities of certain anomaly detection. In simpler terms, we try to quantify the following things about any anomaly detection domain: • Point Difficulty – how difficult is it to distinguish the anomalies from the normal points? • Insider threats want to blend in. • Jet engine failures do not. • Relative frequency – How much of the data is anomalous? • Signal failure might be relatively frequent. • Insider threats might be very very rare. • Semantic variation – How similar are the anomalies to each other? • Are they a de facto class? • Or are they simply not normal? • Feature relevance/irrelevance – How well do statistical outliers in the data map to the application target? • An 8 foot tall man is an outlier. • But he is not an insider threat. “Keep your distance. But don’t look like you’re trying to keep your distance.” Algorithms Work 1 2 3 4 1 2 3 4 Cross Prediction We have also worked on developing anomaly detection algorithms of our own. A high level summary of the algorithms I have explored: • Cross Prediction If our data is described by some number of features, we can treat each of those features like its own supervised learning problem. In other words, find weirdness through cross examination. Example: If two people buy five copies of Catcher in the Rye, that might seem weird. If one of them owns a bookstore and the other is unemployed, maybe only one of them seems strange. • Repeated Impossible Discrimination Ensemble (RIDE) Classification techniques can not only discriminate between classes, but many of them can provide a measure of confidence in their decision. If we randomly separate the data into two indiscernible faux-classes, the machine should have low confidence in most points. But what of the points where the machines has high confidence, even on an “impossible” and meaningless classification task? These points are more easily distinguished from all the rest, and therefore, not normal.
-
Upload
bernadette-norris -
Category
Documents
-
view
217 -
download
1
Transcript of ANOMALY DETECTION Scholar: Andrew Emmott Focus: Machine Learning Advisors: Tom Dietterich, Prasad...
ANOMALYDETECTION
Scholar: Andrew Emmott
Focus: Machine Learning
Advisors: Tom Dietterich, Prasad
TadepalliDonors: Leslie and
Mark Workman
Acknowledgements:Funding for my research is provided by the U.S. Army Research Office (ARO) and Defense Advanced Research Projects Agency (DARPA) under Contract Number W911NF-11-C-0088. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
Further Acknowledgements:Additional funding for my research is provided by an ARCS Scholar Award from the Portland Chapter of the ARCS Foundation. Thanks!
What is Anomaly Detection?
Anomaly Detection is the practice of identifying the strangest data points (items, people, events, etc.) from a large selection of normal ones.
Sometimes this is also called statistical outlier detection, (where, given a large collection of data, we ask which items are the least likely), but in our work we prefer the more general idea of anomalies because most applications are concerned with identifying some rare class of events that are influenced by some external process.
For example, consider a fluke football game where the underdog team won by a very large score.
This event was unlikely, but we might only consider it a true anomaly if we learned that the favored team had thrown the match.
If it was an honest game, then it was still generated by the normal football game process – it was simply an unlikely outcome. A thrown match is a true anomaly because it is caused by forces outside normal football.
Anomaly Normal Normal Normal
A more serious example: Consider that a cyber criminal might go to great lengths to make sure all of their actions appear normal.
They are a rare class of person, and are not following the same rules that normal people follow, but they might not seem weird at first glance.
Thus, it is not easy to establish an all-purpose definition of anomalies. You might be familiar with some applications
of Artificial Intelligence such as face recognition.
These problems are known as supervised learning problems because the machine is supervised when it is told which images contain faces and which don’t. It simply finds rules that distinguish faces from not-faces.Anomaly detection is an unsupervised learning problem. We want to develop
algorithms that can distinguish normal from not-normal without being told which is which.
This is more difficult but also more practical; we are asking the machine to find things we are already having trouble finding on our own!
NOT A FACE
FACE
What Are the Applications?
There are many real-world applications of anomaly detection. In general, whenever you might wish to identify things that are not normal, anomaly detection might be able to help!
Some examples:• Cyber Security & Insider Threat Detection
• Normal: Regular Employees• Not Normal: Data Thief
• Machine Failure Prevention• Normal: Functioning Jet Engine• Not Normal: Failing Jet Engine
• (Also: Elevators, Hard Drives, etc.)• Medical Prognosis
• Normal: Healthy Cells• Not Normal: Cancer Cells
• Surveillance• Normal: Bank teller is approached with a deposit slip.• Not Normal: Bank teller is approached with a gun.
Methodologies Work
Anomaly detection is still a nascent field and there is a lack of consistency in how algorithms are evaluated. The bulk of my work has been exploring the efficacy of various methodologies.
Different applications might have different characteristics and might allow for different algorithms to succeed. We have urged for a more careful and precise quantification of the qualities of certain anomaly detection.
In simpler terms, we try to quantify the following things about any anomaly detection domain:
• Point Difficulty – how difficult is it to distinguish the anomalies from the normal points?
• Insider threats want to blend in.• Jet engine failures do not.
• Relative frequency – How much of the data is anomalous? • Signal failure might be relatively frequent.• Insider threats might be very very rare.
• Semantic variation – How similar are the anomalies to each other?• Are they a de facto class?• Or are they simply not normal?
• Feature relevance/irrelevance – How well do statistical outliers in the data map to the application target?
• An 8 foot tall man is an outlier.• But he is not an insider threat.
“Keep your distance. But don’t look like you’re trying to keep your distance.”
Algorithms Work
𝑥1
𝑥2
𝑥3
𝑥4
𝑝1
𝑝2
𝑝3
𝑝4
Cross Prediction
We have also worked on developing anomaly detection algorithms of our own. A high level summary of the algorithms I have explored:
• Cross PredictionIf our data is described by some number of features, we can treat each of those features like its own supervised learning problem.
In other words, find weirdness through cross examination.
Example: If two people buy five copies of Catcher in the Rye, that might seem weird. If one of them owns a bookstore and the other is unemployed, maybe only one of them seems strange.
• Repeated Impossible Discrimination Ensemble (RIDE)Classification techniques can not only discriminate between classes, but many of them can provide a measure of confidence in their decision. If we randomly separate the data into two indiscernible faux-classes, the machine should have low confidence in most points.
But what of the points where the machines has high confidence, even on an “impossible” and meaningless classification task? These points are more easily distinguished from all the rest, and therefore, not normal.
