Applied Machine Learning in Software Security · Applied Machine Learning in Software Security ......

Approved for Public Release; Distribution is Unlimited

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Applied Machine Learning in Software SecurityEliezer Kanal

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.

Carnegie Mellon® and CERT® are registered marks of Carnegie Mellon University.

DM-0004563

Tom Mitchell, former CMU Machine Learning department chair:

The field of Machine Learning asks the question, “How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?”

Machine Learning seeks to automate data analysis and inference.

What is Machine Learning?

If your problem can be stated as either of the following:

…you would likely benefit from machine learning.

Iwouldliketouse_____datatopredict_____.Iwouldliketouse____data

toguesswhat____is.

Sample Techniques:

• Regression

• K-Means Clustering

Clustering image: Weston.pace, https://commons.wikimedia.org/wiki/File:K_Means_Example_Step_4.svg

Feature Engineering:Using existing data to create more informative data

DataTypes

Image StaticVideo

Timeseries FinancialdataEvent counts

Structuredtext WebformsStructureddata

(JSON,XML)Sourcecode

Freetext NewsTweetsEmail

manymore…

Examples:

o I would like to use incident ticket data to predictcustomer needs .

o I would like to use publicly available code to predict what code I will write .

o I would like to use bug report data to guess the location of undetected bugs in my code .

Examples:

o I would like to use incident ticket data to predictcustomer needs .

o I would like to use publicly available code to predict what code I will write .

o I would like to use bug report data to guess the location of undetected bugs in my code .

Applied ML:Vulnerability Detection

Applied ML: Vulnerability Detection

Analyzer

Codebases

Alerts Today3,147

11,772

48,690

10,000

20,000

30,000

40,000

50,000

60,000

TP FP Susp

Manyalertsleftunaudited!

Analyzer

Codebases

Alerts Today

OurGoal

11,772

48,690

10,000

20,000

30,000

40,000

50,000

60,000

TP FP Susp

66effortdays

12,076

45,172

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

50,000

e-TP e-FP I

AutomatedStatisticalClassifier

• ExpectedTruePositive(e-TP)• ExpectedFalsePositive(e-FP)• Indeterminate(I)

ClassifiersLassoLogisticRegressionCARTRandomForestExtremeGradientBoosting(XGBoost)

Some ofthefeatures usedAnalysistoolsused Tokensinfunc/methodSignificantLOC Alertsinfunc/methodComplexity AlertsinfileCoupling MethodsinfileCohesion SLOCinfileSEIcodingrule Avg TokensFunction/methodlength Avg SLOCSLOCinfunc/method Depthincoderepository#parametersinfunc/meth.

Cyclomatic complexity(func/meth)

Significant improvement!

• 91% Classifier accuracy overall

• Specific rule accuracy at right

• 10x developer time saved!

RuleID LassoLRRandomForest CART XGBoost

INT31-C 98% 97% 98% 97%EXP01-J 74% 74% 81% 74%OBJ03-J 73% 86% 86% 83%FIO04-J* 80% 80% 90% 80%EXP33-C* 83% 87% 83% 83%EXP34-C* 67% 72% 79% 72%DCL36-C* 100% 100% 100% 100%ERR08-J* 99% 100% 100% 100%IDS00-J* 96% 96% 96% 96%ERR01-J* 100% 100% 100% 100%ERR09-J* 100% 88% 88% 88%

* Small quantity of data

RuleID LassoLRRandomForest CART XGBoost

INT31-C 98% 97% 98% 97%EXP01-J 74% 74% 81% 74%OBJ03-J 73% 86% 86% 83%FIO04-J* 80% 80% 90% 80%EXP33-C* 83% 87% 83% 83%EXP34-C* 67% 72% 79% 72%DCL36-C* 100% 100% 100% 100%ERR08-J* 99% 100% 100% 100%IDS00-J* 96% 96% 96% 96%ERR01-J* 100% 100% 100% 100%ERR09-J* 100% 88% 88% 88%

Applied ML:Malware family classification

Applied ML: Malware family classification

ReverseEngineering

Discovery

Refinement

Reflection

NewFamily

ArtifactCatalog

Signature 1Signature 2Signature 3

Files 1a 1b 1c 1d …Files 2a 2b 2c 2d …Files 3a 3b 3c 3d …

Whichfileshang

together?

SignalFlowgraphhighlightsbehaviorrelatingdifferent

malwarefamilies

Programinstructionanalysisshowssimilarityanddiversionofbehavior

StaticAnalysisidentifiesprogramswithsimilarsourcecode

Simplify visualization of extremely complex data through the use of dimensionality reduction and associated visualization techniques

Chernoff faceexperiment

• Ground Truth: SVM trained with expert ground truth labels.

• Turkers Avg: Classifier trained with layperson labels.

Performance surprisingly similar!

Applied ML:Software cost estimation

RobertFerguson,DennisGoldenson,JamesMcCurley,RobertW.Stoddard,DavidZubrow,DebraAnderson.“QuantifyingUncertaintyinEarlyLifecycleCostEstimation(QUELCE)”.Dec2011.http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=10039

Applied ML: Software cost estimation

GeneralAccountingOffice.DefenseAcquisitions:AKnowledge-BasedFundingApproachCouldImproveMajorWeaponSystemProgramOutcomes.ReporttotheCommitteeonArmedServices,U.S.Senate,July2008,GAO-08-619.

Applied ML:Incident report mapping

Applied ML: Incident report mapping

International partners

Agency infrastructure

Threat actors

Phishing campaign

Agency response

teams US-CERT infrastructure

Malware class

Compromised website

Common reference websites

Observation

Landscape Tickets

Inference

Applied ML: Incident report mapping

Indicators across ticketsIndicatorsoccurwithdiversepatternsacrosstickets,reportersandtime.Timeonxaxis,countonyaxis,colorcodedbyreporter.

MaliciousIP

AgencyIP

US-CERTdomain

Similarity of indicatorsBeginningwithareferenceindicator,wefindindicatorssimilartoit.Example:amaliciousIP

• Coloredcirclesaretickets• Greycirclesareindicators• Largeindicatorsnear

centerofcirclehavesimilaroccurrencepatternstothereferenceindicator.

Indicator communitiesButwhatifwearen’tstartingwithareferenceindicator?Weassumethatindicatorsgeneratedbyacoherentrealworldprocesswillbemorelikelytoco-occurinticketsthanarbitrarypairsofindicators.Findgroupsofhighlysimilarindicatorsincompleteindicator-ticketgraph.

Indicator-ticket graph

Asubsetoftheticket-indicatorgraph(forasmallsetofselectedindicators)• Ticketsaregreytriangles• Indicatorsareblackcircles• Edgesconnectticketstotheindicatorsthey

contain

Contact Information

Eliezer KanalTechnical ManagerTelephone: +1 412.268.5204Email: ekanal@sei.cmu.edu

Applied Machine Learning in Software Security · Applied Machine Learning in Software Security ......

Documents

Transcript of Applied Machine Learning in Software Security · Applied Machine Learning in Software Security ......

COMP 551 -Applied Machine Learning Lecture 18 ...wlh/comp551/slides/18-unsupervised.pdf · COMP 551 -Applied Machine Learning Lecture 18 ---Unsupervised learning William L. Hamilton

Naive Bayes Applied Machine Learning - cs.mcgill.ca

Applied Machine Learning - Classes

Machine Learning applied to CERN’s Industrial Control Systems

Applied Machine Learning For Systematic Equities Trading ...

An Empirical Study of Machine Learning Algorithms Applied ...ftp.cs.wisc.edu/machine-learning/shavlik-group/geisler.thesis.pdf · An Empirical Study of Machine Learning Algorithms

Introduction to Applied Machine Learning in R

Machine Learning applied to Go

[POLS 8500] Applied Machine Learning - Harvard UniversityAbout MeApplicationsAbout POLS 8500Intro to machine learning [POLS 8500] Applied Machine Learning Professor Jason Anastasopoulos

Adversarial Machine Learning Applied to Intrusion and ... · N. Martins et al.: Adversarial Machine Learning Applied to Intrusion and Malware Scenarios: Systematic Review of machine

COMP551 Applied Machine Learning 1 Introductionsarathchandar.in/.../comp551/notes/02_Introduction.pdf · Applied Machine Learning 1 Introduction Instructor: Sarath Chandar. Applied

Machine Learning Applied to Musical Improvisation

APPLIED MACHINE LEARNING: PROJECT MANAGEMENT … · Applied Machine Learning: Project Management Performance Prediction| Sefa O. Saglam i APPLIED MACHINE LEARNING: PROJECT MANAGEMENT

CIS 519/419 Applied Machine Learning

COMP 551 –Applied Machine Learning Lecture 3: Linear ...€¦ · COMP-551: Applied Machine Learning • Definition and characteristics of a supervised learning problem. • Linear

CS 438/697 - Syllabus Applied Machine Learning · 2015. 2. 23. · CS 438/697 - Syllabus Applied Machine Learning 4. know how to apply appropriate machine learning algorithms to a

Machine Learning applied to Process Scheduling · Machine Learning applied to Process Scheduling Benoit Zanotti Introduction and deﬁnitions Machine Learning Process Scheduling Our

CIS 519: Applied Machine Learning

MACHINE LEARNING TECHNIQUES APPLIED TO ALZHEIMER’S ...

Introduction To Applied Machine Learning