Data Tactics Data Science Brown Bag (April 2014)

The Power of Partnership – from Vision to Reality

L-3 Data Tactics: Data Science Brown Bag

Welcome! Hard and Soft Clusters and Cyber Data April 22, 2014 !R2 = 500; p<.05 asymptotically approaching perfect

!•Why a (our 3rd) Data Science Brown Bag (Rich H.)? !

•About US & About YOU (Rich H.)!!!•Case Studies in Cyber:

•What is Clustering, Honeypots and Density Based Clustering (Max W.)? •What is Optics Clustering and how is it different than DB Clustering? …and how can it be used for outlier detection. (David P.) •What is so-called soft clustering and how is it different than clustering? …and how can it be used for outlier detection. (Nathan D.)

!•On the horizon...(Rich H.)

DT Data Science Brown Bag: Outline

L-3

DT Data Science Brown Bag: Outline

Learning [close] at a pace similar to the pace at which we learn. !Learning and Educating from/to DS to PMs, SWE, and OPs. !

DS2PM: Provide insights for FRIs/RFPs. PM2DS: Atmospherics from our costumers. !DS2SWE: Integrating algorithms. SWE2DS: Accessing data spaces. !DS2OP: How do you consume the outputs of models?

OP2DS: What models are best to present to OPs?

DS: Data Scientist, PM: Program Managers, SWE: Software Engineers, OP: OperatorsL-3

The Team: (Geoffrey B., Nathan D., Rich H., David P., Ted P., Shrayes R., Jonathan T., Adam VE., Max W.) !Graduates from top universities… …many of whom are EMC Data Science Certified. !Advanced degrees include:mathematics, computer science, astrophysics, electrical engineering, mechanical engineering, statistics, social sciences. !Base competencies (horizontals): clustering, association rules, regression, naive bayesian classifier, decision trees, time-series, text analysis. !Going beyond the base (verticals)...

About Us: DT Data Science Team

L-3


L-3

Clustering || Regression || Decision Trees || Text Analysis Association Rules || Naive Bayesian Classifier || Time Series Analysis

econ

ometr

ics

spatia

l econ

ometr

ics

graph

theo

ry alg

orithm

s

astrop

hysica

l time-s

eries a

nalys

is

path

plann

ing alg

orithm

sba

yesian

statis

tics

const

rained

optim

izatio

ns

numeric

al inte

gratio

n tec

hniqu

es

PCA bagg

ing/bo

osting

hierar

chica

l mod

els

IRT

spac

e-time

latent

class

analy

sisstr

uctur

al equ

ation m

odelin

g

mixture

models

SVM

maxent

CARTau

toreg

ressiv

e mod

els

ICA

factor

analy

sis

rando

m fores

t

dimen

siona

l redu

ction

topic m

odels

sentim

ent a

nalys

is

frequ

ency

domain

patte

rns

unsup

ervise

d by s

uperv

ised

chan

ge-po

int mod

els

LUBAPDLIS

A DBAC

optics

cluste

ring

Hierarchy of Data Scientists


L-3

!!No Free Lunch (NFL) theorems: no algorithm performs

better than any other when their performance is averaged uniformly over all possible problems of a particular type. Algorithms must be designed for a particular domain or style of problem, and that there is no such thing as a general purpose algorithm.

!!!


L-3

ABOUT YOU:35 confirmed, 15 webex, 21 Data Tactics employees, 13 L-3 NSS

employees; Sam Posten was the first to sign-up (webex) and Aaron Glahe was the first to sign-up for in-person! !# define Twitter account names start <- getUser(“L3_NSS”) finish <- getUser(“DataTactics”) !# find all connections independently of each account dt.friends.object <- lookupUsers(start$getFriendsIDs()) l3.friends.object <- lookupUsers(finish$getFriendsIDs()) !#create one large table that relates followers from each account relations <- merge(data.frame(User=“DataTactics”, follower=dt.friends), data.frame(User=l3.friends, Followers=“L3_NSS”), all=TRUE) !#create network layout showing each account’s community and overlap g.followers <- graph.data.frame(relations.followers, directed = T) !#finally plot the graph tkplot(g) L-3

ABOUT YOU:

@DataTactics

@L3_NSS

L-3

ABOUT YOU:@L3_NSS

@DataTactics

L-3

Why Clustering?

L-3

Six Pillars of Data Mining: Clustering has become a workhorse in Big Data and fits into the Six Pillars of Data Mining and our own DS4PM & DS4G framework.

• Anomaly detection: the identification of unusual data records, that might be interesting or data errors that require further investigation.

• Association rule learning: searches for relationships between variables. • Clustering: is the task of discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data.

• Classification: is the task of generalizing known structure to apply to new data. • Regression: finds a function which models the data with the least error. • Summarization: providing a more compact representation of the data set.

!Taxonomy of Questions (ref: DS4PM): !

• Causal Effects: is an approach to the statistical analysis of cause and effect based on the framework of potential outcomes

• Classification/Clustering: identifying to which of a set of observations belong, on the basis of a training set of data or without labels in the clustering approach.

• Outlier Detection: is the identification of events which do not conform to an expected pattern or other items in a dataset.

• Big Data and Analytics: discovering interesting relations between variables in large databases • Measurement Models: statistical models to measure the relationships between the observable variables and the unobserved (or “latent”) quantity

• Text Analysis: refers to the process of deriving high-quality information from text.

Max Watson: Max’s background is in physics and applied mathematics. Max completed his undergraduate degree at University of California, Berkeley and completed his PhD at University of California, Santa Barbara in 2012. Max specializes in large-scale simulations, signal analysis and statistical physics - he joined the Data Tactics team in January 2014 and has supported DHS. Max is an EMC Certified Data Scientist. David Pekarek: David’s background is in Mechanical Engineering and specializes in mechanical control systems, optimization, and spatio-temporal statistics. David finished his PhD in 2010 from California Institute of Technology and joined Data Tactics in the fall of 2012 and currently supports DARPA. Nathan Danneman: Nathan’s background is in political science, with specializations in applied statistics and international conflict. He finished his PhD in June of 2013, and joined Data Tactics in May of that same year. He recently co-authored Social Media Mining with R, is active in the local Data Science community and currently supports DARPA. Nathan is an EMC Certified Data Scientist. !!

Today’s presenters:

L-3

L-3

Cluster Analysis of Honeypot Data

By Max Watson

Outline

14

• What are Honeypots? !• Cluster Analysis! -General Principles

-Density Based Clustering!• Cluster Analysis Applied to Honeypot Data!• Conclusions!

L-3

Honeypots

15

8 websites: (USA, 4), (Singapore, 2), (Brazil, 2) [brought to you by Ted Procita]!Collection Period: October 15, 2013 - November 18, 2013!2 Sources of Data: requests at firewall and requests at webserver !number of webserver requests: ~4000

Honeypots are traps set to detect, deflect, or counteract

attempts at unauthorized use of information systems

some information from a typical ‘hit’ on the webserver:!IP address Country Request Timestamp101.227.4.25 CN /robots.txt 10/17/13 17:58:21

L-3

Goals of Honeypot Analysis

16

• Categorize IP addresses in terms of similar requests!• Determine how requests vary by country!• Detect outliers

L-3

Cluster Analysis

17

Grouping similar objects:

Requirements:! 1) Distance metric

!!

2) Method for grouping nearby objectsL-3

Distance I: Combine Requests

18

1) Gather all unique requests invoked by each IP address:!! IP1 ⇒ { /, /robots.txt, …}! IP2 ⇒ { /HNAP1/, /manager/html, …} . . . . . .

L-3

Distance II: Jaccard Similarity

19

Requests from IP address A: {♣,♦} ! Requests from IP address B: {,♦}

Jaccard Similarity:

intersection(A, B) = {♦} union(A, B) = {♣, ♦, }

J(A, B) = 1/3

Effective Distance: D = 1 - J0 1

D = 0 : A and B issue the same requestsD = 1 : A and B issue completely different requests

L-3

Distance III: All Pairs

20

Calculate effective distance between all pairs of IP addresses !...leaving us with:

But usually in a high number of dimensions!

L-3

Identifying Clusters

21

How many clusters are there?Are there outliers?

Density Based Clustering:

● connectivity

● densityL-3

Connectivity

22

Distance

Threshold

Number of Clusters

Distance Threshold

3

2

1

Cluster 1Cluster 2

L-3

Density Based Clustering

23

2 parameters: distance threshold and minimum number of neighbors (DBSCAN)

example:minimum number = 2

OutliersClusters

L-3

Shiny App for Analysis

24L-3

China

25

Dominant Requests of Each Cluster !❶ /robots.txt ❷ / ❸ /manager/html ❹ www.baidu.com/ !

L-3

China

26

Other !/manager/html

Time (~34 Days)

Hits Over Time

Num

ber o

f Hits

L-3

United States

27

10 Clusters (Malicious & Benign)

L-3

United States

28

Clusters !Outliers

from same IP address

Num

ber o

f Hits

Time (~34 Days)

Hits Over Time

L-3

Accomplishments

29

✓ Categorized behavior of IP addresses based on requests!✓ Detection of outliers

!✓ Determined how requests vary by country (China vs. USA)

L-3

What Clustering Can Do for You

30

Objects + Attributes

Cluster the Objects Cluster the Attributes

Applications:!• IP addresses & their requests • patients & their symptoms • devices & their malfunctions • people & their associates

L-3

Port Based Clustering

of Firewall Activity

By David Pekarek

L-3

Firewall Activity Clustering Workflow

Data Preprocessing!and Vectorization

OPTICS !Clustering

Follow-on Investigations

Honey Pot!Firewall Activity

Aggregated counts of IP’s dest. port hits

Reachability Distance plot identifying user clusters and outliers

Characteristics of outlying IPs and IP clusters

#!#!#!#!#!#

Abc Abc Abc Abc Abc Abc

#!#!#!#!#!#

#!#!#!#!#!#

Abc Abc Abc Abc Abc Abc

~32K!logs

time, host, src IP, location, ports, protocol

#!#!#!#!#!#

#!#!#!#!#!#

#!#!#!#!#!#

~19K IPs

128 ports

#!#!#!#!#!#

#!#!#!#!#!#

outliers

clusters

Distinct activity levels on port

53

• The majority of source IPs make use of only one destination port!• 94% of source IPs fall into some cluster with similar port usage and

traffic volume

L-3

OPTICS: Hierarchical Density Based Clustering

• Clustering algorithms provide a means to sort data without pre-existing labels!• Density-based clustering methods are robust in identifying clusters with non-

uniform shapes

• The OPTICS algorithm is a density-based approach that simultaneously evaluates cluster results at different scales

k-Means!results

Density-based!clustering !

results

Is this one cluster or two?!The answer depends on scale!

L-3

OPTICS: How does it work?

• The OPTICS algorithm performs two major operations on the data:!• determining an ordering of all data points, based on the likelihood of points being

clustered together • assigning each point a Reachability Distance (R.D.): a quantification of the length

scale at which the given point will belong to any cluster • Plotting R.D. vs the ordered data points, clusters appear as troughs

Whole face

Eye EyeSmile

Outliers

L-3

OPTICS: How was it applied?

Data Preprocessing!and Vectorization

OPTICS !Clustering

Follow-on Investigations

• Source IPs used as the identifier for entities with traffic hitting the honey pot firewall.

• Destination ports used to define the dimensions of feature space. Each of the 127 most common ports (those with at least 60 hits from the total population) got its own dimension. The remaining ‘rare’ ports bundled as a single dimension.

• OPTICS algorithm identified clusters of IPs in 128 dimensional space, with clustering results summarized in a 2-D reachability plot !

• Follow-on investigations performed to identify anomalous properties of outlying IPs and commonalities among clustered IPs

L-3

Firewall Port Usage Clustering Results

L-3

Firewall Port Usage Clustering Results

Clusters with some!distinctive activity

Outlying IPs!(Their activity falls into clusters only at

extremely generous length scales)

L-3

Interactive Plotting Demo

Interactive Plotting Demo Here

L-3

Port Usage Cluster Characterization

IPs with minimal activity on highly travelled ports

(22, 53, Other)

Outlying IPs:!Activity on multiple ports or very

seldom used ports

1-15 hits on port 80 (HTTP)

1-10 hits on port 3389 (RDP)

1-14 hits on port 1433 (MSSQL)

1-6 hits on port 445 (Active Directory)

Small clusters with activity !on less used ports!

(3306, 5060, 4899, 135, 25, 23, 45091, 48879, 1234)

L-3

Port 53 Traffic Clustering Validation

OPTICS identifies the multimodal distribution of traffic to port 53 (DNS)

L-3

Conclusions

• Destination ports show little correlation in the firewall logs. Source IPs tend to cluster by the one port to which they sent traffic.

• OPTICS clustering efficiently sorts source IPs as outliers or belonging to a cluster of common port usage.

• Interactive plotting tools allow for the rapid characterization of clusters.

L-3

Latent Dirichlet Allocation: Characterizing

normal behavior and identifying deviations

from normality By Nathan Danneman

L-3

Outline

• What is Latent Dirichlet Allocation (LDA)?

• How does it compare to other clustering tools?

• LDA by example: analyzing log files

L-3

LDA is a Mixture Model

• Mixture Models: – Identify sets of variables that co-occur (behavioral patterns) – Determine what behavioral patterns each individual exhibits

• Example: The Sports Equipment Analogy

Golf Clubs Tennis Racket Golf Balls Tennis Balls Baseball Bat

John 12 4

Susan 14 1 6 3

Chris 2 3

Jane 1 11 1

L-3

• Mixture Models: – Identify sets of variables that co-occur (behavioral patterns) – Determine what behavioral patterns each individual exhibits

• Example: The Sports Equipment Analogy


John 12 4

Susan 14 1 6 3

Chris 2 3

Jane 1 11 1

LDA is a Mixture Model

L-3

LDA Utilizes Soft Clustering

• Hard Clustering: every point is assigned to one group

• Hard Clustering with Outliers: every point is assigned to one or no groups

• Soft Clustering: every point is assigned to zero, one, or several groups. x1

x2

L-3

x1

x2


John 12 4

Susan 14 1 6 3

Chris 2 3

Jane 1 11 1

• Hard Clustering: every point is assigned to one group

• Hard Clustering with Outliers: every point is assigned to one or no groups

• Soft Clustering: every point is assigned to zero, one, or several groups.

LDA Utilizes Soft Clustering

L-3

Input Data for LDA: Cyber Data

• LDA takes a matrix of counts

• Data: log files from a large network; 8700 users, 85 log types

• Each row represents a user

• Each column represents a log type

Connection!Success

Termination!Success

Invalid !Login

...

User 1 0 3 2

User 2 12 3 0

User 3 3 0 18

User 4 2 22 1

User 5 7 5 9

... ...

L-3

LDA Estimates Two Mixtures

• Output 1: logs that co-occur, forming behavioral patterns

Log Type 1 !Log Type 2 !Log Type 3 !Log Type 4 ...

Behavioral Pattern 1 !!Behavioral Pattern 2

Each log relates to zero, one, or many behavioral patterns

L-3



• Output 2: which behavioral pattern(s) characterize each user



User 1 !User 2 !User 3 !User 4 ...

Users exhibit zero, one, or many behaviors

L-3

LDA Workflow

• Build the N (observation) by P (log types) matrix of counts

• Use an empirical method to determine the optimal number of behavioral patterns to estimate

• Estimate the model

Connection (Successful)

Connection (Failure)

Termination (Successful)

Connection (Time-Out)

User1 15 15 0 3

User2 8 12 2 0

L-3

Output 1: Mapping Log Types to Behavioral Patterns:Behavioral Pattern #3

Firewall.Connections.Successful

Firewall.Connections.Terminations



L-3

Output 1: Mapping Log Types to Behavioral Patterns:Behavioral Pattern #4



Windows.Hosts.User.Logins

Windows.Hosts.User.Logoffs

Windows.Hosts.User.Privileged.Use.Successful

L-3

Behavioral Pattern Characterization

• Behavioral Pattern 1: • Windows Hosts: Failed Logins

• Behavioral Pattern 2: • Firewall: Connections • Windows Hosts: Logins, Logoffs

• Behavioral Pattern 3: • Firewall: Connections, Terminations

• Behavioral Pattern 4: • Windows Hosts: Logins

• Behavioral Pattern 5: • Firewall: System Normal, Connections, Terminations

• Behavioral Pattern 6: • Web Logs: System Normal

• Behavioral Pattern 7: • Firewall: System Errors

Normal Activity

Abnormal Activity

Abnormal ActivityL-3

Behavioral Pattern Characterization

Wind

ows H

osts:

Faile

d Lo

gins

Firew

all: C

onne

ction

s,

Term

inatio

ns

Wind

ows H

osts:

Login

sFir

ewall

: Sys

tem

Nor

mal,

Conn

ectio

ns, T

erm

inatio

ns

Web

Logs

: Sys

tem

Nor

mal

Firew

all: S

yste

m E

rrors

Firew

all: C

onne

ction

s

Wind

ows H

osts:

Login

s, Lo

goffs

L-3



• Output 2: which behavioral pattern(s) characterize each user



User 1 !User 2 !User 3 !User 4 ...

Users exhibit zero, one, or many behaviors

L-3

Characterizing Users with Behavioral Patterns

User # 2

Essentially, entirely firewall connections and terminations.

L-3

Characterizing Users with Behavioral Patterns

User # 43

Lots of failed logins!

Normal activity: connections, terminations, logins, logoffs

L-3

Visualizing Two-Level Mixtures

Behavioral Pattern 3

Behavioral Pattern 4

A User Characterized by: 45% Behavior 3 and 55% Behavior 4

L-3

Outlier Detection with LDA

• Mixture models make predictions about the proportion of each log type a user will have

• We can compare the predicted proportions to each user’s actual proportions to see how well the model captures each user’s actions

• Typical users should be well-characterized by mixtures of common behavioral patterns – these are “normal” users

• Users whose actions are not mixtures of common behavioral patterns are doing things that are uncommon – these are outliers

L-3

Measuring User-Level Discrepancy

Cosine Similarity = 0.99

Proportions of All Log Types for a Single User

L-3

Measuring User-Level Discrepancy

Cosine Similarity = 0.02

Proportions of All Log Types for a Single User

L-3

Cosine Similarity between Predicted and Observed Data (all users)

~99% of users are well-explained

L-3

Cosine Similarity between Predicted and Observed Data (poorly fit users)

L-3

LDA Detects Univariate Outliers

One user had 77% Windows Hosts Failed Logins; mean for data is 0.002%

Proportion of Windows Hosts: Failed Login Logs

User # 12

L-3

LDA Detects Conditional Outliers

User # 53 has a typical proportion of Firewall Termination logs... !

!!

However, User 53 has more than twice as many Firewall Terminations as users with his/her same proportion of Firewall Connections. !!

Percentage of Logs that are Firewall Terminations

Num

ber

of U

sers

User 53

Firewall Terminations comprise about 50% of many users’ logs

Percentage of Logs that are Firewall Terminations

Num

ber

of U

sers

Firewall Terminations among users with 53’s proportion of Firewall Connections

User 53

L-3

Conclusions

• LDA allows an analyst to: – Succinctly characterize common behavioral patterns – Capture nuance through soft clustering – Identify both simple and conditional outliers !

• Next Steps: – Radically improve parallelized versions of LDA – Build enhanced visualizations that allow analysts to interact with data !

• Previous Steps:– Cyber IR&D II - Honeypots & Topic Graphs

– https://portal.data-tactics-corp.com/sites/analytics/Shared%20Documents/honeypots.pdf

L-3

https://portal.data-tactics-corp.com/sites/analytics/Shared%20Documents/honeypots.pdf

•Query based analytics are tenuous for data with large feature spaces and population sizes. For complete answers, we must analyze with comprehensive algorithms.

•Cyber systems regularly lack reliable (or stationary) models and priors. Hence we have been focused on questions of pattern detection (hard) and outlier detection (harder) for big cyber data, primarily obtaining results via clustering analyses.

•There are many, many clustering algorithms, each with distinct features and requirements (No Free Lunch for Theorems). Choosing the most appropriate tool requires a deep understanding of the available data, the questions at hand, and the pros and cons of applicable methods.

Final Thoughts…

L-3

•L-3 Data Tactics has several minimally viable products (MVP) working of very hard elements of the cyber analytics problem set.

•These MVPs can be used in a support function to existing security protocol and signature based systems - or provide those systems already in place with pattern and anomaly detection.

•Previous and future honeypot collection will further define L-3’s cyber competencies in proactive cyber analytics.

Final Thoughts…

L-3

...on the Horizon: !Honeypots and Twitter Collection Platforms !Summer Data Science Internship Program (Robert R. & USMA cadets): Honeypots analytical application development USA Civil Affairs & CERDEC Analytics http://glimmer.rstudio.com/gosystems01/Stability/ Next Data Science Brown Bag late July. !DS4G & DS4PM both making appearances this year. !Data Science on display at the L-3 Technology Exchange 2014… more to come.

… on the horizon.

L-3

http://glimmer.rstudio.com/gosystems01/Stability/

The Data Science Team

http://datatactics.blogspot.comL-3

http://datatactics.blogspot.com

The Data Science Team

https://github.com/DataTacticsCorpL-3

https://github.com/DataTacticsCorp

Homepage: http://www.data-tactics.comBlog: http://datatactics.blogspot.com

Twitter: https://twitter.com/rheimann

Or, me (Rich Heimann) at [email protected]

Questions?

L-3

Twitter: https://twitter.com/DataTactics

Twitter: https://twitter.com/mwatsonTwitter: https://twitter.com/ndanneman

http://www.data-tactics.com



mailto:[email protected]




Data Tactics Data Science Brown Bag (April 2014)

Data & Analytics

Transcript of Data Tactics Data Science Brown Bag (April 2014)