Post on 12-Oct-2020
Zurich Research Laboratory
IBM | Jul 06 | Network Anomaly Detection www.zurich.ibm.com/aurora
Andreas KindPaul HurleyJeroen MassarXenofontas Dimitropoulos
Network Anomaly Detection Based on Behavioral Traffic Pattern Recognition
2
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Network AnomaliesUnusual and significant changes in network traffic characteristics
– Data volume (octets, packets)
– Flows (number, duration, size, service type)
– Communication matrix (src/dst IP, src/dst ports)
– Packets (size, flags)
Caused by…– “Season”
– Organizational change (eg, new application, new user group, new business process)
– Flash crowd
– Vulnerability scan
– Outage, fault, misconfiguration (eg, port scanning AFS, DNS used by IDS)
– DoS attack, self-propagating attack (virus, worm)
– Research on networks
3
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Network Anomalies
IPIPxIPPorts
FlowsOctets
Flow duration
PacketsOctets/packets
4
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Detection Requirements
Scalable for data centersNo additional equipment (eg, splitters, taps, meter appliances)No traffic insertion (eg, active probing)No agents, no credentialsNo access to traffic payloadNo increase in monitoring trafficReal-time operationLow hardware costsNo explicit configuration of thresholds and confidence intervalsApplicable to highly varying workloads
… which is a bit of a contradiction
No automatic prevention, no prediction, but deployment in combination with flow-based network profiling system
– Which are the end-to-end flows causing the anomaly?
5
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Related Work
Signature-based approaches– Too slow, payload needed, only know worms/viruses are addressed
Statistical approaches– Typically based on abrupt changes and therefore error-prone with varying
workloads in distributed environments
Rule-based approaches– Difficult to train, complex rule-sets too slow
Service spoofing– Traffic destined to unused addresses is a priori suspicious
– Most effective for worms
Pattern-based approaches– Capture traffic patterns from network characteristics and compare with
baseline pattern
– How to compose and compare traffic patterns in order to address detection requirements?
6
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
.
.
.
.
.
.Time
Behavioral Traffic Patterns(ie, histograms of measured network parameters during time period dx)
OK
OK
not OK (ie, traffic anomaly)
OK
Measured network parameters (via NetFlow/IPFIX only)
dx
Desired Detection System
7
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Network Traffic PatternsDefined as histograms that display the frequency of flow parameter ranges during observation periodExamples: IP address range, TCP/UDP port range, flow duration
8
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Network Traffic Patterns
9
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Behavioral Analysis of Virus ActivityS
rc A
ddr
Mon Tue
Dst
Add
r
Wed Thu Fri
Host scan prefix mask 255.0.0.0prefix mask 0.255.0.0
10
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Distance Between Traffic PatternsDefined as the number of changes in the relative order between two patterns
0 if ( w1[i]≥w1[mod(i,n)+1] ∧w2[i]≥w2[mod(i,n)+1] ) ∨
ord(w1[],w2[],i) = ( w1[i]≤w1[mod(i,n)+1] ∧w2[i]≤w2[mod(i,n)+1] )
1 otherwise
ExampleGiven w1 = (1,2,3,4), w2 = (0,7,2,1)
ord(w1,w2,1) = 0
ord(w1,w2,2) = 1
ord(w1,w2,3) = 1
ord(w1,w2,4) = 0
11
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Distance Between Traffic PatternsDistance function
d(w1, w2) = 1/n∑ ord(w1, w2, i) for 0 < i ≤ n
ExampleGiven w1 = (1,2,3,4), w2 = (0,7,2,1)
d(w1, w2) = 1/4 * 2 = 0.5
12
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Archetypes
13
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Clustering Traffic Patterns
Tree Clustering– Joining patterns into successively larger clusters using distance function
– Results in hierarchical tree
– But: How to determine mean (most likely “dummy”) pattern for which variability in distances to other cluster members is the smallest?
k-Means Clustering– Given fixed member of k clusters
– Assign patterns to clusters so that overall variability in distances to other cluster members is minimized
14
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Traffic Pattern Archetypes
Traffic pattern archetypes are computed with k-means clustering
Find w1, …, wi ∈ W so that ∑ MIN d(wi , wk) with wk ∈ W \ {w1, …, wi} is minimized
– Find the i patterns for which the sum of the minimum distances to all other patterns is minimized
– We used i up to 4
i
15
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Validation
Abnormal patterns
e
Maximum distance of baseline pattern to archetypes
Patterns ordered by distance to archetypes
Dis
tanc
e
Threshold at 0.25
16
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Distance is above
threshold?
. . .Training set of patterns
Select Archetypes
. . .
Archetypes
System in normal
operation
Determine distance of
current pattern from
archetypes
Mark pattern
OK
Mark pattern not OK
no
yes
Step 1 Step 2
Step 3
Determine threshold
System in current
operation
Get current pattern
Initialization
Start
Threshold
17
Zurich Research Laboratory
A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation
Future Work
http://www.zurich.ibm.com/aurora
Continue the theoretic and empirical work on this approachExperiment with different distance functions and clustering algorithmsProve the time advantage of behavioral network problem prediction
Close integration with IBM’s flow-based network profiling systemUse approach with server workloadsVisualization with force-directed graphs (ie, attractive/repulsive forces)
– ip08, duration, …
THANKS!
Time
Max
Failure & threshold events
Impact
∆t1
Behavioral abnormality
∆t2
Avalanche analogy