Detection and Identification of Network Anomalies Using Sketch Subspaces
description
Transcript of Detection and Identification of Network Anomalies Using Sketch Subspaces
Detection and Identification of Network Anomalies Using
Sketch Subspaces
Xin Li, Fang Bian, Mark Crovella, Christophe Diot, Ramesh Govindan, Gianluca Iannaccone, and Anukool Lakhina
ACM Internet Measurement Conference2006
Speaker: Chang Huan Wu
2009/5/1
2
OutlineIntroductionPrevious ApproachDefeatEvaluationConclusions
3
Introduction (1/3)Unusual traffic patterns arise from
network abuse as well as from legitimate activity
These traffic anomalies are often difficult to detect at a single link and require scrutiny of the entire network
4
Introduction (2/3)Characterizing “normal” traffic
using IP flows representation is intractable– High dimension
Reduce dimension and identify anomalies
5
Introduction (2/3) Previous work aggregate n
etflow into origin-destination (OD) flows
Modify this approach and increases the detection rate while reducing false alarms and identify the IP-flows responsible for the anomaly
Points of Presence, PoPLink
6
Previous ApproachReference: Anukool Lakhina, Mark Cro
vella, Christophe Diot, "Mining Anomalies Using Traffic Feature Distributions," In ACM SIGCOMM 2005
7
Volume vs. Traffic Feature Distribution
Volume based detection schemes have been successful in isolating large traffic changes– But a large of anomalies do NOT cause
detectable disruptions in traffic volume Using traffic feature distribution
– Augments volume-based anomaly detection– Traffic distributions can reveal valuable
information about the structure of anomalies
8
Port scan anomalies viewed in terms of traffic volume and in terms of entropy
But stands out in feature entropy
Port scan dwarfed in volume metrics…
9
Traffic Feature Distributions Anomalies can be detected and distinguished
by inspecting traffic features:– 4-tuple: SrcIP, SrcPort, DstIP, DstPort
Entropy based scheme In volume based scheme, # of packets or bytes per tim
e slot was the variable. In entropy based scheme, in every time slot, the entrop
y of every traffic feature is the variable. This gives us a three way data matrix H.
– H(t, p, k) denotes at time t, the entropy of OD flow p, of the traffic feature k.
To apply subspace method,we need to unfold it into a single-way representation.
11
Normal subspace, : first k principal components
Anomalous subspace, : remaining principal components
Then, decompose traffic on all links by projecting onto and to obtain:
11
Traffic vector at a particular point in time
Normal trafficvector
Residual trafficvector
Subspace Decomposition
12 12
Traffic on link 1
Traffic
on lin
k 2
y
In general, anomalous traffic results in a large value ofUse to identify if it is anomalous
Geometric illustration
Multiway Subspace Method:(Multi-way to single-way)
Decompose into a single-way matrix Now apply the usual subspace decomposition
(PCA)– Every row of the matrix will be decomposed into
# od-pairs
# tim
ebins
H(SrcIP) H(SrcPort) H(DstPort)H(DstIP)
H(s
rcIP
) H(d
stIP
)
H(s
rcPor
t)
H(d
stPor
t)
# od-pairs# od-pairs
# tim
ebins
# tim
ebins
H(SrcIP) H(SrcPort) H(DstPort)H(DstIP)
H(s
rcIP
) H(d
stIP
)
H(s
rcPor
t)
H(d
stPor
t)
14
Defeat (1/2)
Use random aggregations of IP flows (sketches) Put an IP flow into different hash functions (h1,
h2…)
h1
h2
h3
h4
h5
s buckets
R1, SrcIP
h1
h2
h3
h4
h5
s buckets
R2, SrcIP
h1 s bucketsh1 s buckets
…
Entropy of h1
…
Entropy of h1
Entropy of h1
t1
t2
…
Entropy of h1tn
R1
R2
15
Defeat (2/2)
Apply multiway subspace method to each hash function
In all m hash functions, see how many ones are identified as anomalous– Voting approach
Entropy of h1
Entropy of h1
t1
t2
…
Entropy of h1tn
SrcIPEntropy of h1
Entropy of h1
…
Entropy of h1
SrcPortEntropy of h1
Entropy of h1
…
Entropy of h1
DstIPEntropy of h1
Entropy of h1
…Entropy of h1
DstPort
16
Identify Anomalies
Find the element in hash functions that is identified as anomalous
The intersection of the key sets over all hash functions which has raised the alarms, identifies the keys of the IP flows that caused the anomaly (with high likelihood)
Entropy of h1t1 Entropy of h2 Entropy of h3 Entropy of h4
s buckets s buckets s buckets s buckets
17
Evaluation (1/2)
18
Evaluation (2/2)
5 or 6 hash functions is enough If m is the number of hash functions, m−2 or more votes may b
e enough
19
ConclusionUses multiple random traffic
projections to robustly detect anomalies
Higher detection rate and fewer false alarms
Able to automatically infer the IP flows responsible for an anomaly
20
CommentsOnly can handle offline dataCan other fields in packet header
be used for anomaly detection?