1

Unveiling Anomalies in Large-scale Networks via Sparsity and Low Rank

Morteza Mardani, Gonzalo Mateos and Georgios Giannakis

ECE Department, University of Minnesota

Acknowledgments: NSF grants no. CCF-1016605, EECS-1002180

Asilomar ConferenceNovember 7, 2011

22

Context

Backbone of IP networks

Traffic anomalies: changes in origin-destination (OD) flows

Motivation: Anomalies congestion limits end-user QoS provisioning

Goal: Measuring superimposed OD flows per link, identify anomalies

by leveraging sparsity of anomalies and low-rank of traffic.

Failures, transient congestions, DoS attacks, intrusions, flooding

33

Model Graph G (N, L) with N nodes, L links, and F flows (F >> L)

(as) Single-path per OD flow xf,t

є {0,1}

Anomaly

LxT LxF

Packet counts per link l and time slot t

Matrix model across T time slots

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

f1

f2

l

4

Low rank and sparsity

X: traffic matrix is low-rank [Lakhina et al‘04]

A: anomaly matrix is sparse across both time and flows

0 100 200 300 400 5000

1

2

3

4x 10

7

Time index (t)

|xf,

t|

0 200 400 600 800 10000

2

4x 10

8

Time index(t)

|af,

t|

0 50 1000

2

4x 10

8

Flow index(f)

|af,

t|

55

Objective and criterion

(P1)

Given and routing matrix , identify sparse when is low rank

R fat but XR still low rank

Low-rank sparse vector of SVs nuclear norm || ||* and l1 norm

66

Distributed approach

Goal: Given (Yn, Rn) per node n є N and single-hop exchanges, find

Y=n

Nonconvex; distributed solution reduces complexity: LT+FT ρ(L+T)+FT

Centralized

(P2)

XR=LQ’Lxρ

M. Mardani, G. Mateos, and G. B. Giannakis, ``In-network sparsity-regularized rank minimization: Algorithms and applications," IEEE Trans. Signal Proc., 2012 (submitted).

≥r

77

Separable regularization Key result [Recht et al’11]

New formulation equivalent to (P2)

(P3)

Proposition 1. If stationary pt. of (P3) and ,

then is a global optimum of (P1).

88

Distributed algorithm

Network connectivity implies (P3) (P4)

(P4)

Consensus with neighboring nodes

Alternating direction method of multipliers (AD-MoM) solver

Primal variables per node n :

n Message passing:

99

Distributed iterationsDual variable updates

Primal variable updates

1010

Attractive features Highly parallelizable with simple recursions

Low overhead for message exchanges Qn[k+1] is T x ρ and An[k+1] is sparse

FxF

Recap(P1) (P2) (P3) (P4)

CentralizedConvex

LQ’ fact.Nonconvex

Sep. regul.Nonconvex

ConsensusNonconvex

Stationary (P4) Stationary (P3) Global (P1)

Sτ(x)

τ

1111

Optimality

Proposition 2. If converges to ,

and , then:

i)

ii)

where is the global optimum of (P1).

AD-MoM can converge even for non-convex problems

Simple distributed algorithm identifying optimally network anomalies

Consistent network anomalies per node across flows and time

1212

Synthetic data Random network topology

N=20, L=108, F=360, T=760 Minimum hop-count routing

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

False alarm probability

Det

ecti

on p

roba

bili

ty

PCA-based method, r=5PCA-based method, r=7PCA-based method, r=9Proposed method, per time and flow

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

Pf=10-4

Pd = 0.97

---- True---- Estimated

1313

Real data Abilene network data

Dec. 8-28, 2008 N=11, L=41, F=121, T=504

0100

200300

400500

0

50

100

0

1

2

3

4

5

6

Time

Pf = 0.03Pd = 0.92Qe = 27%

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

False alarm probability

Det

ecti

on p

roba

bili

ty

r=1, PCA-based methodr=2, PCA-based methodr=4, PCA-based methodProposed, per time and flow

---- True---- Estimated

1414

Concluding summary

Anomalies challenge QoS provisioning

Identify when and where anomalies occur

Unveiling anomalies via convex optimization

Distributed algorithm

Missing data

Ongoing research

Online implementation

Thank You!

Leveraging sparsity and low rank