Simple math for anomaly detection toufic boubez - metafor software - monitorama pdx 2014-05-05

49
1

description

This is my presentation at Monitorama PDX in Portland on May 5, 2014 Simple math to get some signal out of your noisy sea of data You’ve instrumented your system and application to the hilt. You can now “measure all the things”. Your team has set up thousands of metrics collecting millions of data points a day. Now what? Most IT ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this mountain of data and extracting signal from the noise is not easy. The choice of what analytic method to use ranges from simple statistical analysis to sophisticated machine learning techniques. And one algorithm doesn’t fit all data.

Transcript of Simple math for anomaly detection toufic boubez - metafor software - monitorama pdx 2014-05-05

Page 1: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

1

Page 2: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

Some Simple Math for Anomaly Detection

#Monitorama PDX2014.05.05

Toufic Boubez, Ph.D.Co-Founder, CTOMetafor [email protected]@tboubez

Page 3: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

3

Preamble

• I lied!– There are no “simple” tricks– If it’s too good to be true, it probably is

• I usually beat up on parametric, Gaussian, supervised techniques– This talk is to show some alternatives– Only enough time to cover a couple of relatively simple but very useful techniques– Oh, and I will still beat up on the usual suspects

• Adrian and James are right! Listen to them! – What’s the point of collecting all that data if you can’t get useful information out of

it!?

• Note: real data• Note: no y-axis labels on charts – on purpose!!• Note to self: remember to SLOW DOWN!• Note to self: mention the cats!! Everybody loves cats!!

Page 4: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

4

• Co-Founder/CTO Metafor Software• Co-Founder/CTO Layer 7 Technologies

– Acquired by Computer Associates in 2013– I escaped

• Co-Founder/CTO Saffron Technology• IBM Chief Architect for SOA• Co-Author, Co-Editor: WS-Trust, WS-

SecureConversation, WS-Federation, WS-Policy• Building large scale software systems for >20 years (I’m

older than I look, I know!)

Toufic intro – who I am

Page 5: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

5

Wall of Charts™

Page 6: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

6

The WoC side-effects: alert fatigue

“Alert fatigue is the single biggest problem we have right now … We need to be more intelligent about our alerts or we’ll all go insane.”

- John Vincent (@lusis)

(#monitoringsucks)

Page 7: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

7

Watching screens cannot scale + it’s useless

Page 8: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

8

Gotta turn things over to the machines

Page 9: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

9

TO THE RESCUE: Anomaly Detection!!

• Anomaly detection (also known as outlier detection) is the search for items or events which do not conform to an expected pattern. [Chandola, V.; Banerjee, A.; Kumar, V. (2009). "Anomaly detection: A survey". ACM Computing Surveys 41 (3): 1]

• For devops: Need to know when one or more of our metrics is going wonky

Page 10: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

10

Attempt #1: thresholds …

• Roots in manufacturing process QC

Page 11: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

11

… are based on Gaussian distributions

• Make assumptions about probability distributions and process behaviour– Data is normally distributed with a useful and

usable mean and standard deviation– Data is probabilistically “stationary”

Page 12: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

12

Three-Sigma Rule

• Three-sigma rule– ~68% of the values lie within 1 std deviation of the mean– ~95% of the values lie within 2 std deviations– 99.73% of the values lie within 3 std deviations: anything

else is an outlier

Page 13: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

13

Aaahhhh

• The mysterious red lines explained

Page 14: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

14

Stationary Gaussian distributions are powerful

• Because far far in the future, in a galaxy far far away:– I can make the same predictions because the

statistical properties of the data haven’t changed– I can easily compare different metrics since they

have similar statistical properties• Let’s do this!!• BUT…• Cue in DRAMATIC MUSIC

Page 15: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

15

Then THIS happens

Page 16: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

16

3-sigma rule alerts

Page 17: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

17

Or worse, THIS happens!

Page 18: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

18

3-sigma rule alerts

Page 19: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

19

WTF!? So what gives!?

• Remember this?

Page 20: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

20

Histogram – probability distribution

Page 21: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

21

Histogram – probability distribution

Page 22: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

22

Attempts #2, #3, etc: mo’ better thresholds

• Static thresholds ineffective on dynamic data– Thresholds use the mean as predictor and alert if

data falls more than 3 sigma outside the mean• Need “moving” or “adaptive” thresholds:

– Value of mean changes with time to accommodate new data values/trends

Page 23: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

23

Moving Averages “big idea”

• At any point in time in a well-behaved time series, your next value should not significantly deviate from the general trend of your data

• Mean as a predictor is too static, relies on too much past data (ALL of the data!)

• Instead of overall mean use a finite window of past values, predict most likely next value

• Alert if actual value “significantly” (3 sigmas?) deviates from predicted value

Page 24: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

24

Moving Averages typical method

• Generate a “smoothed” version of the time series– Average over a sliding (moving) window

• Compute the squared error between raw series and its smoothed version

• Compute a new effective standard deviation by smoothing the squared error

• Generate a moving threshold:– Outliers are 3-sigma outside the new, smoothed data!

• Ta-da!

Page 25: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

25

Simple and Weighted Moving Averages

• Simple Moving Average– Average of last N values in your time series

• S[t] <- sum(X[t-(N-1):t])/N– Each value in the window contributes equally to

prediction– …INCLUDING spikes and outliers

• Weigthed Moving Average– Similar to SMA but assigns linearly (arithmetically)

decreasing weights to every value in the window– Older values contribute less to the prediction

Page 26: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

26

Exponential Smoothing techniques

• Exponential Smoothing– Similar to weighted average, but with weights decay exponentially over

the whole set of historic samples• S[t]=αX[t-1] + (1-α)S[t-1]

– Does not deal with trends in data• DES

– In addition to data smoothing factor (α), introduces a trend smoothing factor (β)

– Better at dealing with trending– Does not deal with seasonality in data

• TES, Holt-Winters– Introduces additional seasonality factor– … and so on

Page 27: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

27

Let’s look at an example

Page 28: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

28

Holt-Winters predictions

Page 29: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

29

A harder example

Page 30: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

30

Exponential smoothing predictions

Page 31: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

31

Hmmmm, so are we doomed?

• No!• ALL smoothing predictive methods work best

with normally distributed data!• But there are lots of other non-Gaussian

based techniques– We can only scratch the surface in this talk

Page 32: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

32

Trick #1: Histogram!

Page 33: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

33

THIS is normal

Page 34: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

34

This isn’t

Page 35: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

35

Neither is this

Page 36: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

36

Trick #2: Kolmogorov-Smirnov test

• Non-parametric test– Compare two probability

distributions– Makes no assumptions (e.g.

Gaussian) about the distributions of the samples

– Measures maximum distance between cumulative distributions

– Can be used to compare periodic/seasonal metric periods (e.g. day-to-day or week-to-week)

http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Page 37: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

37

KS with windowing

Page 38: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

38

Page 39: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

39

Page 40: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

40

Page 41: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

41

Page 42: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

42

Page 43: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

43

KS Test on difficult data

Page 44: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

44

Trick #3: Diffing/Derivatives

• Often, even when the data itself is not stationary, its derivatives tends to be!

• Most frequently, first difference is sufficient:dS(t) <- S(t+1) – S(t)

• Can then perform some analytics on first difference

Page 45: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

45

CPU time series

Page 46: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

46

Its first difference – possible random walk?

Page 47: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

47

We’re not doomed, but: Know your data!!

• You need to understand the statistical properties of your data, and where it comes from, in order to determine what kind of analytics to use.– Your data is very important!– You spend time collecting it so spend time analyzing it!

• A large amount of data center data is non-Gaussian– Guassian statistics won’t work– Use appropriate techniques

Page 48: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

48

More?

• Only scratched the surface• I want to talk more about algorithms, analytics,

current issues, etc, in more depth, but time’s up!!– Come talk to me or email me if interested.

• Thank you!

[email protected]@tboubez

Page 49: Simple math for anomaly detection   toufic boubez - metafor software - monitorama pdx 2014-05-05

49

Oh yeah, and we’re hiring!

In Vancouver, BC