Give Me the Bad News Straight: Why Models are a Broken Approach to Alerting

TechTalk:GiveMetheBadNewsStraight: WhyModelsareaBrokenApproachtoAlerting

DavidB.Martin

DevOps:AgileOps

CATechnologiesAPMProductManagerDO5T41T

#CAWorld

2 ©2015CA.ALLRIGHTSRESERVED.@CAWORLD #CAWORLD

GiveMetheBadNewsStraight:WhyModelsareaBrokenApproachtoAlerting

The industry standard approach to automatic alerts is to create modelsfrom base-lining application latencies. But when something goes wrong,is it because something is really broken or because the model wasincorrect? Training the model to avoid mistakes is complex and time-intensive. CA Application Performance Management (CA APM) 10replaces the whole approach with a brand new one: react to changes inapplication stability as they occur. Outliers are automatically ignored,while tremors in latency register progressively bigger values for theintensity of an event, a little like the richter scale for earthquakes. Jointhe discussion and learn how CA APM transforms automatic alerting.

DavidB.MartinCATechnologiesProductManager


©2015CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2015presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Agenda

WHYMODELSAREFAILING

ABRIEFHISTORYOFAPMALERTING

CATECHNOLOGIESDIFFERENTIALANALYSIS

MODELSAREMADETOBEBROKEN

DATA-DRIVENDIVEINTOAUTOMATICALERTINGMODELS

SHEWHARTSAVESTHEDAY

1

2

3

4

5

6


Keepingmypromise!

§ Iwillbeginthissessionbymakingadetailed,data-centriccaseforwhyCATechnologiesnewdifferentialanalysisfeatureisasuperior,market-leadingapproachtoautomaticalerting.

§ No,Iwillnotthenpullarabbitoutofahat.‘Cuz thisain’tmagicpeople…evenifitlookslikemagic.

§ “Anysufficientlyadvancedtechnologyisindistinguishablefrommagic.”—A.C.Clarke


WhatwasCA’slastanswer?

§ Intheearly90s,WilyimplementedHolt’sLinearExponentialSmooth(HLES)tocalculatebaselines for metrics.

§ Baselineswerefooledbyregularproductionevents—manyweremoreaboutregularpatternsinloadthanaboutmaintenanceevents.Seasonalitydebutstoaddressit.

§ Thisleadstorules—andrulesengines—toaddressedgecasesthatseasonalitydoesnotaddress(e.g.“+3std dev frombaseline”todeadenthesensitivityoftriggers).

Andwhatareourcompetitorsdoing?


What’stheproblemwiththestate-of-the-art?

§ Asthefollowingslideswillexplain,seasonalbaselinesmissproblemsthatyoudon’twanttomiss.

§ Inevitably,theyalsoreporttoooften.

§ Whentheydo,youhavetowriterulesresolvetheissuewithyourissues.

§ Nowyou’vefailedtofindtheautomaticalertinggrail.

§ Itmayactuallybemoreefficienttogobacktowritingstaticthresholdsforyourkeycomponents.

Or,agood reasonforteachingyousomeinterestingmath.

8 ©2015CA.ALLRIGHTSRESERVED.@CAWORLD #CAWORLD440

460

480

500

520

540

560

580

600

620

AverageResponseTime

+1StdDev

+2StdDev

+3StdDev

Thisisastableapplicationresponsetime,withbandsofstandarddeviation.Mostbaselinesarefancyformsofstandarddeviationthattakeintoaccount thingslikeseasonality.


200

400

600

800

1000

1200

1400

1600

1800

Anoutlier…Whattodo?Ifit’sinaseasonalwindow,ithastobeabiggeroutlier,buttheproblemof,“ToAlertorNottoAlert,”remains

thesame.

Youmusteithersendanalertforthissinglespikeorwritearuletosaythatthespikehastobe“sobig”beforeyoucare(whichisusuallydonewithamanuallywrittenrulelike

“>3stddev”).

“Mr.Opswon’tevenputdownhissandwichforasinglefailedtransaction.”


500

1000

1500

2000

2500

Whatabout thesituationofasustainedspike?

Supposedly, seasonalitycancelsout thenormaloperations.Buthowmanyofyouhaveappsinwhichasingleuserlogsinandstartsrunningexpensive(e.g.reporting)transactions?

Traditionalapproachhastoagaindecide:whentoalert?Ifappusersloginatirregularintervalsandperformthistypeoftransaction,thentriggeringalertson theirnormal(non-seasonal)activity?

“catalerts/dev/null”.

Buthowlongdoyouwaitthen?Onceagain,adecisionyou havetomakeand

configureforeachofyourapps.


500

1000

1500

2000

2500

3000

Betterhope thatsustained,normalchangesinresponsetimeareseasonalwhentheyhappen…Ifnot,youmustwriterules!

Andifyouwriterules,youmightaccidentallydeadenthethresholdtoactualproblems.Dang,gum!


OurHero:WalterShewhart

§ Inthe1920s,WalterShewhart etalworkedonqualitycontrolforburiedtelephonelines.

§ Shewhart observedthatwhileeverylinedisplaysvariation,somelinesoccasionallydisplayuncontrolledvariation.Likeaseismometer,therearenormalfluctuationsandthenthereareearthquakes.

§ Shewhart inventedcontrolchartsandtheWesternElectricRulestoidentifyuncontrolledvariance,earninghimselfthetitle:“FatherofStatisticalQualityControl.”


Translationplease!

§ Shewhart taughtustofavorrealtimeobservationovermathematicalmodelsofasignal’sbehavior.

§ Westillbaselinethesignal,buttheWesternElectricRulesdefinethesituationsinwhichthesignalshouldbeconsideredinabadstateandnotasimpledeltafromthebaselinemodel.

§ Shewhart’smethodofcharacterizingthequalityofasignalmirrorsthebehaviorofahumanobserver.

Trustus,youwillunderstand thismath.


Shewhart’s WesternElectricRulesStraightoffWikipedia…

ThecanonicalWesternElectricRulesuseplain,oldstandarddeviationastheirrealtimemeasure.Eachruleidentifiesapatterninthesignal:

Rule#1– Astatisticallyinterestingoutlier

Rule#2– Twosomewhatinterestingoutliersoutofthreemeasurements.

Rule#3– Foursmalleroutliersoutoffivemeasurements.

Rule#4– Manysmalloutliersovermanymeasurements.

Thismuchweflatoutstolefrommathhistory!

SeeCommentstotheright


CATechnologiesInnovation

§ WesternElectricRulesarebrilliantforbothrealtimeanalysisoftelephonesignalsandapplicationsignals.

§ Asinglerulebreach,however,istoodullabladeforslicingthroughthistoughproblem.

§ Byassigningweightstoeachrulebreach,keepingarunningsumandagingoutoldbreaches,wecanproduceasingle,normalizedvalueforvarianceintensity.

CAAPM10hasseveralpatentspending.


Inabusysystem,therearealwaysvaryinglevelsofstability.

Inthispicture,canyou tellwhichsignalsareleaststable?


Thissignalexperiencedanoutlier,butitdidn’tturnblue.

Asinglerulebreachisn’tenough for“Petetoputdownhissandwich.”


Inthiscase,thechangeinstabilitywassustainedoveraboutfortyminutes.

Whathappened? Click tofindout…


Thisapplicationexperiencedaremarkabledegradationinperformanceoveraforty-minuteperiodoftime.

Botholdandournewapproachwouldalerthere,butCA’salertwouldhappenearlyintheeventandtriggertracecollectionautomatically.

Theoldapproachmightnothaveletanoperatorknowforthirtyminutesormore,basedontherulestheyconfigured.


Triageisabattlefieldmedicineterm:wherearethewoundedsoldiers?

CA’sapproachmeansidentifyingchronicproblemsaswellasacuteones.Whichoftheselinesaremorestable,but stillhavingchronicstabilityeventsatregularintervals?


DifferentialAnalysisDefaultConfiguration


CATECHNOLOGIESTEAMPEGASUSClockwisefromleft:

PrashantPathak,MarkLoSacco,WeiniYu,PrasannaRamVenkatachalam,NareshChippada,CareyFeldstein,

PaulCallahanandSai KrishnaRayanapati.[notpictured:me]


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5X189SHowtoAchieveaCustomer-Centric ViewinanOmni-ChannelWorld 11/18/2015 at1:00pm

DO5X194SMonitorMicroservices, Containers, Cloud Foundry andNodewithCAApplication PerformanceManagement 11/18/2015 at4:30pm

DO5X193SCustomizeCAApplicationPerformanceManagementwithTipsforUsingtheCAApplicationPerformanceManagementOpenAPIs

11/19/2015 at4:30pm


MustSeeDemos

ApplicationPerformanceManagementandDevOps,featuringAPMuseinpreproduction scenarios

ApplicationPerformanceManagementTheater5

ApplicationPerformanceManagement,ModernMonitoring, featuringthenewAPMTeamCenter


Ensuringa“5star”mobileappexperiencewithCAMobileAppAnalytics

MobileAppAnalyticsTheater5

UnifiedMonitoring:APMIntegrationsincludingUIM



FollowOnConversationsAt…

SmartBarApplicationPerformanceManagementTheater5

TechTalksApplicationPerformanceManagementTheater5

[email protected]

Give Me the Bad News Straight: Why Models are a Broken Approach to Alerting

Technology

Transcript of Give Me the Bad News Straight: Why Models are a Broken Approach to Alerting