Dehydrated Treats Raw Meaty Bones Frozen Raw Diets Holistic ...
How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of...
Transcript of How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of...
![Page 1: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/1.jpg)
Copyright©2016Splunk Inc.
DavidVeuveStaffSecurityStrategist,Splunk
HowtoScale:From_rawtotstats (andbeyond!)
![Page 2: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/2.jpg)
Disclaimer
2
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
![Page 3: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/3.jpg)
HowtoUseThisPresentation
ThisPDFisintendedtobeareferenceguide,tocomplementtheactualpresentation.Ifyou’vealreadydabbledintstats,feelfreetoreadthrough.Ifyou’renewtotstats,Ihighlyrecommendwatchingthevideopresentationfirst.Idon’tdoquiteagoodenoughjobwiththisslideverison forittostandalone.Pleasefindthevideorecordingonthe.conf website,maybemid-to-lateOctober(askyourSplunkteamforupdatesifyoudon’tseeitbythen)
3
![Page 4: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/4.jpg)
Agenda1. Intro2. David’sStory3. OverviewofTechniques(SI,RA,AP,tstats)4. DataModels– Whatyouneedtoknow5. Howtotransitionfrom_rawtotstats6. WhenDataModelAccelerationdoesn’twork7. RealWorldExamples8. AdvancedTopics
![Page 5: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/5.jpg)
PersonalIntroduction
5
• DavidVeuve – StaffSecurityStrategist,SecurityProductAdoption• SMEforArchitecture,Security,Analytics• [email protected]• FormerSplunkCustomer(For3years,3.xthrough4.3)• PrimaryauthorofSearchActivityapp• FormerTalks:
– SecurityNinjutsu PartThree:.conf 2016(Thisyear!)– SecurityNinjutsu PartTwo:.conf 2015– SecurityNinjutsu PartOne:.conf 2014– PasswordsareforChumps:.conf 2014
![Page 6: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/6.jpg)
Whyallthis?• Gettingresultsfastisgreat,butonlyhalfthepuzzle.• Ifyou/yourteamarewritingsearchesthatwillrunfor100cpu hoursperday,supposethat’s50%ofyourcluster’stime.
• Whatifwecouldshrinkthatto10CPUhours?Yourclusterjustwentfrom75%utilizedto30%utilized.
• SearchaccelerationlowersyourTCO• Searchaccelerationsavesyoutimewaiting• Searchaccelerationletsyouaskallofthequestions
![Page 7: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/7.jpg)
Whythistalk?Whynow?• tstats isn’tthathard,butwedon’thaveverymuchtohelppeoplemakethetransition.
• EverythingthatSplunkInc doesispoweredbytstats.• I’vetaughtalotofpeopleinsmallergroupsaboutSearchAccelerationtechnologies.
• Tothemasses!
![Page 8: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/8.jpg)
Whoareyou?• Youareeithera*super*hardcoredev,oryou’renotbrandnewtoSplunk.
• You’veplayedwithSPL.Youunderstandhowitworks.• You’reprobablycomfortablewithstats.• Peopleprobablycomeaskyouforhelpbuildingqueriesorsolvingproblems.
![Page 9: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/9.jpg)
Whatwillyouget?• You’llunderstandhowtomakequeriesthatwowpeople.• You’llcementyourselfas*the*officeoruser-groupsearchninja.• You’llhappilylearnhoweasyitis.
![Page 10: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/10.jpg)
David’sStoryJustaboy,standinginfrontofasearchcommand,askingittoshowthesyntaxerror.
![Page 11: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/11.jpg)
WhereIStarted• Customeratanadvertisingcompany• Wasacasualuser,whenIwashandedaBusinessAnalyticsproject• Goingfromtensorhundredsofdatapointstomillions• Builttieredsummaryindexes• Auto-switchedbetweenhighgranularityandlowbasedonselectedtimewindows
• TonsofhelpfromNickMealy@Sideview
![Page 12: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/12.jpg)
ThenItookabreak• ItooktwoyearsoffofSplunk,missing5.xandtheinitial6.0release.
• SplunkreleasedReportAcceleration• SplunkreleasedDataModelAcceleration
![Page 13: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/13.jpg)
ICametoSplunk• Irebuiltmydashboard.FromSplunk4toSplunk6,loadtimewentfrom1.5minto27seconds.
• IusedReportAcceleration– loadtimewentdownto6seconds
• ButthenIhadabunchofdifferentsearchesrunning..
![Page 14: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/14.jpg)
IHelpedaFinanceCompany• Theywantedmultipledashboards,drilldown,searches,on18keyfieldsin2000lineXMLdocuments.
• Builtanaccelerateddatamodelwith18calculatedspath fields• Usedthepivotinterfacetobuilddashboards• 30dayunaccelerated loadtimewouldhavebeen2days ifIcouldwait• 30dayacceleratedloadtimewas15seconds
![Page 15: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/15.jpg)
IHelpedaHealthCareCompany• Theywanteddistinctcountofdest_ip persrc_ip perday,averagedandstdev’d.
• Runningoverrawwasn’tevenconsidered.• Dependingontheanalysis,wecansearchandprocessover1billionresults/minute.
![Page 16: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/16.jpg)
TechniquesIt’sallaboutthetechnique..
![Page 17: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/17.jpg)
SummaryIndexing• Takethesearchyou’rerunningrightnow,andstoretheresultsinanewindex.Nolicenserequired.
• How:– Justadd|collectinyoursearch,specifyingdestinationindex(maybe
”summary”)– Probablydon’twanttousesistats,sitop,si..anything.They’renotreally
valuable.– http://www.davidveuve.com/tech/how-i-do-summary-indexing-in-splunk/
• Examples:– Store#oflogins,#ofdistincthosts,#of…peruser/device/etc– Emaillogsarehorribleandslowtoprocess– storetheoutput– ITSIMetricsearches
![Page 18: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/18.jpg)
SummaryIndexing(2)• Why:You’renotacceleratingrawevents,you’reacceleratingtheresultofasearch.Wecan’taccelerateasearchbaseddatamodel.So:summaryindexing
• Whynot?– NoMultipleLevelsofTimeGranularity--------->– Manualcoordinationofsummaryindexing---->– Missedsearches-------------------------------------->
![Page 19: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/19.jpg)
ReportAcceleration• Takesasinglesavedsearch,withstats/timechart/top/chartandpre-computestheaggregatesatmultipletimebuckets(per10m,perhour,perday,etc.,basedonyouraccelerationrange).
• Automaticallyswitchesbetweenaccelerationandrawdataaccesswhenneeded.
• Youcannotquerythedatainwaysthatyoudidn’tplanfororiginally
![Page 20: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/20.jpg)
ReportAcceleration(2)• How:
– GointothesavedsearchconfigurationandchecktheAcceleratebox
– Decideonoverwhattimerangeyou’dliketoaccelerate– Keepinmindthatlongertimeranges=>lessgranularity(so
ifyouchoose1year,you’lllose10minor1hr buckets
• Example– Myexecdashboardneedstoload,like,immediately.
![Page 21: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/21.jpg)
NormalSearchExample• Youaskforastatisticalsearch
• Indexersreturnminimumnecessarystatistics(e.g.,anavg needssum/count)
• SHcomputesfinalresult(sum(sum)/sum(count))
![Page 22: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/22.jpg)
ReportAccelerationExample• SHregularlyrequestsminimumnecessarystatistics(e.g.,avg needssum/count)splitintotimebuckets
• Later,whenuserrequestsvalues,theSHalreadyknowstheanswer.
![Page 23: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/23.jpg)
ReportAcceleration(3)• Why?
– You’vegotasmallmodestdatasetwithlowsplit-bycardinalitywhereyouarewillingtobecraftytorunmultiplequeries
– Autofallbacktorawlogs,autobackfillandrecovery,autotimegranularity– SUPERFAST– Easy
• WhyNot?– Mostlylimitedtoasinglesearchperjob------->– Onlysupportforbasicanalytics------------------>– Kinda ablackart,notthatwidelyused--------->
![Page 24: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/24.jpg)
AcceleratedPivot• Draganddropbasicstatsinterface,withtheoverwhelmingpoweroveraccelerateddatamodelsonthebackend
• How:– Buildadatamodel(moreonthatlater)– Accelerateit– Usethepivotinterface– Savetodashboardandgetpromoted
• Examples– Yourfirstforayintoacceleratedreporting– Anythingthatinvolvesstats
![Page 25: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/25.jpg)
AcceleratedPivot(2)• Why?
– Supereasy– Automaticallyswitchbetweenrawlogsandaccelerated
data– DataModelAcceleration=💯
• WhyNot?– Notentirelyacceleratedbydefault---------------->– Can’tgosummariesonly inUI----------------------->– Pivotsearchlanguageisweirderthantstats ---->
![Page 26: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/26.jpg)
tstats• Operatesonaccelerateddatamodelsortscollect files(andindex-timefieldextractions,suchassource,host,index,sourcetype,andthoseITSIoroccasionalothers)
• Canonlydostats– norawlogs(today!)• Isfasterthanyou’veeverimaginedlifetobe.• How:
– Differentsearchsyntax,whichtakesadjustment,butactuallyreallysimilartonormalstats.
– |tstats countwhereindex=*groupby indexsourcetypeê Bringafour-pointseatharness‘cause we’regoingFAST
![Page 27: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/27.jpg)
tstats (2)• Why?
– Distributedindexedfieldsearchingwiththeflexibilityofsearchlanguagetodefinesyntax
– summaries_only=t– Fasterthanyou’veeverbeen.
![Page 28: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/28.jpg)
DataModels– WhatyouneedtoknowSomethingcleverhere..
![Page 29: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/29.jpg)
DataModelBasics• Essentiallyanythingyoucandefineinpropsandtransformscangointoanaccelerateddatamodel
• Onlyrawevents– can’taccelerateadatamodelbasedonsearches,orwithtransaction,oretc.– Gocheckoutsummaryindexing
• Favoriteexample:|eval myfield=spath(_raw,“path.to.my.field”)isslow.Putthatinyourdatamodel,andpivot/tstats querieswillbesuperfast
• NextfiveslidesfromDavidMarquardt’s.conf2013Presohttp://conf.splunk.com/session/2013/WN69801_WhatsNew_Splunk_DavidMarquardt_UnderstandingSplunkAccelerationTechnologies.pdf
![Page 30: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/30.jpg)
30
Splunk EnterpriseIndexStructure
IDX1IDX2
IDX3
ColdPath
ThawedPath
Rawdata
TSIDXhot_v1_100
hot_v1_101
db_lt_et_80
db_lt_et_101
*.data*.tsidxrawdata
db_lt_et_70
apple
beer
LEXICON
POSTING
“applepieandicecreamisdelicious”
“anappleadaykeepsdoctoraway”
150100
etet
ltlt
itit
apple beer cokeice java …
HomePath
Source/Sourcetype/HostMetadata1source::/my/log2source::/blah
cream
![Page 31: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/31.jpg)
31
RawdatastoredatoffsetsPostingvalue Seekaddress _time
0 42 1331667091
1 78 1331667091
2 120 1331667091
3 146 1331667091
4 170 1331667091
5 212 1331667091
6 240 1331667091
Rawevents
DeeplikesBudlight
Amrit likesMakers
Ledion likescognac
DavelikesJackDaniels
Zhanglikesvodka
DeeplikesMakers
DavelikesMakers
![Page 32: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/32.jpg)
32
RawDataGetsIndexed Term PostingsListAmrit 1Bud 0Daniels 3Dave 3,6Deep 0,5Jack 3Ledion 2Makers 1,5,6Zhang 4cognac 2likes 0,1,2,3,4,5,6light 0vodka 4
Rawevents
DeeplikesBudlight
Amrit likesMakers
Ledion likescognac
DavelikesJackDaniels
Zhanglikesvodka
DeeplikesMakers
DavelikesMakers
• Eachwordintheraweventisindexed• TheTSIDXwillstoretheoffset#,andlocationinthegzip’d journal
• Queryingdave makersreturns#6
![Page 33: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/33.jpg)
ReadingCompressedRawdatajournal.gz
078148236380434506
33
Example:Readingoffsets(120,170)1. Groupoffsetsintoresidingchunks
120fallsintorange(78,148)170fallsintorange(148,236)
2.Readdataoffdiskanddecompress3.Runthroughfieldextractions4.Recheckfilters5.Runcalculations
Thisisdisk+CPUEXPENSIVE
![Page 34: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/34.jpg)
34
StoringIndexedFieldsinTSIDXTerm PostingsListbar::AB 1,3,7,39,98bar::cez 0,6,9,12bar::xyz 3,4,5,6baz::1 3,6,85baz::2567 0,5baz::462 3,24,45baz::98 2,3,5,8,9baz::99023 1,5,6,76,99foo::afdjsi 4,567,2345foo::aghdafo 2,234,6667foo::bazcxuid 0,1,623,7777foo::cef 0,1,2,3,4,43foo::zaz 4
BigIdea:Usethelexiconasafieldvaluestore!
Bysimplyseparatingfieldsandvalueswith“::”wecanstoresufficientinformationtorunmoreinterestingqueries.
DataModelqueriesdon’t evervisitrawlogs.TheyliveentirelywithinTSIDX!
![Page 35: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/35.jpg)
HowtoTransitionfrom_rawtotstatsAwholenewworld(don’tyoudarecloseyoureyes)
![Page 36: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/36.jpg)
ProcessOverview• Buildyourdatamodelwithwhateverfieldsyoucouldcareabout• Startwithyourrawsearch• Identifytheaggregationthatyouwanttodo
– Statsavg(bytes),dc(host),whateverelse
• Maketheminorsyntaxadjustmentsfortstats
![Page 37: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/37.jpg)
ExampleWithoutDataModelsRawindex=*|statscountbyindex,sourcetype
Tstats|tstats countwhereindex=*groupby index,sourcetype
![Page 38: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/38.jpg)
ExampleWithDataModelsRawtag=networktag=traffic|statsdc(dest_ip)bysrc_ip
Tstats|tstats dc(All_Traffic.dest_ip)fromdatamodel=Network_Trafficgroupby All_Traffic.src_ip
![Page 39: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/39.jpg)
Challenge:IdentifyingFields• Whatfieldsareactuallyinadatamodel?• HowdidIknowtouse“All_Traffic.dest_ip”insteadof“dest_ip”orinsteadof“Network_Traffic.dest_ip?
• Tofigureitout,wecanlookatthedatamodeldefinitionviapivot,orattheresultingtsidx filesviawalklex
• Pivotdoesn’trequireSSHaccess,butstillleavesyouguessingforparts• walklex ismuchmoreaccurateandpreferable
![Page 40: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/40.jpg)
IdentifyingFieldsviaPivot
ThedatamodelisNetwork_Traffic,andtherooteventnodeis
“All_Traffic”sofieldsshouldmostlybeAll_Traffic.fieldname
![Page 41: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/41.jpg)
IdentifyingFieldsviaWalklex• FindtheTSIDXFileonyourindexer(let’sassumeadatamodel)
– Pathsetinyourindexconfig,butbydefaultintheindexfolder– Usually$SPLUNK_HOME/var/lib/splunk/<INDEX>/datamodel_summary/<BUCKET_ID>
/<SEARCH_HEAD_GUID>/<DATAMODEL_NAME>/<TIMERANGE>.tsidx– Goodnews:That’sbyfarthehardpart– Example:/opt/splunk/var/lib/splunk/defaultdb/datamodel_summary/1772_813B72E7-6743-
4F46-9DE6-536F78929EDD/813B72E7-6743-4F46-9DE6-536F78929EDD/DM_Splunk_SA_CIM_Network_Traffic/1466344886-1466326949-3864670955536478127.tsidx
• Runwalklex,eitherwithanemptystring“”orawildcard“*dest_ip*”– $SPLUNK_HOME/bin/splunk cmd walklex <TSIDXFILE>“”
![Page 42: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/42.jpg)
ExampleWalklex
![Page 43: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/43.jpg)
ExampleWalklex foraParticularField
![Page 44: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/44.jpg)
ExampleDistinctCountofWalklex Fields• /opt/splunk/bin/splunk cmd walklex 1457540473-1457196480-3287925045170504614.tsidx""|tr
-s""|cut-d""-f3|grep"::"|awk -F"::"'{print$1;}'|sort|uniq -c
![Page 45: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/45.jpg)
tstats whereclause• Workssurprisinglyliketheinitialsearchcriteriaofarawsearch• whereindex=*sourcetype=pan_traffic ORsourcetype=pan:traffic
– Justlikenormalsearch
• |tstats countwhereindex=pan10.1.1.1– Withnon-datamodel data,10.1.1.1willbeinthetsidx.
• whereearliest=-24h– Notethatthereisabugin6.3,6.4whereamorerestrictivetimepicker range
doesn’toverridetheearliest=…(unlikeinrawsearch– thisisabug)
![Page 46: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/46.jpg)
tstats groupingby• Whengroupingbyvalues(e.g.,src_ip,sourcetype,etc.)it’slikeanormalstats….by…– |tstats countwhereindex=*groupby source,index
• Youcanalsogroupbytime,withoutusingthebucketcommand– |tstats countwhereearliest=-24hindex=*groupby index_timespan=1h
![Page 47: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/47.jpg)
BugsandSurprises• There’sabugin6.3/6.4withearliestandlatestwheretstats doesn’toverridethetimepicker,soeasiesttoleaveyourtimepickeratalltime.
• Sometimeststats handleswhereclausesinsurprisingways.Forexample: nounderscoresinvalues,nosplunk_server_group,nocidrmatches (All_Traffic.dest_ip!=172.16.1.0/24– Fail.All_Traffic.dest_ip!=172.16.1.*– Success)
![Page 48: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/48.jpg)
WhenDataModelAccelerationortstats Don’tWorka sad,sadday….
![Page 49: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/49.jpg)
Ontheoutputofastatscommand• Sadly,youcan’taccelerateasearch-baseddatamodel,sonoluck.• ThisiswhereSummaryIndexingcomesin• Youcanalsodoindex-timefieldextractionsonsummaryindexesifyou’refancy,andthentstats onthose!
![Page 50: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/50.jpg)
fields.conf:[indexed_itsi_kpi_id]INDEXED=true
Workaround:Stats->SI+IndexTime->tstats• Creatingindextimefieldsisahassle,involvingfields.conf,props.conf,transforms.conf,butitworksonsummaryindexeddata.
• Forexample,fromITSI,weindexthefieldindexed_itsi_kpi_id fromsummaryindexedsearches(sourcetype:stash_new)
props.conf:[stash_new]TRANSFORMS-set_kpisummary_index_fields = set_kpisummary_kpiid
transforms.conf:[set_kpisummary_kpiid]REGEX = itsi_kpi_id\s*=\s*([^\s,]+)WRITE_META = trueFORMAT = indexed_itsi_kpi_id::$1
![Page 51: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/51.jpg)
WhenYourCardinalityisCrazyHigh• Tstats canprocesshugenumbersofevents(billions,trillions,noproblem).
• Butifwehavetostoremillionsofrowsinmemorybasedonyoursplit-by,thatcanberough
• Example:300,000personcompanytracks#ofloginsperuserperdayover100days.300,000*100=30Mrows,whichmeanswritingpartialresultstodiskandsadness.
• Betterapproachistosummaryindexeachday,andthenusetstats toprocessthoseresultseitherviaindex-timesummarizationorDMA
![Page 52: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/52.jpg)
WhenAnyCardinalityiscrazycrazyhigh• tstats efficiencyisfundamentallybasedontheassumptionthataparticularvaluewillbeusedafewtimes.
• Ifyouhavemillionsofevents,eachwith10datapoints,with10pointsofprecisionsuchthatrepeatvaluesareunlikely,yourtsidx filewillbeabsolutelymassive.
![Page 53: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/53.jpg)
RealWorldExamplesWhenthingsstopbeingslow,andstartgettingreal.
![Page 54: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/54.jpg)
Splunk(x)- IndexSearches• ForrunningourSplunkInternalUBAproject,weneededtoknowwhatsourcetypes wereinthesystem.
• _raw:index=*earliest=-24h|bucket_timespan=1h|statscountbysourcetype,_time– Timetocomplete:68,476seconds(19hours)
• tstats:|tstats countwhereindex=*groupby sourcetype _timespan=1h– Timetocomplete:6.19seconds
• SpeedDifference:11,062x (notpercent,eleventhousandtimesfaster)• QueryLengthdifference:18charactersshorter
![Page 55: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/55.jpg)
FinancialCustomerXMLUseCase• WhatTechnology?
– AcceleratedDataModelswithPivot
• Why?– HeavyXMLParsingmeantsearchquerieswereterriblyslow– Pivotwasveryeasytouse
• Result– Veryhighscale,veryhappycustomer
![Page 56: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/56.jpg)
FinancialCustomerXMLUseCase(2)• NoXMLextraction• Raw:8.811seconds
– index=xx-xxxx sourcetype=xxx-xxxsplunk_server=myserver01.myserver.localParticularLogIdentifier host=*ServerType*|timechart countbyhost
• AcceleratedPivot:1.25seconds– |pivotXXXXXXX YYYYYY count(YYYYYY)AS"NumberofEvents"SPLITROW_timeAS_timePERIODautoSPLITCOLhostFILTERhostis
"*ServerType*"SORT100_timeROWSUMMARY0COLSUMMARY0NUMCOLS100SHOWOTHER1
• Tstats SummariesOnly:0.896seconds– |tstats summariesonly=tcountfromdatamodel=XXXXXXXwhere(nodename =YYYYYY)(YYYYYY.host="*ServerType*")groupby _time
• SpeedDifference:9.9xFaster• QueryLength:18charactersshorter
![Page 57: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/57.jpg)
FinancialCustomerXMLUseCase(3)• SingleXMLExtractionviaspath• _raw:299.763seconds
– index=xx-xxxx sourcetype=xxx-xxxsplunk_server=myserver01.myserver.localParticularLogIdentifier |eval RuleId=spath(_raw,"___path____.___to__._____very_______._____long___._:_xml___._:____._:_____.______.__________.__________.______________.______")|timechart countbyRuleId
• AcceleratedPivot:2.4seconds● |pivotXXXXXXXYYYYYYcount(YYYYYY)AS"NumberofEvents"SPLITROW_timeAS_timePERIODautoSPLITCOLRuleId SORT100_time
ROWSUMMARY0COLSUMMARY0NUMCOLS100SHOWOTHER1
• tstatssummariesonly:2.04seconds– |tstats summariesonly=tcountfromdatamodel=XXXXXXXwhere(nodename =YYYYYY)groupby RuleId_time
• SpeedDifference:about146.9xfaster• QueryLength:50charactersshorter
![Page 58: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/58.jpg)
FinancialCustomerXMLUseCase(4)• HeavyXMLExtraction(mentionedearlier).Searchesanonymized...• AnEntireDashboardofUnaccelerated PivotswithlotsofXMLspath
– Timetocomplete:172,800seconds(2days)
• AnEntireDashboardofAcceleratedPivots– Timetocomplete:16seconds
• SpeedDifference:about10000x• TimeTakentoBuild14PanelDashboardviaPivot:15minutes
![Page 59: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/59.jpg)
ESEndpoint+Proxy+AV• WhatTechnology?
– ESDataModels+tstats
• Why?– ESDataModelswerealreadybuilt,andmultipledatasourcessotstats append=t
• Result– Superfastsearch,highscalable.– DataModelsmakethingseasier
• Downside– Inthiscase,a19secondsavingsevery15minutes=a$211ROI/yearona$300k
Splunkinfrastructure…maybenotenough?
![Page 60: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/60.jpg)
ESEndpoint+Proxy+AV• Fromlastyear’sSecurityNinjutsu PartTwo,correlatingsysmon withproxyandAVdata.
• _raw:[searchtag=malwareearliest=-20m@mlatest=-15m@m|tabledest |renamedest assrc ]
earliest=-20m@m(sourcetype=sysmon ORsourcetype=carbon_black eventtype=process_launch)OR(sourcetype=proxycategory=uncategorized)
|statscount(eval(sourcetype="proxy"))asproxy_events count(eval(sourcetype="carbon_black"ORsourcetype="sysmon"))asendpoint_events bysrc
|whereproxy_events >0ANDendpoint_events >0– 21seconds
![Page 61: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/61.jpg)
ESEndpoint+Proxy+AV(2)• tstats:|tstats prestats=tsummariesonly=tcount(Malware_Attacks.src)asmalwarehits fromdatamodel=MalwarewhereMalware_Attacks.action=allowedgroupby Malware_Attacks.src
|tstats prestats=tappend=tsummariesonly=tcount(web.src)aswebhits fromdatamodel=Webwhereweb.http_user_agent="shockwaveflash"groupby web.src
|tstats prestats=tappend=tsummariesonly=tcount(All_Changes.dest)fromdatamodel=Change_Analysis wheresourcetype=carbon_black ORsourcetype=sysmon groupby All_Changes.dest
|renameweb.src assrc Malware_Attacks.src assrc All_Changes.dest assrc
|statscount(Malware_Attacks.src)asmalwarehits count(web.src)aswebhits count(All_Changes.dest)asprocess_launches bysrc
– 2seconds
![Page 62: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/62.jpg)
ESEndpoint+Proxy+AV(3)• SpeedDifference:10.5x
– Itdoesn’talwayshavetobe10,000x.10xoreven3xisstillahugereductioninresources.
• QueryLengthdifference:282characterslonger– Multiplenamespacescanmakethingslonger,andalsomaybemorecomplicated
sometimes.Worthitthough.
![Page 63: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/63.jpg)
AdvancedTopicsBecauseit’sbeenstraightforwardsofar,right?
![Page 64: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/64.jpg)
allow_old_summaries andsummaries_only• Thesetwosettingsareperhapsthemostimportanttotstats.• summaries_only meansthatwewon’tautomaticallyfallbacktorawdata– thismeansfastresults,andmuchmoreofadifferencethanyouwouldprobablyexpect.Ifsearching100daysofdata,and15minutesaren’taccelerated,weprobablydon’tcare.
• allow_old_summaries iskeyfortwoscenarios:– Youleveragethecommoninformationmodel,whichisperiodicallyupdated,
andyouwanttobeabletosearchdatafromanearlierversion(verylikely)– Youhavemultipleappswithdifferentglobalconfigsharingsettings,andyou
wanttosearchfromanappthatdidn’t*generate*thedatamodeloriginally.
![Page 65: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/65.jpg)
allow_old_summaries andsummaries_only (2)• WhilethesesettingsareautomaticallysettotrueinES(andprobablyotherSplunkownedapps),becausetheyaresokeyyoumaywanttosetthemtotrueautomaticallyacrossthesystemvialimits.conf
• Bigimpact:pivotwillusewhateverthedefaultis– Note:thepivotuserinterfaceactuallyruns
tstats.Thepivotsearchcommandisnotimpacted– Iknow,Iknow.
![Page 66: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/66.jpg)
prestats=t• Tstats canbefedintoupstreamstats.Forexample,tstats _timespan=…putdirectlyintoagraphlooksterrible.
• |tstats prestats=tcountwhereindex=*groupby _timespan=1dindex|timechart span=1dcountbyindex
![Page 67: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/67.jpg)
chunk_size• Howmuchdatawillberetrievedbytstats fromatsidx fileatonce• Tradeoffbetweenmemory,sorting,andotherfactors• Defaultvalue(10000000– 10MB)isusuallytherightfit.
– Loweringthatcouldsignificantlyhurtperformance.– Forveryhighcardinality,raisingitto50MBor100MBmaybebeneficial– Worthtestingoutonlyforalong-runningsearchyouwilluseregularly
![Page 68: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/68.jpg)
SearchingAcrossMultipleNamespaces• Withnormalsearch,youcanuseasmanydifferentindexes,sourcetypes,etc asyouwant,withrecklessabandon.
• Withtstats,youcanuseappend=t,butrequiresprestats=t.Frequentlyrequiresmungingwitheval alongtheway.
• |tstats prestats=tdc(All_Traffic.dest)fromdatamodel=Network_Trafficgroupby All_Traffic.src|tstats prestats=tappend=tcountfromdatamodel=MalwaregroupbyMalware_Attacks.dest|eval system=coalesce('All_Traffic.src','Malware_Attacks.dest')|statsdc(All_Traffic.dest),countbysystem
![Page 69: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/69.jpg)
SearchingAcrossMultipleNamespaces(2)• Ifyouarequeryingthesameparametersinthefirstandsecondquery,suchascomparingtimespansorlookingattwocounts,useeval withcoalecese todefineafield
|tstats prestats=tappend=tcountfromdatamodel=Malwarewhereearliest=-24hgroupby Malware_Attacks.dest|eval range="current"|tstats prestats=tappend=tcountfromdatamodel=Malwarewhereearliest=-7dlatest=-24hgroupby Malware_Attacks.dest|eval range=coalesce(range,"past")|chartcountoverMalware_Attacks.dest byrange
![Page 70: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/70.jpg)
SearchingAcrossMultipleNamespaces(3)
70
Youcanalsousedifferentfields,suchascount(Malware_Attacks.src),count(web.src),andetc.• |tstats prestats=tsummariesonly=tcount(Malware_Attacks.src)as
malwarehits fromdatamodel=MalwarewhereMalware_Attacks.action=allowedgroupby Malware_Attacks.src
• |tstats prestats=tappend=tsummariesonly=tcount(web.src)aswebhits fromdatamodel=Webwhereweb.http_user_agent="shockwaveflash"groupby web.src
• |tstats prestats=tappend=tsummariesonly=tcount(All_Changes.dest)fromdatamodel=Change_Analysis wheresourcetype=carbon_black ORsourcetype=sysmon groupbyAll_Changes.dest
• |renameweb.src assrc Malware_Attacks.src assrcAll_Changes.dest assrc
• |statscount(Malware_Attacks.src)asmalwarehits count(web.src)aswebhits count(All_Changes.dest)asprocess_launches bysrc
PullMalwareData
PullWeb(Proxy)Data
PullEndpointData
NormalizeFieldNames
DoCount
![Page 71: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/71.jpg)
Drilldown• Drilldownsfromtstats queriesdon’toftenworkcorrectly• Besttoputthatinadashboardwhereyoucanmanuallydefinethedrilldown
![Page 72: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/72.jpg)
_indextime• WhiletheSplunkUIdoesn’tshow_indextime normally,youcanuseitbecauseitisanindexedfield.Just|eval _time=_indextime
• Youcan’tdoaggregationsonit,butyoucanfilter!• Boththetimerangepicker*AND*_indextime apply|tstats countmin(_time)asmin_time max(_time)asmax_time where
[|statscountassearch|eval search="_indextime>".relative_time(now(),"-7d")|tablesearch]
index=*groupby _indextime
|eval lag=_indextime - (min_time +max_time)/2
|eval _time=_indextime
|timechart avg(lag)
![Page 73: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/73.jpg)
ASpecialNoteAboutTime_timeisspecialwithtstats,foracoupleofreasons:• Youcan’tdoavg(_time)orrange(_time)• Youcandomin(_time)andmax(_time)andofcoursegroupby _timespan=10m(orwhatevertime)
![Page 74: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/74.jpg)
Cardinality• Datamodelsarephenomenalwithsplit-bycardinality,e.g.:
– |tstats avg(bytes)fromdatamodel=Network_Traffic groupby All_Traffic.dest_ip
• Datamodelsarelessgreatwithoverwhelmingfieldcardinality,whentrackingmetricdata
• Roundoffirrelevantdatapoints.Ifyouhavetemperatureto7decimalplaces,but1decimalplaceisallthatactuallymatters,justacceleratethat.– Don’tincludetheunroundedfieldinyourdatamodel,becausethenthe
accelerationwillstoreitandyou’llusemorediskspace.
![Page 75: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/75.jpg)
SchemeonWhat?• DataModelsareagreatcombinationofschemaonreadandschemaonwrite.
• AswitheverythinginSplunk,youcanflexiblydefineandchangeyourschema,rebuildtsidx,etc.
• Butforaccelerateddatamodels,yougetalltheperformanceofschemeonwrite…withoutlosingtheflexibilitytoredefineandrebuildasneeded.– Obviously,forVERYlargedatamodels,youmightnotwanttowaitforarebuild,
butyoucanaffectmovingforward
![Page 76: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/76.jpg)
QuirksofDataModelAcceleration• Secondcompression.Youcan’tlookatmillisecondsormicrosecondsfor_timewithouthijinks(separatefieldandseparatefiltering)
• Requiresstats.It’scalledtstats forareason– there’snotstatsraw in6.4.
• |datamodel searchcommandwasthedevil<6.4– muchbetterinnewestrelease
• Interrogatingfieldsisahassle• TSIDXtradesdiskspaceforperformance
![Page 77: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/77.jpg)
SummaryLet’spullitalltogether,team
![Page 78: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/78.jpg)
Summary
78
Gettingstarted:useacceleratedpivotondatamodelsGettingstartedw/tstats:usetstats onnormalindexeddata– countingevents– lookingforindextime lagtstats isactuallyreallyeasyThatsaid,therearesomeweirdquirks.– CheckoutthePDF
![Page 79: How to Scale: From raw to tstats (and beyond!) - SplunkConf · Disclaimer 2 During the course of this presentation, we may make forward looking statements regarding future events](https://reader031.fdocuments.us/reader031/viewer/2022021710/5c49ffa893f3c314364483ab/html5/thumbnails/79.jpg)
THANKYOU