SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4...
Transcript of SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4...
![Page 1: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/1.jpg)
Copyright©2016Splunk Inc.
Splunk GoneWild!– InnovatingALargeSplunk SolutionAtTheSpeedOfManagement
![Page 2: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/2.jpg)
Disclaimer
2
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
![Page 3: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/3.jpg)
Introductions
KevinDalianTeamLeadServerHostingTools
FordMotorCompanyNothingInterestingAbout Kevin,he’saboringwork-aholic.
GlenUpretiDirectorEnterpriseandCloudTechnologies
Sierra-CedarTerribleatJenga
![Page 4: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/4.jpg)
Agenda
4
WhereWeCameFromWhereWePlannedtoGoWhereWeEndedUpInstallationOnBoardingDataWhatWe’reuptoNowQ&A
![Page 5: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/5.jpg)
WhereWe Started2SplunkEnvironments– NetworkandServerOperationsServerOps– 4StandaloneSearchHead/Indexers– 3DeploymentServers– 20Gblicense– +11,500UniversalForwarders
![Page 6: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/6.jpg)
WhereWePlannedtoGo
6
MajornewFordInitiative– FordPass/Connected-X– PCF– PivotalCloudFoundry– MicrosoftAzureCloud– Mixtureofinternalandexternalapplicationsanddata– 100Gb/day
![Page 7: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/7.jpg)
WhereWePlannedtoGo(cont’d)
PCF
Azure
PublicInternet
PeerIndexer1
SearchHead1
DeploymentServer/DMC/LicenseMaster
SearchHead1 SearchHead3
PeerIndexer2 PeerIndexer MasterIndexer
SearchHeadMaster
ServerInfrastructure
SplunkForwarders
Syslogfirehose SyslogRelay
SearchHead/Indexer
JMXRESTSQL
SyslogReceiver(w/SplunkUF)
SyslogRelay
Syslog
Syslog
DMZ Intranet
UniversalForwarder
![Page 8: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/8.jpg)
WhereWeEndedUp
8
MobileSearchHead2
Syslog
PCF-Prod
AzureNA2
SyslogReceiver(w/SplunkUF)
DMZ
Intranet
PublicInternet
PeerIndexer1
SearchHead1
DeploymentServer/BatchProcessor/
Archive/DMC/LicenseMaster
SearchHead2 SearchHead3
PeerIndexer2 PeerIndexer MasterIndexer
SearchHeadMaster
ServerInfrastructure
SplunkForwarders
FMCHeavyForwarder
Syslogfirehose
JMXRESTSQL
UniversalForwarder
SSL
Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
PCF-Prod
AzureNA1SSL
Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
TCP
SyslogRelay
SyslogRelay
SyslogRelay
Syslog
Syslog
UniversalForwarder
TCP
SSL
SSL
SearchHead/Indexer
SearchHead/IndexerSyslogfirehose
JMXRESTSQL
Syslogfirehose
JMXRESTSQL
PCFFMC-Prod
JMXRESTSQL
PCFFMC-
PreprodECCHeavyForwarder
Syslogfirehose
JMXRESTSQL
PCFECC-
Preprod
SyslogfirehoseJMXRESTSQL
PCF-Dev
PCF-Dev
MobileSearchHead1
PCF-Prod
21VCN1
Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
SyslogRelay
SearchHead/IndexerSyslogfirehose
JMXRESTSQLPCF-Dev
SSL
SSL
syslog
Http
Android
FMCSearchHead/Indexer
ECCSearchHead/Indexer
PCF-Prod
21VCN2
Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
SyslogRelay
SearchHead/IndexerSyslogfirehose
JMXRESTSQLPCF-Dev
syslog
SSL
SSL
Https
Https
Https
Https
Apple
Syslogfirehose
PCFECC-Prod
PCF-Prod
AzureEU1Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
PCF-Prod
AzureEU2
Syslogfirehose
HeavyForwarders
JMXRESTSQL
JMXRESTSQL
SyslogRelay
SyslogRelay
SearchHead/Indexer
SearchHead/IndexerSyslogfirehose
JMXRESTSQL
Syslogfirehose
JMXRESTSQL
PCF-Dev
PCF-Dev
SSL
SSL
SSL
SSL
syslog
syslog
![Page 9: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/9.jpg)
InstallingGlenandKevinmeetandplanforinstallationtasksHaveaPOCenvironmentinAzureAzuretoon-premise,HOW???Startedwithtemporarystandaloneinstance
![Page 10: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/10.jpg)
Installing(Surprises)Hardwareshowsup&everythingfallsapart– Hardwarearrivedpiecemeal– NotenoughCPUs– AzureVMs,wewerethefirsttemplateinstall– Serversin‘Public’DMZweren’tpubliclyaccessible
EvenwithissuesSHClusterandIDXClusterallinstalledwithindays!
![Page 11: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/11.jpg)
OnBoarding DataOnboardeddatafrom– Pivotal CloudFoundry– MicrosoftAzurePAASviaDBConnect– Thirdpartyandcustomdevelopedinputs
![Page 12: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/12.jpg)
Onboarding DataWhen onboardingalwaysset– TIME_PREFIX– TIME_FORMAT– MAX_TIMESTAMP_LOOKAHEAD– SHOULD_LINEMERGE– LINE_BREAKER– TRUNCATE
![Page 13: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/13.jpg)
Onboarding Data(Surprises)‘Ohbytheway…’– Newinputs– Newregions– Newenvironments(pre-production)– Newteams– NewSplunkLicense– SensitiveData- Needforobfuscation
![Page 14: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/14.jpg)
GotData,NowWhat?PrototypedDashboardwrapupquick.WhatDoesThisMean?– Engagedevelopers andusercommunities
KeepCreating– Alwaysbemovingforward
Alerts– Alertingisaniterativeprocess– Bepreparedforalotofnoiseatfirst– Refine,refine,refine
![Page 15: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/15.jpg)
GotData,NowWhat?(Surprises)Surprises:– MassiveDashboards– NewUsersandRoles– DataSecurity– RetentionTimes
Ohyeah,andmobile…
![Page 16: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/16.jpg)
MobileMadnessSplunkAdd-onforMobileAccess– CrazyEasy!InitialPOCinAzureworkedlikeachampPlanned,preparedandmovedtoDMZNotificationsdon’twork– Newmanagementsurprise…thekindyoudon’twantBacktothedrawingboard
![Page 17: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/17.jpg)
MobileMadness(TempSolution)
PCF
Azure
PublicInternet
PeerIndexer1
SearchHead1
DeploymentServer/DMC/LicenseMaster
SearchHead1 SearchHead3
PeerIndexer2 PeerIndexer MasterIndexer
SearchHeadMaster
Syslogfirehose SyslogRelay
SearchHead/Indexer
JMXRESTSQL
DMZ Intranet
UniversalForwarder
Http
AndroidApple
![Page 18: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/18.jpg)
MobileMadness(Eventually)
![Page 19: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/19.jpg)
WhereAreWeNow?PlanningfortheFuture/ScalingRefininganddocumentingMigratingdata/appsfromoriginalenvironmentExpandingthecustomerbaseStillrefiningdashboardsRe-sourcetypingPreparingformoremanagementshenanigans
![Page 20: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/20.jpg)
AdviceMovingForward1. Insistnon-productionenvironment2. Workwiththecustomertofurtherunderstandingofdata3. Define/DocumentallCustomerrequirementsandgetsignoff4. Avoidthedatagraveyard5. Splunkisveryflexible,keepanopenmindandstaycalm!
![Page 21: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/21.jpg)
AndRemember…
21
“Fallseventimesandstandupeight.”
- JapaneseProverb
![Page 22: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/22.jpg)
Q&A
Questions?
![Page 23: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd1fcee71ca376f073fba7b/html5/thumbnails/23.jpg)
THANKYOU