CenturyLink Customer Presentation
-
Upload
splunk -
Category
Technology
-
view
245 -
download
0
Transcript of CenturyLink Customer Presentation
Copyright©2016SplunkInc.
CenturyLinkThomWoldahlLeadEngineer
2
AboutCenturyLink• Foundedin1930• LeadingproviderofInternet,TV,Voice• AglobalcommunicaIons,hosIng,cloudandITservicescompany
• 250,000-route-mileUSfibernetwork• Morethan55globaldatacenters
• HeadquarteredinMonroe,Louisiana• S&P500,Fortune500
3
AboutMe• Leadengineer• BasedinPhoenix,AZcoveringtheMountainWestarea
(Colorado,NewMexico,Arizona)• FirstIPTVengineerinthefieldforCenturyLinkonPRISMproduct• Previouslyworkedinnetworkplanningandengineering
• AsaRegionalOrganizaIon,wesupportourlocalfieldoperaIonsteam• Install,maintenance,repairvideoandhigh-speedInternet
productsaswellasPOTS(plainoldtelephone)
• We’remeasuredoninstallaIonandtroubleshooIngsuccessrates• Keepcostsdown/customershappy
4
SplunkatCenturyLink• SplunkposiIonedtocollectmachinedatafromenIreCenturyLink
network
• Datasources:• SyslogfromIP/MPLSbackbone(routers/switches),servers,
appliances,gateways,videodeliverysubsystems,SNMP,callcentercalllogs
• Databaselogs,applicaIon(transacIonal)logs• AcIve,passiveandtransacIonalmeasurementsthroughoutthe
network:datapulledintoSplunksodwareusingnetworkprobesandscriptedinputintodatabase
• Indexingupto600GBdataperday
• Splunkusers:750regular(2000+unique)
• TeamsusingSplunk:NOC,Development,Architecture,Planning,OperaIons,Engineering,ServiceProvisioning,eCommerceteam,OrderProcessing(IVR),MarkeIng,Product,IT,etc.
• SplunkApps:DeploymentMonitor,WindowsApp,SplunkonSplunk,SplunkDBConnect,ourowncustomizedSplunkApps
5
BeforeSplunk:inaccessibledata,toomanycustomervisits
• MulIplevisitsofon-sitecustomersupport• “chroniccustomers”(3+troublecallsinameasuredperiod)• Iwouldonlybecalledinastechsorsupervisorswouldbecomefrustrated
withineffecIverepairsorunabletoidenIfyaspecificissue
• Enduredlaborious,manualprocesses• Ionlyhadaccesstotestdata,ifIwas“onsite”duringtroubleshooIng• Technicianswerechallengedtocapturethedata,moveittoaUSBdrive,
andsendittomeinanemailforreview• Timeconsuming,clunkyprocess(andatonof“windshieldIme”drivingto
customershomes)
• WeknewthedatacouldtelluscriIcalinformaIonifwecouldviewthetestresults
“Techniciansweredeveloping
rela9onshipswithcustomersbecauseitwasnotuncommonforthemtobeat
theirhousesmul9ple9mesforrepair.
6
BeforeSplunk:• Istartedwith“postitnotes”withtestreadingswrikendownonthem• Begantouseaspreadsheetwithdataenteredbyhand• CreatedaMacrothatwouldimporttestresultsintothespreadsheetthattechswouldemail
• MassiveamountofdataneededautomaIon• Original“AutomaIon”waspulngthetestdataintoaphpbasedwebsite
• Wehadthedatashow“redorGreen”forpassorfail…….Buttherewasnowaytoreportthetestfailures
Toobtaindata:• TechniciansmanuallyuploadresultsfromthetestdevicetoaUSB• Plugintoalaptop• surftoawebaddress• uploadthedatafiles• IdenIfytheJobInformaIon• Technicianwhoperformedthework
7
ChoosingSplunk• Withthepreviouswebsite,Iwasnotabletomakereportsorchangeswithout
askingtheoriginaldeveloperto“pleasedoitforme”
• Hemadechoicesofwhatdatatoparseandwhichakributetodisplay
• Ofthe127testakributes,only27wereabletoview
• Therewasnotawayformeto“work”withresultseasily………..
• IlearnedaboutSplunkataCenturyLinkDSLconferenceinDenver• AcolleaguemadeakeyintroducIon,whichledtoSplunk
• WewereabletoputaSplunkconnecIonontheexisIngdatabasetocapturenewtestresultsaswellasingestALLtheprevious300K+testuploadsintoSplunk.
• Iimmediatelybeganworkingwiththedata,allthewhiledreamingofwhatcouldbeaccomplishedasfaras“themeaning”ofthetestresults.
• Itwaslikegoingfroma“closet”(whichhungdatainarelaIveorientaIon)
• Toa“sockDrawer”whereIcoulddigaroundandlookforthemissingsock
“WithSplunk,wewereableto
figureoutwhatdrivestrouble,addressthe
standardsandadjusttosupport“performance”
8
SplunkBenefits
• Webeganlookingatakributesofcustomercalls&dispatchIckets&andcomparingtouploadedtestdata
• Webuiltdashboardsforallofthe127differentakributestoshowthemedianresultsandtheupperandlowerthresholds.
• Wecouldsearchonanyakributeoranydistancewewanted
• ThenwecouldlookupcustomercallinrecordsandtroubleIcketswhich“were”inseparatedatasilos
“Splunkintegratesintoeverypor9onofwhatwedo.
We’reabletobringsiloeddatabasesanddifferentinforma9onalltogether.”
9
SplunkBenefits• SplunkwasthesoluIonformanagingourdataandprovidinganopportunitytomakeinformeddecisions
• SplunkhasstreamlinedandautomatedouroperaIonstestdatacollecIonandanalysis• Wecanconnect“siloed”databasesthatcontainrelatedtypesofinformaIon
• Forinstance,thetechniciancannowsavethetestdataonhishandheldtestunitandsubmitdirecttoourSplunkdashboards.
• Thedashboards“know”who,where,andwhenparIculartaskisworked,noneedforthemtofillintheinfotosubmitdata.
• Splunkintegratesintoeverydisciplineofwhatwedo• WiththeconnecteddatabasesanddifferenttypesofinformaIon-wegetapictureofwhatishappeningwitha
customerservice.Wecanbegintovalidate“performance”notjusttestresults• AllthedatacomestogetherinSplunkasoneview-asingleviewtoseemulIpledatasets• A“singlelogon”notmul9pledisconnectedwebpages
• WithSplunk,ourdatabasescancommunicatewitheachothertogiveusabekerview• Splunkgivesususefulvisuals–graphicalrepresentaIons–notjustlogsandnumbers
SplunkBenefits
PerformanceTracking
Log/DataManagement
ServiceManagement/Efficiency
Visibility/ReporIng/Dashboards
12
SplunkforLog/DataManagement• ASplunkforwardernowsendsdatadirectlyfromthemeter
toourindexers• Eliminatesmanualinputbythetechnician• Weknowwhodidwhatandwhere
• WecombinealllogdataintoSplunk• Machinedatasources• Databases• ModeminformaIon• Dispatchreports• Customercallindata
• Ourprocessesaremuchmorestreamlined• Independentdatabasescommunicatewitheachother• Whichgivesmuchimprovedview,whichinturnprovidesbekercustomer
experience
“Techniciansnolongerhavetoput
inwhatjobsthey’reworkingon.Wealready
know.”
13
Users/Partners• NaIonalOperaIonsteams• Analysts• DocumenIngEvaluaIng/UpdaIng/MaintainingProcessandStandards
• RegionalFieldOperaIonsteams,• Engineers• Analysts• CreatecompliancereporIngthatgoestoLocalOperaIonsteams
• LocalOperaIonsTeams• VP/GMs,Directors,AreaManagers,PlantSupervisors• Technicians
• Growingbythethousands— Weexpect5,000+usersbytheendoftheyear
14
SplunkforVisibility/ReporIng/Dashboards
“Splunkhasbecomeour
periscope.Someofthedashboardswehavearequiteincredible,totellyouthetruth.”
• Daily,weeklyandmonthlyreports• AutomatedemailsfromSplunkto“management”• Compliancereports,whouploadedwhattestdataforaservice• InventorycontrolandtrackingofSettopBoxesandGateway/Modems,whatwas
installedandwhere• Wasthe“process”followedduringcustomer“turnup”• TechniciansarenowaccountabletothetestsperformedforserviceinstallaIon
• ProcessRepeatabilityandmonitoring• Makesuretechniciansarefollowingtherightsteps• WeTestwiththesamemethods• Runthesametestscriptsanduploadresults(basedoncompanystandardsto
supportperformance)• ThetestequipmentismonitoredforFirmwareRevision• WecantracethetestequipmenttoUserandmaintaintheInventoryofCapitol
equipment• TheImetocompleteatask
• Othergroupsareaskingforourhelpto“unlock”theirinformaIon
15
• WemakereporIngavailableonalocal,regionalandnaIonallevelusingthesamedatasourcefromwithinSplunk.• WereacttothesameunfilteredinformaIon
16
SideBenefit:WeareabletoviewRevisionofFirmwareonatechnicianstestset,aswellasthelocaIonandcapabilityoftestequipment
17
18
SplunkforPerformanceTracking• WeuseSplunktotrackkeyperformancemetrics:• Settopboxperformance• Modemperformance• Copperperformance(notjust“tesIng”anylonger)
• Wetrackproblemsolvingvariablesaswell• Technicians,modems,copper,oracustomerissue(what’sontheLAN)• WithSplunk,wecannarrowitdown&pinpointtheissue
• SeeingintoallthemovingpartsofcomplexsituaIonsislikethe6sidesofaRubik’scube–wecanseetheactualsourcesofproblemsandhowtheyarerelatedtoeachother
19
SplunkasanMobileDeviceInterface
20
40Mcircuitthatthecustomerisexperiencing~7.5Mdownload
• Anexampleof“mashingdata”frommulIplesourcestocreateanopportunityforevaluaIon
21
Missingbandwidth
Bounceindicates“Bridgetap”
Fulluseofbandwidth
”looksgood”
poorperformingloop Anacceptabletest
22
SplunkUniquetool:The“PromiseChart”
Bluelinerepresenttestdataas“Losspertone”
ColoredlinesrepresentMAXlosspertoneforaService
Ifthebluelinedoesnottouchthecoloredbar,theloopcanNOTsupportthatprovisionedspeed
23
24
“goodsyncrates”
Uncorrectederrorsarepresent
25
• OriginalDispatchDataSetallowsfor45daysofhistory,wenowkeeptheinformaIonindexedinSplunkfor18months
• Wechangedthe“clunky”interface
• WeaddedMappingtothedispatch
26
SideBenefit:Weareabletoreview“technicianperformance”Didtheyperformatest,wasittherequiredtests,anddidthetestsmeetthestandard
27
28
SplunkforServiceManagement/Efficiency
• Wecannowlookatdifferentakributesfrommanysourceswithinourcompany,tofigureoutwhatdrivesacustomercall-in/poorexperience
• Wenarroweddown175testitemstothehandfulthatdrivetroublecalls• Wehaveadjustedourteststandardsto“whatdrivesapoor
customerexperience”
• Splunkhelpsusprovewhereweneedtomakeadjustmentstoavoidtroubles
• TechniciansusethisinformaIontoknowwhattolookfortofixthecustomersproblem
“WithSplunk,Iknowwehavesaved
expensedollarsusing‘costavoidance’byadjus9ngourtes9ng
standards.”
29
WeseethelengthofImeonanyjoborcrew.Wetrackthelengthof
Imethatjobstakeasthedaymoveson…………
Wereportthepercentageofuploads
Wecandrilldowntothedetailofthejobanduploads
30
SplunkWordstotheWise….• Youhavetohaveacoreunderstandingofwhatthedatarepresentstomakeituseful
• The“users”havetobeapartoftheimplementaIonandputinefforttomakethedatameaningful• Reportanddashboarddevelopment• IdenIfyindependenthowever“Related”datasourcesfor
ingesIonintoSplunk.
31
SplunkSuccesses• We’veimprovedmanagementsawarenessoftechnicianefficiencythroughautomated
reporIng.
• WeknowSplunkhasreducedcost.• PrevioustoourSplunkanalysisofourtestdata,wewoulddispatch“twotruckrolls”foraVideo
customerservedoveracopperloop.• Wenowonlydispatchonceforaninstall• Whereacustomermayhave“failedstandard”andwasdeniedservice,wenowhavestandards
thataremeaningfultoperformanceandwecanevaluatequickly
• WenowcanvalidateacustomersservicebecausewehaveinstallaIonhistorythroughuploadedtestresults.
• WeareeffecIvely“mapping”our“lastmile”performanceusingSplunk• Installsandrepairs• GatewayandSTBautomatedreporIng• DSLAMperformance
32
SplunkingAhead….• Googlemapping• We’regoingtogetmoregraphicalintermsof
layingoverinstallsandtroubleImes• Thiswillbekerhighlightwherewehave
performanceissues
• IdenIfyingwheretoinvestforahigherreturnoninvestment• Maintenanceversusinstallingnewtechnology
33
Lookingahead……….WhatifwecombinedourtestdatatodispatchinformaIon?AndthenhaveSplunkmapit?• WethencouldidenIfyissuesandfigurewheretospendourImeinmaintenance.• WecouldidenItywheretoplacenewtechnologies• WeCouldfigureoutwhyacertainareatakeslongertoinstallorwhywehaveincreasedrepairacIvity• WecouldbegintoPredict…………………..Toan9cipateresultsbasedonFactualinformaIon.
ThankYou