SplunkLive New York 2011: DealerTrack
-
Upload
splunk -
Category
Technology
-
view
524 -
download
1
Transcript of SplunkLive New York 2011: DealerTrack
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Chris DeMeo Sr. Director Technology Architecture
Copyright © 2011, Splunk Inc. New York, October 27, 2011
About Me
3
! One of the founders of DealerTrack
! With the company since incepJon
! Technology Architecture Lead -‐ Infrastructure
Copyright © 2011, Splunk Inc. New York, October 27, 2011
About DealerTrack
4
! Leading provider of on-‐demand soOware and services for the automoJve retail industry
! Industry’s First and Largest Credit ApplicaJon Network
! ConnecJng approximately 17,000 dealers with more than 1,000 lenders
! Dealer Management SoluJons, Inventory SoluJons, Sales SoluJons and Processing SoluJons
Copyright © 2011, Splunk Inc. New York, October 27, 2011
The Challenge
5
• Web logs, proprietary logs, database logs, operaJng system logs – No way to look through them easily, it took hours
• Over Jme, acquired several companies resulJng in more dependencies and the need to consolidate logs across geo's and datacenters
• In POC there was doubt about the value of the tool. How many hours would it take to get up and running? How quickly before we see value?
"We needed a tool that could correlate all logs."
Copyright © 2011, Splunk Inc. New York, October 27, 2011
The First of Many Aha Moments
6
• Oracle RAC database cluster failover -‐ root cause analysis • E-‐contracJng transacJon -‐ correlaJon of customer impacJng bug using 3 correlated log types
• Database connecJons – custom input allowed trending in new ways
• With Splunk ability to quickly get to root cause (assuming you logged something!)
• Different from anything we used before • Ability to show a developer the anatomy of a problem – transacJon tracing
Copyright © 2011, Splunk Inc. New York, October 27, 2011
The Second of Many Aha Moments
7
• Capacity Planning – now has the ability to easily validate transacJon mixes and volumes
• Capacity Planning – idenJfied “bunching” of transacJons during load tests
• Increased product usage and customer retenJon – through idenJficaJon of their legacy product usage
• Outage Impact – more accurate outage impact with access to addiJonal logs
Copyright © 2011, Splunk Inc. New York, October 27, 2011
What Also Helped…..
8
• AOer 3 months execuJves felt Splunk paid for itself
• Ease of Use • Development can build their own dashboards
• Business folks saw value and asked for informaJon
• We had Splunk up and running in less than 2 months own dashboards
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Vale Add to Lending Partners
9
• Real-‐Jme visibility into credit applicaJon and decision volumes across all lenders
• Report to lenders where they stand against the compeJJon—drives more compeJJve response
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Historical Data Search
10
! Log view uJlity for our own proprietary logs… ! Our own tool doesn’t search over more than 7 days of data ! Now we can search historical data for trending analysis
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Current Environment and Set up
11
! Deployed across 4 applicaJon environments ! Non-‐Cloud based -‐ Run our own equipment ! Each environment has 1 server (will make into HA configuraJon)
! Everything runs on a single server ! Virtual machines in non-‐producJon environments ! Local storage
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Coming Soon
12
! Syslog inputs ! HA configuraJon with mulJple search heads and SAN storage ! Looking to install Web AnalyJcs app for business users ! IntegraJon with other monitoring tools ! User adopJon-‐-‐work on user educaJon-‐-‐how they are using it ! Want to give Security group ability to search across years of data ! Rollout to subsidiaries so each has their own indexer
"Finding data in our logs that we never even knew was there and that people are ge8ng value from."
Copyright © 2011, Splunk Inc. New York, October 27, 2011
Lessons Learned
13
! Before we didn't have the data, now we want to save everything ! Professional Services and EducaJon – They armed us and made us more than dangerous!
! Developed internal user doc with best pracJces and use cases ! Knowledge of your logs will help to realize value quickly