Growing SplunkTyler Rutschman - Garmin International
Tuesday, May 15, 12
About Me
Linux System Administrator
Husband and Father of 2 Kids
DevOps, Productivity Hacks and Tools, The Big Lebowski
Tuesday, May 15, 12
OH: (during an outage)I don’t want to live in a world without Splunk.
Tuesday, May 15, 12
Backstory
Free instance installed in 2009 by the Network TeamSingle Instance on Central Log serverUpgrade to Enterprise
Tuesday, May 15, 12I started as an IT intern my mentor had a free copy of version 2.x running on the log server.
I was tasked with finding a solution for SOX & PCI requirements. (Which was mind expanding for an intern, to say the least)
Worked with purchasing to get a small license for the enterprise features.
My project ended up piping Splunk output into a python program that no one but I understood that printed out a text report that (I felt at least) was superior to the one in place at the time. (Big surprise, didn’t end up using it).
Building Blocks
Split Splunk off onto dedicated instanceLicense overwhelmed by single appLimited visibility and use
Tuesday, May 15, 12When I came back there was some cursory interest in the app, but no major users and no project champion.
Welcome, back Tyler... Splunk Expert (by Default).
I was also attached to Garmin Connect, which is our awesome fitness tracking site, after getting more comfortable in my settings, I began to integrate the site logs into Splunk
IF YOU HAVE MORE INPUTS THAN LICENSE
YOU’RE GONNA HAVE A BAD TIME
Tuesday, May 15, 12Obvious, but this was my experience during the first dedicated instance. We had a small license and it was all being used by Garmin Connect. It really wasn’t taking hold like I knew it could.
Plan for Expansion
Decided to make application more robustRead the DocumentationPlanned roll out
Multiple Applications
License Increase
Scalable Architecture
Tuesday, May 15, 12After I became more comfortable in my position, I felt impelled to make the application more robust and widespread.
I went to .conf last year, attended some training sessions and read up on the Administration documentation.
Enterprise ArchitectureElements
Puppet DeployInfrastructure LayoutGotchasFuture Plans
(so far)
Tuesday, May 15, 12Overview of the Current Architecture Elements, will then go in depth a bit more on each subject.
Puppet
Search, Indexer and Forwarder are “turn-key”ex: include splunk::indexer ...done
Really Awesome for Forwarders
Tuesday, May 15, 12Puppet makes deployment simple. Servers are built with one include statement.
Forwarders are split up based on role and inputs. Customize the inputs a bit if necessary and include the splunk forwarder class in the puppet node definition.
Infrastructure
Tuesday, May 15, 12Describe layers and functions.
Search is load balanced.
Search, Index and Forwarders are horizontally scalable.
Network/Taiwan instances aren’t pictured but are separate dedicated instances. Will move the network index into the main infrastructure real soon now.
How We Use SplunkWeb Access Logs
Service Usage Metrics
Feature Tracking
Diagnosing Problems in Production
Internal Application AuditsWindows Security Events
Tuesday, May 15, 12We don’t have a wide variety if inputs into Splunk at the moment.
We currently use it on all of the major IT web applications to obtain service metrics, track new features and diagnosing issues in Production.
The developers are also starting to cater their applications to output Splunk friendly logs
Windows security events are queried via WMI ad filtered to specific IDs, this helps keep the volume down while delivering value for the Windows guys.
Why I like Splunk
Makes Users HappyReal Time DataNo Alternatives
Tuesday, May 15, 12Ease of configuration, having the one stop shop for user-land configs. LDAP integration is super simple.
Able to generate detailed reports and drill into the data on the fly is a killer feature and something that you simply won't find with any other application.
User community and Documentation.
There are no real alternatives to Splunk. Some tools touch on some of the features gained with the app, but there is no offering that matches what splunk can give you. I’ve tried SEC, logwatch, Logstash, and Spiceworks. None were as user friendly and robust as Splunk.
Gotchas
Don’t Index a lot of data over NFSShared Knowledge Bundle Time SyncTag and Search permissions
Tuesday, May 15, 12Keeping up with the demand. From a license and user request perspective, I limited amount of time to handle the requests at hand. Familiar position for me at least, but a good problem to have.
Mounted Bundles must have the same time across the board.
Watch your permissions on saved searches and tags. They are usually private when I share them with another user and they cannot access.
Future Plans
Fix Central LoggingCheck Out Deployment ServerMore InputsTraining
Tuesday, May 15, 12Currently only one centralized syslog server, want to scale it out and put a farm of syslog servers behind a load balancer. Splunk will be the defininitive timeline for syslog events.
Read about Deployment Server but passed on it at the time. Would like to pick it back up and see how it could be beneficial.
Add additional inputs to the application
I’ve been tasked with training my coworkers on how to use the application. Once they pick it up and figure it out, they can do awesome things.
Tips and Advice
WMI Event Filter for Windows EventsSplunkbase (stack overflow engine)
Tuesday, May 15, 12
Questions & Feedback
Tuesday, May 15, 12
Top Related