Puppalitics Insights Puppet landscape v1.0...
19
Use Big Data tooling to get insights in a Puppet landscape Puppalitics 24-09-15 Version 1.0 by Kevin Schaul
Transcript of Puppalitics Insights Puppet landscape v1.0...
Use Big Data tooling to get insights in a Puppet landscapePuppalitics
24-09-15Version 1.0 by Kevin Schaul
Who am I?� Kevin Schaul� Data engineer @ Itility� Analytics toolset
- Splunk- Python- R / Stata
24-09-15 www.itility.nl 2
Big Data = Hot (or not?)
24-09-15 www.itility.nl 3
Main Question of today
How to use a Big Data toolset on Puppet andother machine generated data
(and why should I care?)
24-09-15 www.itility.nl 4
Agenda� Introduction to Splunk� Data Collection� Analytics approach� Dashboard demo� Hypothesis session� Questions
24-09-15 www.itility.nl 5
Splunk
24-09-15 www.itility.nl 6
Why Splunk� A Big Data toolset (for ingest, store, query, visualize)
� Connects any data, of any size, any location and any source- Splunk connects to IT infra through API’s, real-time data gathering- Combine, correlate and visualize machine-generated IT Data
Splunk provides a solution, not necessarily the solution, for the problem:
“How to manage infrastructure more efficiently than in the past”
24-09-15 www.itility.nl 7
Data collection� Python scripts perform API calls to PuppetDB to obtain data
� 6 categories- Events- Facts- Resources- Catalogs- Reports- Nodes
� Additional source: Github, vCenter, ServiceNow24-09-15 www.itility.nl 8
24-09-15 www.itility.nl 9
Example data collection
Puppet API-endpoints� PuppetDB data can be queried with a REST API� Queries are processed immediately� Multiple endpoints
24-09-15 www.itility.nl 10
Getting data in
Source: Patrick Maddox – Puppet labs
24-09-15 www.itility.nl 11
NODE
PUPPETMASTER
1 Facts
2 Catalog
4Report
3Report
SSLsecure
Report Collector
PuppetDB
Getting data in – Puppet to Splunk
24-09-15 www.itility.nl 12
NODE
PUPPETMASTER
1 Facts
2 Catalog
4Report
3Report
SSLsecure
Report Collector
PuppetDB
ForwardingAgent
Hypothesis session� Regular sessions to refine dashboard needs� Hypothesis approach for out-of-the-box thinking
24-09-15 www.itility.nl 13
viaPhysicalmodels
Observed data
HYPOTHESISExperiments
Scientific method
Simulated data
Simulation
Scientific method extended
viaMathematical
modelsRepresentations ofThe real system
Hypothesis session
Hypothesis True False ResultA Puppet Master has < 70% CPU loadLess than 1% of a Puppet Master log contain errors95% of runs take less than 10 secondsPuppet runs never fail longer than 24 hours in a rowWindows R2 servers show less Puppet deviations(events) than non-R2 servers
24-09-15 www.itility.nl 14
Demo
24-09-15 www.itility.nl 15
Hypothesis brainstorm
Think of hypotheses that intrigue you:
24-09-15 www.itility.nl 16
PuppetDBEngineer/Module-
development
In combinationwith any other
datasource
Insights fromSplunk backto Puppet
Way forward� Connect with multiple data sources
- vCenter- Netapp- Github- ServiceNow
� Which other data sources would you add?
� Other tools which provide similar or additional insights?
24-09-15 www.itility.nl 17
End notes� Tutorial
- https://www.splunk.com/web_assets/v5/book/Exploring_Splunk.pdf
� Documentation- http://docs.splunk.com/Documentation/Splunk
� Best practices- http://wiki.splunk.com/Community:More_best_practices_and_processes
24-09-15 www.itility.nl 18
Tutorial Documentation Best practices
Contact us at [email protected] of presentation
