Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ •...
Transcript of Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ •...
![Page 1: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/1.jpg)
Copyright © 2016 Splunk Inc.
Tom Kopchak Hurricane Labs Tim Baldwin Hurricane Labs
Quis Custodiet Ipsos Custodes? (Who Watches The Watchmen?) Or, How Do You Know When Splunk Stops Searching?
![Page 3: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/3.jpg)
About Hurricane Labs About Hurricane Labs
3
![Page 4: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/4.jpg)
Splunk And You
4
![Page 5: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/5.jpg)
What Will Be Covered
• Best pracIces around Splunk monitoring and alerIng • Types of monitoring available • The difference between monitoring and good monitoring
5
![Page 6: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/6.jpg)
What Will NOT Be Covered
• This is NOT an end-‐to-‐end walkthrough
• We will NOT tell you which monitoring and/or alerIng plaPorm to select
• We will NOT show the specific configuraIon file changes that will be needed
6
![Page 7: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/7.jpg)
How Many Of You Have Heard This Before?
7
![Page 8: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/8.jpg)
Wouldn’t it be awesome if users never experienced a Splunk issue?
8
![Page 9: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/9.jpg)
How Does Sam Do It?
• Who is Sam? Sam is a (gender neutral) Splunk ninja
• She used to be reactive
• Now, he is proactive
9
![Page 10: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/10.jpg)
• Sam used to just assume that all problems could be solved by users wriIng beVer searches
• As she gained more experience, he learned that is not always the case
• She now monitors Splunk so that he can solve problems more quickly, oXen before his users even noIce the issue
Sam’s Story
10
![Page 11: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/11.jpg)
Splunk Data – Splunk Saved Searches/Splunk API
Splunk-‐related Processes/Services – Splunk API/OperaIng System agent
Splunk-‐related OS operaIons and/or se[ngs – SNMP/OS Agent
Normal OS operaIons and/or se[ngs – SNMP/OS Agent
What Does Sam Monitor?
11
![Page 12: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/12.jpg)
Problem: Sam’s user called up to complain about not being able to find firewall data Sam spent two days trying to “help” the user write a beVer search to “find” the data
Monitor Splunk Via Splunk
12
![Page 13: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/13.jpg)
Finally found that the data stopped flowing into Splunk Fixing the issue was easy once it was discovered Sam realized that (s)he could have prevented a Icket if he/she was monitoring for data that stopped coming in
Monitor Splunk Via Splunk
13
![Page 14: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/14.jpg)
Splunk Searches • Broken Sources Sanity Check • index=_introspecIon • | REST
DMC • Splunk Distributed Management Console • Processor and Memory • Licenses -‐ ExpiraIon and Quota Usage • Missing forwarders • Disk Usage • Processing Queues • Search Peers
Monitor Splunk Via Splunk
14
![Page 15: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/15.jpg)
Broken Sources Sanity Check • Runs a search using “| metadata” to pull last Ime that a host sent data • Is “tunable” using a lookup table • Available on splunkbase: hVps://splunkbase.splunk.com/app/3247/
index=_introspecIon • Resource usage on a per-‐search or system-‐wide basis
• Disk uIlizaIon • Example in the Appendix
| REST • Pull informaIon about Splunk system health, license uIlizaIon, etc. • Example in the Appendix
Finding When Data Stops Flowing
15
![Page 16: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/16.jpg)
Splunk Distributed Management Console (DMC)
16
![Page 17: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/17.jpg)
DMC Distributed Mode
17
![Page 18: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/18.jpg)
DMC Alerts Setup
18
![Page 19: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/19.jpg)
Problem: Sam’s user called up to complain about not being able to find firewall data…. Again Sam didn’t get an alert from the Broken Sources Sanity Check Sam reverted to the default assumpIon that the user was running poor searches
Monitor Splunk Outside Of Splunk
19
![Page 20: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/20.jpg)
Sam finally found that the data stopped flowing into Splunk – Same problem as before
But wait -‐ didn’t we have monitoring for this?!?
Monitor Splunk Outside Of Splunk
20
![Page 21: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/21.jpg)
The Broken Sources Sanity Check didn’t alert because the search was ge[ng skipped
What’s Going On?
21
![Page 22: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/22.jpg)
Direct REST API • Use REST API endpoints to show and alert on certain informaIon
• Listens on port :8089 by default • Could replace some of the “| REST” searches to reduce the search concurrency
• Splunk Messages • Licensing informaIon • Deployment Client status
• Check that specific clients are checking in
• Indexer Cluster Search factor/ReplicaIon Factor • Indexer Cluster Node status • Many others
• hVp://docs.splunk.com/DocumentaIon/Splunk/latest/RESTREF/RESTprolog
Monitor Splunk Outside Of Splunk
22
![Page 23: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/23.jpg)
Monitoring Splunk using Splunk Searches
So, We Should Be Good, Right? Monitoring Splunk outside of
Splunk (REST API)
23
![Page 24: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/24.jpg)
• Problem: Sam’s user called up to complain about not being able to find firewall data…. Again?!
• She did not get an alert from the Splunk Searches
• But, he did get an alert from the Splunk REST API checks:
Splunk GUI message: “Cannot write data to index path "/mnt/splunk_warm/firewall/db" because you are low on disk space on parIIon "/mnt/splunk_warm". Indexing has been paused. Will resume when free disk space rises above 5000MB.”
Monitor Splunk-‐related OS Se[ngs
24
![Page 25: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/25.jpg)
Alerts should be: • Relevant • Timely • AcIonable
Monitoring Vs. Good Monitoring
There is a difference
25
![Page 26: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/26.jpg)
Found that the splunk_warm parIIon ( /mnt/splunk_warm ) has very liVle disk space available But wait -‐ didn’t the REST API check noIce this ?!?
– Yes, it did -‐ but not Imely
– Sam needs to know BEFORE Splunk stops working
Sam realized that (s)he could have prevented a Icket if he/she was monitoring for Splunk-‐related parIIons and processes
Monitor Splunk-‐related OS Se[ngs
26
![Page 27: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/27.jpg)
Disk Usage • Could replace the index=”_introspecIon” searches to reduce search concurrency • $SPLUNK_HOME disk space • Splunk hot/warm/cold disk space
Processes • Splunk processes • Syslog-‐ng processes • API processes (Java bridge, for example)
Listening Ports • 443 or 8000 • 8089
HTTPS checks • Cert expiraIon
Monitor Splunk-‐related OS OperaIons Se[ngs
27
![Page 28: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/28.jpg)
Monitoring Splunk using Splunk Searches
So, We Should Be Good, Right?
Monitoring Splunk outside of Splunk (REST API)
Monitor Splunk-‐related OS operaIons se[ngs
28
![Page 29: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/29.jpg)
Problem: Sam’s user called up to complain about not being able to find firewall data…. Again!!!! She did not get an alert from the Splunk Searches He did not get an alert from the Splunk REST API checks There was no alert from any of the Splunk disk or process monitoring checks
General (Non-‐Splunk) OperaIng System Things
29
![Page 30: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/30.jpg)
• Finally found that the root parIIon ( / or C:\ ) of the operaIng system filled up
• But wait -‐ didn’t Sam have disk space monitoring?
• Sam realized that (s)he could have prevented a Icket if he/she was monitoring for issues on the OperaIng System
General (Non-‐Splunk) OperaIng System Things
30
![Page 31: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/31.jpg)
• Disk Space and mount points • Disk/RAID health • SSHD Process • Port 22 listening • Server upIme
– Alert if system was unexpectedly rebooted – Alert if system hasn’t been ge[ng rebooted during patching cycle
• Memory usage • CPU usage • Load Average • System Time -‐ Important that NTP is working for Splunk • Network interface(s)
General (Non-‐Splunk) OperaIng System Things
31
![Page 32: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/32.jpg)
• Sam’s CIO was giving live demo to the CEO to request funding for a larger Splunk license
• Sam was alerted that firewall logs stopped coming in and fixed the issue within minutes
• CEO was so impressed with the cool stuff they were doing with Splunk that she gave them funding to double their Splunk license!
General (Non-‐Splunk) OperaIng System Things
32
![Page 33: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/33.jpg)
• Sam is now alerted before the users noIce
• Sam is proacIve
• We should all be more like Sam
Sam No Longer Has Users Calling With Issues
33
![Page 34: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/34.jpg)
• Monitoring system can trigger proacIve acIon
• This is the future state of monitoring
• Examples:
– Restart SSHD if cannot connect
– Run ntpdate if Ime is not synced with NTP
– When disk space gets low, automaIcally open a Icket with the storage team
– ReauthenIcate API when the API key expires
– Restart Splunk if no splunkd processes are running
What’s Next? AcIve Response
34
![Page 35: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/35.jpg)
Wrap Up And QuesIons
35
![Page 36: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/36.jpg)
Slide # 15 • Broken Hosts App for Splunk:
• hVps://splunkbase.splunk.com/app/3247/
Appendix – Slide #15
36
![Page 37: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/37.jpg)
Slide #15: index=_introspec7on search example:
ê index=_introspecIon sourcetype=splunk_resource_usage component=Hostwide | Imechart Median(data.cpu_system_pct) AS "System CPU" Median(data.cpu_user_pct) AS "User CPU"
| REST search example:
ê | rest /services/licenser/licenses Addi7onal REST endpoint informa7on:
ê hVp://docs.splunk.com/DocumentaIon/Splunk/latest/RESTREF/RESTprolog
Appendix – Slide #15 (ConInued)
37
![Page 38: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/38.jpg)
Slide # 21 • Simple: index=_internal sourcetype=scheduler status=skipped
• Advanced: index=_internal sourcetype=scheduler status=skipped | eval reason=if(like(reason, "%historical concurrent system-‐wide%"),"system-‐wide concurrent", reason) | Imechart span=30m count by reason
Appendix – Slide #21
38
![Page 39: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/39.jpg)
Slide #25 • Relevant
ê Reduce false negaIves and reduce false posiIves ê Never alert when it’s not a problem and always alert when it is a problem
• Timely ê Be as proacIve as possible ê Not too early but not too late
• Ac7onable ê If there’s nothing that can be done, then it should not be an alert ê We may want to know about trending issues if they persist
Appendix – Slide #25
39
![Page 40: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/40.jpg)
Checked with REST API: License Master Connection Splunk Messages Concurrent Searches Search Peer Connection Deployment Client Status License Usage Cluster Replication Factor Status Cluster Search Factor Status Cluster Peer Status Cluster Maintenance Mode Checked with port scan: Splunkd TCP Port (8089) available SplunkWeb Port Checked with host agent: SSH Port Splunkd Process Syslog Daemon
Checked with SNMP: SNMP Time SNMP Environment Status Disk/ParIIon Free Space Memory/Swap Usage Load Average UpIme Interface Status Checked with HTTPS connec7on: HTTP/HTTPS availability CVE-‐2009-‐3555 (TLS RenegoIaIon) Vuln
CVE-‐2011-‐3389 (BEAST) Vuln CVE-‐2014-‐0160 (Heartbleed) Vuln CVE-‐2014-‐3566 (POODLE) Vuln SSL CerIficate ExpiraIon Checked with Splunk Search: Broken Sources Sanity Check DMC Alerts Notable Event Outage
Appendix – How We Monitor
40
![Page 41: Quis$CustodietIpsos$Custodes?$$ … · WhatWill$Be$Covered$ • $BestpracIces$around$Splunk$monitoring$and$alerIng$ • $Types$of$monitoring$available$$ • $The$difference$between$monitoring$and$good](https://reader034.fdocuments.us/reader034/viewer/2022042309/5ed6a75b39f16f294d573679/html5/thumbnails/41.jpg)
THANK YOU