HIDS as a Service
Transcript of HIDS as a Service
![Page 1: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/1.jpg)
HIDS as a Service
Ivan AgarkovSecurity Infrastructure Engineer
20k HIDS cluster definitive guide
![Page 2: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/2.jpg)
About myself
- Ivan Agarkov- 2003-present - security guy- Securing Wargaming since 2015- SELinux & Perl fan- Internal trainer- Doing ‘security RnD’- ‘Extreme’ CTF tasks author
@annmuor
![Page 3: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/3.jpg)
What the f***ing HIDS?
![Page 4: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/4.jpg)
• H - Host• I - Intrusion• D - Detection• S - System
What does HIDS mean?
![Page 5: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/5.jpg)
collect
How it works?
analyze
detectalert
![Page 6: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/6.jpg)
Collection
logs
file checks
active checks
rootkit checks
normalize data compress data send for analysis
![Page 7: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/7.jpg)
Detection
log classify
generate eventfill meta
set priority
ruleset check
![Page 8: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/8.jpg)
Analysis
● How many times an event fired?
● What was changed since the last run?
● Is it eligible to generate an alert?
● Is it eligible to set alert as ‘multiple’?
![Page 9: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/9.jpg)
Alert
alert store / archive
send report( later ) notify
urgent?email
messenger
phone call
escalation
![Page 10: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/10.jpg)
OSSEC-related
collect ossec-agentlessd
ossec-logcollector
ossec-syscheckd
ossec-agentd
ossec-remoted
detect &analyze
ossec-analysisd
alert ossec-reportd
ossec-maild
ossec-integrated
![Page 11: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/11.jpg)
A long time ago, in a galaxy far far away...
![Page 12: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/12.jpg)
![Page 13: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/13.jpg)
2010 - 2014
- 50 - 5000 servers- Manual log handling- syslog + ansible to
collect- cat/grep to find
something- how did we live?
like that!
![Page 14: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/14.jpg)
2014 - 2016
- 5000-10000 servers- ELK stack to collect
logs- Kibana to find
something- What could go wrong?
136M logs ( strings ) per day, oops
![Page 15: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/15.jpg)
2016 - present
- 10k-20k servers- HIDS agent on each server- Collect only significant- Alert if something goes
wrong- Kibana is still here
200-300k events per day ( now )
![Page 16: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/16.jpg)
Building the cluster
![Page 17: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/17.jpg)
First try
ossec central
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
![Page 18: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/18.jpg)
First try - results
osseccentral
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
databaselosing data
UDP
![Page 19: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/19.jpg)
Second try
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
![Page 20: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/20.jpg)
Second try - results
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
bottle neck
![Page 21: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/21.jpg)
Switched to WAZUH
=
![Page 22: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/22.jpg)
Third try
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
UDP
elasticlogstash
![Page 23: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/23.jpg)
Third try - results
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
UDP
udp overload
elasticlogstash
![Page 24: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/24.jpg)
Third try - details
UDP TCP
![Page 25: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/25.jpg)
Finally
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
TCP
elasticlogstash
![Page 26: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/26.jpg)
Data collection
![Page 27: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/27.jpg)
Data collection scheme
ossec cluster
elasticlogstash wg plugin
redis
ossec rulesetlogs
kibana
![Page 28: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/28.jpg)
OSSEC ruleset● Based on wazuh PCI DSS ruleset● Works as puppet submodule● Alerts count was reduced 20 times● 60% of ruleset is useless● Custom rules based on our needs● Reduces logs 450 times !Collect 1
week stats
Lower levels
Find useless
6 000 000 3 000 000 300 000 in 3 month
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
![Page 29: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/29.jpg)
WG plugin
● Put server’s metadata into the alert● Put user’s metadata into the alert● Normalize alert’s data● Hides secret data
server responsible
user real name
ssh key owner
server owner
alert
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
![Page 30: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/30.jpg)
WG plugin/redisossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
ssh keys archive
CMDB
redisSLOW FAST
![Page 31: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/31.jpg)
WG plugin/sample
vs
![Page 32: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/32.jpg)
Elasticsearch
nfs data node
node1 node2
curator jobs
kibana
logstash output
● Alias per project/owner● Archive old indexes
SOC network
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
![Page 33: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/33.jpg)
Elasticsearch/curator
● Runs once per day● Creates ‘aliases’● Hides some data from teams● Prevents information disclosures
![Page 34: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/34.jpg)
Last but one boring scheme
thing 2
thing 1
database
![Page 35: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/35.jpg)
Kibana
kibana
● ADFS + mod_mellon to authenticate● nginx + mod_lua to authorize● user groups = server groups = aliases
User network
apachemod_mellon
nginxmod_lua
ADFS user groups
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
![Page 36: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/36.jpg)
Command & Control
![Page 37: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/37.jpg)
Server lifecyclesetup
ready
production shred
free
![Page 38: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/38.jpg)
HIDS lifecycle
production shred
![Page 39: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/39.jpg)
Production
● Install HIDS agent● Find HIDS server● Do a handshake● Download agent configuration● Start agent service● Deal with failures
![Page 40: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/40.jpg)
HIDS puppet code package
$server from hiera
config.erbagent-auth
service
zabbix logrotate
![Page 41: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/41.jpg)
Deal with failures
● Service failed puppet failed● No logs from agent zabbix trigger● Port is down zabbix trigger
![Page 42: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/42.jpg)
Shred
● Remove host from HIDS server
![Page 43: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/43.jpg)
Remove host from HIDS server
● Each server is running Wazuh API● API allows to manage agents● Cleanup agents on shred● Cleanup agents on ‘connection loss’
![Page 44: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/44.jpg)
Making profit
![Page 45: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/45.jpg)
How can we help engineers?
![Page 46: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/46.jpg)
Track users
![Page 47: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/47.jpg)
Debug SELinux
![Page 48: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/48.jpg)
Figure out how new feature breaks our web
![Page 49: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/49.jpg)
Find puppet bugs
![Page 50: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/50.jpg)
Look for hacking attempts
![Page 51: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/51.jpg)
How can we help business?
![Page 52: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/52.jpg)
Control our employees
![Page 53: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/53.jpg)
Generate reports & trends
![Page 54: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/54.jpg)
Inform about significant events
![Page 55: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/55.jpg)
Create annual reports
![Page 56: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/56.jpg)
How to get more?
![Page 57: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/57.jpg)
Take my money!
![Page 58: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/58.jpg)
More money!
![Page 59: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/59.jpg)
No more money :(
![Page 60: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/60.jpg)
Afterword
![Page 61: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/61.jpg)
- Worth it?- Sure- Will help to secure my business?- Indirect ways mostly
![Page 63: HIDS as a Service](https://reader030.fdocuments.us/reader030/viewer/2022032422/6233d723f96b642281428236/html5/thumbnails/63.jpg)