Eko10 Workshop Opensource Database Auditing
-
Upload
juan-berner -
Category
Technology
-
view
164 -
download
1
description
Transcript of Eko10 Workshop Opensource Database Auditing
![Page 1: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/1.jpg)
OPEN SOURCE DATABASE MONITORING AT SCALE
#eko10
![Page 2: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/2.jpg)
What is this talk about?● Auditing tools available for databases, commercial and otherwise
● What problems they have
● How we can make them scale as much as we might need
![Page 3: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/3.jpg)
About us
● Juan Bernero @89berner / [email protected]
o Hobbies = [‘Movies/Series','Reading','Programming']
o Mostly Blue Team
o http://secureandscalable.wordpress.com/
● Pablo Garbossao @pgarbossa / [email protected]
o Fully Blue Team
![Page 4: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/4.jpg)
About MercadoLibre● Devops culture (everyone and their mothers can access the boxes)
● Different DBs technologies
● Hybrid Cloud
● Database servers > 1K && Servers > 15K
● More than 100000 qps
![Page 5: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/5.jpg)
Commercial products● Expensive
● Lots of functionalities you might not need
● Don’t scale so well
● Will make you choose what to log
![Page 6: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/6.jpg)
Audit options● Inline / TAP / sensors or agents
● Plugin based
● Sniffers
● Client loggers
![Page 7: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/7.jpg)
Mysql Audit Options● Commercial products
● Mysql General Log
● MySQL Enterprise Audit Log Plugin
● Mysql audit plugins
● Mysql sniffer
![Page 8: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/8.jpg)
Mysql General Log● Easy to activate, by default in mysql
● Can be customized by modifying the log table to a degree
● As of 5.1 can be activated on the fly
● Less freedom than audit plugins
![Page 9: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/9.jpg)
MySQL Enterprise Audit
Log Plugin● Available for Mysql Enterprise
● Uses the open MySQL Audit API
● Does not log triggers or prepared statements
● Allows asynchronous or synchronous logging
![Page 10: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/10.jpg)
Mysql Audit Plugin (1)● Works using API created by Mysql to replace the general log
● Available in Github
● Flexibility to choose objects to inspect, types of queries to log or users to
whitelist
● Similar restrictions as Mysql Enterprise Plugin
![Page 11: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/11.jpg)
Mysql Audit Plugin (2)● Steps to audit
o Download the plugin from github
o Move the library to /usr/lib/mysql/plugin/
o Enable with INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so';
o Either send it to a file or to a socket
![Page 12: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/12.jpg)
Mysql Audit Plugin (3)● The log now looks like:
"msg-type":"activity","date":"1414531661274","thread-id":"72","query-
id":"1600563","user":"workshop","priv_user":"workshop","host":"ip-172-31-32-202.us-west-
2.compute.internal","ip":"172.31.32.202","cmd":"show_fields","query":"show_fields"
![Page 13: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/13.jpg)
Mysql Audit Plugin (4)We can parse it with logstash into:
{
….
"@timestamp": "2014-10-29T04:10:37.000Z",
"type": "mysqlplugin",
"host": "0.0.0.0",
"path": "/var/log/mysqlplugin-2014-10-29.log",
"date": "Oct 29 04:10:37",
"agent": "54.200.106.239",
"user": "workshop",
"priv_user": "workshop",
"srcip": "54.69.169.73",
"command": "show_fields\",\"",
"query": "show_fields"
}
}
![Page 14: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/14.jpg)
Mysql Audit Plugin DEMO
![Page 15: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/15.jpg)
Problems with the Plugin● Generates overhead on the host
● Can’t log all events (audit api limitations)
● Not available in sniffing only situations
![Page 16: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/16.jpg)
Sniffing options● Span port which sends you the traffic
● Sniffing and parsing from the server
● Forwarding the traffic from the agent (ie: iptables)
● Using agents to sniff traffic and forward it to repeaters (which repeat locally
the traffic with the original address)
![Page 17: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/17.jpg)
Mysql Sniffer● Client / Server architecture
● Sniffs for common queries (select/insert/update/delete)
● Beta phase
● Has to keep up with protocol changes
![Page 18: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/18.jpg)
Mysql Sniffer Agent● It will sniff traffic on Mysql port 3306 and send it elsewhere
● Small use of resources
● Must be tweaked to work in high load situations (ie: increase buffer for
packets to be processed)
./agent eth0 3306 DESTINATION 9200 1000 5000 5
![Page 19: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/19.jpg)
Mysql Sniffer Repeater● Application that listens at a tcp port for connections
● Receives packets and does a local replay of them
● Packets are seen as coming from the original client
./repeater 9200
![Page 20: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/20.jpg)
Mysql Sniffer Parser● Listens to traffic on the interface for the mysql port
● Parsers queries and keeps track of connections
● Writes output to logfile:
Wed Oct 29 00:20:24,54.69.169.73,55981,172.31.32.202,workshop,test,
select,"select * from test"
![Page 21: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/21.jpg)
Mysql Sniffer DEMO
![Page 22: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/22.jpg)
Mysql Sniffer Problems● Not reliable
● Depends on the protocol not to change or something weird not to happen
● Only a limit subset of types of queries which represent most queries
● Shouldn’t be used on databases with small activity
![Page 23: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/23.jpg)
You can use a combo● Mysql sniffer to audit common queries without giving overhead to the
mysql server
● Mysql Audit Plugin to audit all other queries or specific objects with more
reliability
● Be creative
![Page 24: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/24.jpg)
MongoDB● Document oriented database
● Great scaling capabilities
● Bson Data Store
● Most popular NoSQL (according to wikipedia)
![Page 25: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/25.jpg)
MongoDB Operations● Insert: db.scores.save({a:99})
● Delete: db.scores.remove({server: 999});
● Update: db.scores.update({a: 5}, {server:999});
● Query: db.scores.find();
![Page 26: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/26.jpg)
MongoDB auditing options● Server log
● MongoDB Enterprise Auditing
● Query to the oplog
● Mongosniff
![Page 27: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/27.jpg)
MongoDB Mongosniff● Gives you detailed output of operations in MongoDB
● Does not come in the default package, you need to compile it
● Uses the mongo libraries to parse the commands
● Sample output: 111.22.33.44:6612 <<– 22.33.44.55:42947 262 bytes id:6a89eb 6982123 –
308293
reply n:4 cursorId: 0
{ _id: “db”, partitioned: false, primary: “Segmon_RS1″ }
![Page 28: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/28.jpg)
MongoDB Mongosniff
(Modified)● Some pcap tweaks to reduce dropped packets
● Minor bug fixes
● Different output format:
172.31.36.172:56228,54.68.230.224:6612,test.$cmd,,query,{ authenticate: 1, nonce:
"745ad1e4a6075a25", user: "workshop", key: "869c8d69703e2d1bb9394ddf4c116dcb" }
ntoreturn: 1 ntoskip: 0AAAAAAA
![Page 29: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/29.jpg)
MongoDB Mongosniff
Wrapper● Ruby wrapper
● Handles extra functions we would need without modifying mongosniff
● Output format:
Oct 29 03:43:11,workshop,54.68.230.224,workshop,test,172.31.36.172:56231, 54.68.230.224:6612,
test.cmd,,query,{ isMaster: 1.0, forShell: 1.0 } ntoreturn: -1 ntoskip: 0
![Page 30: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/30.jpg)
MongoDB Mongosniff
Architecture
![Page 31: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/31.jpg)
Mongo Sniffer DEMO
![Page 32: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/32.jpg)
MongoDB Sniffer Problems● No support for packet fragmentation
● Not 100% reliable
● Not it’s intended use
![Page 33: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/33.jpg)
References
● https://github.com/89berner/MysqlAudit
● https://github.com/mcafee/mysql-audit
![Page 34: Eko10 Workshop Opensource Database Auditing](https://reader036.fdocuments.us/reader036/viewer/2022062419/559b52341a28ab8e4e8b46ca/html5/thumbnails/34.jpg)
thank you!
we’re hiring ;)