Leveraging MongoDB as a Data Store for Security Data

MongoDB as a Data Store for Security DataScaling out the mongod node

Daniel Bauman

Sr. Cyber Intelligence Analyst

LM-CIRT

© 2012 Lockheed Martin Corporation. All Rights Reserved.

2

Contexts

Information01101100011011010110001101101111

Influence (Application)

Intelligence


3 Key Brick Walls

© 2014 Lockheed Martin Corporation. All Rights Reserved.3

1• Isolation

2• Retention

3• Access

4

Isolated Information


01101100011011010110001101101111

01101100011011010110001101101111

01101100011011010110001101101111

01101100011011010110001101101111

5

Isolated Information


01101100011011010110001101101111

01101100011011010110001101101111

01101100011011010110001101101111

01101100011011010110001101101111

6

Pizza Boxes


✔Action

7

Single Pizza Box Throughput


✔Action

8

Pizza Boxes


✔Action

9 © 2014 Lockheed Martin Corporation. All Rights Reserved.

2• Retention

10

The Dream – MongoD Standard Install


Documents per SecondData Size

Data Size vs Documents/sec

Size

time

Doc

umen

ts/s

ec

11


The Reality – MongoD Standard Install



File size vs Inserts

Size

time

Doc

umen

ts/s

ec

12

The Dream – Data Retention




Size

time

Doc

umen

ts/s

ec

13

Mongo DatabaseDisk Is FULL

Single Pizza Box Data Retention


Trash

14

The Reality – MongoD Capped Collection



File size vs Inserts

Size

time

Doc

umen

ts/s

ec


3• Access

16

The Dream - Querying the Cloud


Query Response

0110110001101101011000110111000110110101100011010110011010110110001101101011000110101101011000110110001111000110101101100011011010

17

And now for something less technical


18

172.100.178.247

Information Retrieval

172.100.27.143 172.100.164.66 172.100.255.250 172.100.235.24 172.100.195.178 172.100.7.227 172.100.215.227 172.100.31.0 172.100.81.242 172.100.156.25 172.100.139.53 172.100.235.229 172.100.25.137 172.100.171.91 172.100.71.242 172.100.108.64 172.100.96.73 172.100.126.217 172.100.77.25 172.100.214.219 172.100.102.211 172.100.124.176 172.100.96.81 172.100.131.150 172.100.98.250 172.100.178.247 172.100.138.157 172.100.45.67 172.100.122.239 172.100.138.218 172.100.102.110 172.100.49.93 172.100.245.74 172.100.213.39 172.100.80.14 172.100.41.125 172.100.150.202 172.100.1.184 172.100.149.233 172.100.98.83 172.100.199.75 172.100.244.223 172.100.140.69 172.100.187.27 172.100.209.228 172.100.6.249 172.100.60.48 172.100.138.64 172.100.130.181 172.100.188.177 172.100.142.25 172.100.109.79 172.100.70.58 172.100.65.184 172.100.250.150 172.100.215.195 172.100.137.136 172.100.49.64 172.100.148.19 172.100.244.227 172.100.178.131 172.100.255.199 172.100.65.112 172.100.201.249 172.100.53.21 172.100.235.60 172.100.84.205 172.100.16.194 172.100.216.90 172.100.45.88 172.100.240.174 172.100.248.179 172.100.48.70 172.100.8.200 172.100.45.130 172.100.235.59 172.100.171.231 172.100.29.124 172.100.239.204 172.100.172.241 172.100.158.216 172.100.70.109 172.100.227.117 172.100.144.199 172.100.223.36 172.100.166.60 172.100.48.61 172.100.70.76 172.100.51.152 172.100.157.95 172.100.71.133 172.100.0.25 172.100.167.58 172.100.94.133 172.100.93.92 172.100.192.109 172.100.176.25 172.100.169.236 172.100.164.186


“1.0 second is about the limit for the user’s flow of thought to stay

uninterrupted” – Nielson (1993)

J. Nielsen, "Response times: the three important limits," 1993

19

Information Retrieval – 10 seconds


1968 R. Miller, "Response time in man-computer conversational transaction,"

“response delays of a standard ten seconds will not permit the kind of

thinking continuity essential to sustained problem solving”

– R. Miller(1968)

20

Diving Back In


21

Random Data Access


past recent

Documents

Python-MongoR (R for Retention)

Distributed database expansion to MongoDB designed to optimize scale-out, write intensive document storage


23

Data Buckets


past recent

Documents

24

MongoR Buckets


past recent

DB DB DB DB DB DB

25

MongoR Automated Segmenting


past recent

DB DB DB DB DBDB DB DB DB DBGenerator

26

Mongo

Disk Is Full

Mongo

MongoR Retention


Trash

Mongo

Mongo Mongo

27

MongoR

Mongo

MongoR “Capped Collection”


Mongo

Mongo Mongo

28

MongoR Destructor


past recent

DB DB DBDB GeneratorDestructor

29

MongoR Destructor


past recent

DB DB DB DB DB DB DB DBDB DB DB DB DB DB DB DB DB DB DBDB DB DB DBGenerator

30

The Real




Size

time

Doc

umen

ts/s

ec

31

MongoR Production Behavior.


32

Best Practices – Bucket Size

Bucket size = ¼ RAM size


System RAM Mongo Mongo

Mongo Mongo

33

Best Practices – Bucket Limit

Bucket Limit = 85-90% Capacity


System Drive Capacity

34

Python-mongor In Production

• MIT Licensed

– https://github.com/lmco/python-mongor


Questions


Leveraging MongoDB as a Data Store for Security Data

Technology

Transcript of Leveraging MongoDB as a Data Store for Security Data