Leveraging MongoDB as a Data Store for Security Data

MongoDB as a Data Store for Security DataScaling out the mongod node

Daniel Bauman

Sr. Cyber Intelligence Analyst

LM-CIRT

Contexts

Information01101100011011010110001101101111

Influence (Application)

Intelligence

3 Key Brick Walls

1• Isolation

2• Retention

3• Access

Isolated Information

01101100011011010110001101101111

Isolated Information

01101100011011010110001101101111

Pizza Boxes

✔Action

Single Pizza Box Throughput

✔Action

Pizza Boxes

✔Action

2• Retention

The Dream – MongoD Standard Install

Documents per SecondData Size

Data Size vs Documents/sec

The Reality – MongoD Standard Install

File size vs Inserts

The Dream – Data Retention

Mongo DatabaseDisk Is FULL

Single Pizza Box Data Retention

The Reality – MongoD Capped Collection

File size vs Inserts

3• Access

The Dream - Querying the Cloud

Query Response

0110110001101101011000110111000110110101100011010110011010110110001101101011000110101101011000110110001111000110101101100011011010

And now for something less technical

172.100.178.247

Information Retrieval

172.100.27.143 172.100.164.66 172.100.255.250 172.100.235.24 172.100.195.178 172.100.7.227 172.100.215.227 172.100.31.0 172.100.81.242 172.100.156.25 172.100.139.53 172.100.235.229 172.100.25.137 172.100.171.91 172.100.71.242 172.100.108.64 172.100.96.73 172.100.126.217 172.100.77.25 172.100.214.219 172.100.102.211 172.100.124.176 172.100.96.81 172.100.131.150 172.100.98.250 172.100.178.247 172.100.138.157 172.100.45.67 172.100.122.239 172.100.138.218 172.100.102.110 172.100.49.93 172.100.245.74 172.100.213.39 172.100.80.14 172.100.41.125 172.100.150.202 172.100.1.184 172.100.149.233 172.100.98.83 172.100.199.75 172.100.244.223 172.100.140.69 172.100.187.27 172.100.209.228 172.100.6.249 172.100.60.48 172.100.138.64 172.100.130.181 172.100.188.177 172.100.142.25 172.100.109.79 172.100.70.58 172.100.65.184 172.100.250.150 172.100.215.195 172.100.137.136 172.100.49.64 172.100.148.19 172.100.244.227 172.100.178.131 172.100.255.199 172.100.65.112 172.100.201.249 172.100.53.21 172.100.235.60 172.100.84.205 172.100.16.194 172.100.216.90 172.100.45.88 172.100.240.174 172.100.248.179 172.100.48.70 172.100.8.200 172.100.45.130 172.100.235.59 172.100.171.231 172.100.29.124 172.100.239.204 172.100.172.241 172.100.158.216 172.100.70.109 172.100.227.117 172.100.144.199 172.100.223.36 172.100.166.60 172.100.48.61 172.100.70.76 172.100.51.152 172.100.157.95 172.100.71.133 172.100.0.25 172.100.167.58 172.100.94.133 172.100.93.92 172.100.192.109 172.100.176.25 172.100.169.236 172.100.164.186

“1.0 second is about the limit for the user’s flow of thought to stay

uninterrupted” – Nielson (1993)

J. Nielsen, "Response times: the three important limits," 1993

Information Retrieval – 10 seconds

1968 R. Miller, "Response time in man-computer conversational transaction,"

“response delays of a standard ten seconds will not permit the kind of

thinking continuity essential to sustained problem solving”

– R. Miller(1968)

Diving Back In

Random Data Access

past recent

Documents

Python-MongoR (R for Retention)

Distributed database expansion to MongoDB designed to optimize scale-out, write intensive document storage

Data Buckets

past recent

Documents

MongoR Buckets

past recent

DB DB DB DB DB DB

MongoR Automated Segmenting

past recent

DB DB DB DB DBDB DB DB DB DBGenerator

Disk Is Full

MongoR Retention

Mongo Mongo

MongoR

MongoR “Capped Collection”

Mongo Mongo

MongoR Destructor

past recent

DB DB DBDB GeneratorDestructor

MongoR Destructor

past recent

DB DB DB DB DB DB DB DBDB DB DB DB DB DB DB DB DB DB DBDB DB DB DBGenerator

The Real

MongoR Production Behavior.

Best Practices – Bucket Size

Bucket size = ¼ RAM size

System RAM Mongo Mongo

Mongo Mongo

Best Practices – Bucket Limit

Bucket Limit = 85-90% Capacity

System Drive Capacity

Python-mongor In Production

• MIT Licensed

– https://github.com/lmco/python-mongor

Questions

Leveraging MongoDB as a Data Store for Security Data

Technology

Transcript of Leveraging MongoDB as a Data Store for Security Data

Storage Solutions for MongoDB Analytics...MongoDB allows customers to identify key data points in extremely large petabyte-scale datasets of unstructured data. MongoDB MongoDB is an

Open Government Data and MongoDB

MongoDB Data Models Guide v3.0

Reduce MongoDB Data Size - Percona · 2 Outline • MongoDB Cluster Architecture • Advantages to Reduce Data Size • Several Cases To Reduce MongoDB Data Size • Case 1: Migrate

Building Spring Data with MongoDB

Spring Data MongoDB - Reference Documentation · The Spring Data MongoDB project applies core Spring concepts to the development of solutions using the MongoDB document style data

The Spring Data MongoDB Project

MongoDB Data Models Guide

MongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines

MongoDB for Time Series Data

Time Series Data in MongoDB - files.meetup.comfiles.meetup.com/1742411/DC Meetup - Time Series... · Time Series Data in MongoDB Senior Solutions Architect, MongoDB Inc. Dave Erickson

Introduction to MongoDB NGUYEN VO – A02213331 1. Content I. Overview II. SQL vs MongoDB III. MongoDB Data Model IV. MongoDB Queries V. Installation VI.

Mongodb + Hadoop Big Data Solyanka

Time Series Data in MongoDB - files.meetup.com Meetup - Time... · Time Series Data in MongoDB Senior Solutions Architect, MongoDB Inc. Dave Erickson #mongodb . Agenda ... Time Series

MongoDB - javascript for your data

Spring Data com MongoDB

MongoDB Foreign Data Wrapper Guide · 2020. 11. 12. · MongoDB Foreign Data Wrapper Guide, Release 5.2.8 The MongoDB Foreign Data Wrapper (mongo_fdw) is a Postgres extension that

Data Processing and Aggregation with MongoDB

Spring Data MongoDB - Reference Documentation · Spring Data MongoDB - Reference Documentation 1.5.1.RELEASE Spring Data MongoDB - Reference Documentation 3 2. Additional Help Resources

MongoDB Data Modeling - Sample Chapter