A streaming architecture for Cyber Security - Apache Metron
-
Upload
simon-elliston-ball -
Category
Technology
-
view
509 -
download
4
Transcript of A streaming architecture for Cyber Security - Apache Metron
A streaming architecture for Cyber Security
with NiFi, Hadoop, Storm and Metron
Simon Elliston Ball
• Product Manager
• Data Scientist
• Elephant herder
• @sireb
IoT: Mirai
Reports of 1.2 Tbps
500,000 devices at peak
DDoS attacks on Dyn DNS services
Drowning in Data
The value of real time
Data in Motion: why wait until it’s at rest?
Correct context: the world moved on
Better data = analyst efficiency
Fully enriched data
Real context
Consistency
= faster triage and better coverage
Network Level Taps
Data Sources and Aggregation
Open standards for data models = more productive data scientists + shareable models
Business level data sources link security to real business risk.
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Se
rvices an
d In
tegratio
n Laye
r
ModulesReal-time ProcessingCyber Security Engine
TelemetryParsers
Apache Metron: a framework for Big Data Driven cyber security
Tele
metry In
gest B
uffe
r
TelemetryData Collectors
Real-timeEnrich / ThreatIntel Streams
PerformanceNetwork
IngestProbes
/ OtherMachine Generated Logs(AD, App / Web Server,
firewall, VPN, etc.)
Security Endpoint Devices (Fireye, Palo Alto,
BlueCoat, etc.)
Network Data(PCAP, Netflow, Bro, etc.)
IDS(Suricata, Snort, etc.)
Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)
TelemetryData Sources
Data Vault
Real-Time Search
Evidentiary Store
Threat Intelligence Platform
Model as a Service
Community Models
Data Science Workbench
PCAP Forensics
Threat IntelligenceEnrichment
Indexers and WriterProfiler Alert Triage
Cyber SecurityStream Processing Pipeline
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment is the key to context
Human Resources Database
Metron Data
Metron Data
App
Logs
Active Directory
Network Traffic
Logs
IoT
Asset Database
Geo, Threat, Traditional Security
data sources
Business Risk DataMetron
Data
Standard, Consistent Data Format
Streaming enrichment
Batch enrichment
Fully Enriched data ready for analysis
Wide variety of real-time and batch
sources
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
But time is context too… profiling by time
t = 1 t = 2 t = 3 t = n
Wide range of algorithms including:
HyperLogLogPlus
Bloom filters
T-digests
Statistical Baselining
Hashing functions
Outlier detection
GeoHashing over time
Locality Sensitive HashingApprox.
Data SketchApprox.
Data SketchApprox.
Data SketchApprox.
Data Sketch
Combined Baseline
Statistic
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stellar: Excel functions for Cyber security
{"profile": "auth_distribution","foreach": "'global'","onlyif": "profile == 'attempts_by_user'","init": {
"s": "STATS_INIT()"},"update": {
"s": "STATS_ADD(s, total_count)"},"result": "s"
}
Building a Profile Using a Profile
window := PROFILE_WINDOW('...')profile := PROFILE_GET('attempts_by_user', user, window)distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile))distribution_profile := PROFILE_GET('auth_distribution', 'global', window)stats := STATS_MERGE(distribution_profile)distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5)distinct_auth_attempts_stddev := STATS_SD(stats)
• Simple• Expression based• Function composition• Boolean operators• In-stream
Thank you!
Apache Metron: http://metron.apache.org
Twitter: @sireb