Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform
-
Upload
carolyn-duby -
Category
Data & Analytics
-
view
47 -
download
0
Transcript of Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform
![Page 1: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/1.jpg)
Apache MetronMeetup
Carolyn DubySolutions Engineer @ Hortonworks
Apache Metron Subject Matter Expert
![Page 2: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/2.jpg)
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Metron Architecture
• Telemetry Parsing
• Enrichment
• Threat Intelligence
• Alert Triage
• Index and Write to Storage
• Getting Started
Agenda
![Page 3: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/3.jpg)
The Good GuysSecurity
Practitioner
I have too many tools I need to learn
I don’t have a centralized view of my data
My tools are too expensive
I can’t find enough talent
I can’t keep relying on static rules
I need to discover bad stuff quicker
Most of my alerts are false positives
I have too many manual tasks
SOC Manager
Threat landscape too dynamic
More assets/users to manage
Attack surface increases
Legacy techniques don’t work anymore
Metron will make it easier and faster to findthe real issues I need to act on
Metron is a more cost effective way for my team to deal with the fast moving threat landscape
![Page 4: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/4.jpg)
The Bad GuysAdvancedPersistent
Threat
ScriptKiddie
My techniques are predictable and known
My attack vectors are also known
You are not the only person I’ve attacked
I brag about what I did or will do
I set off a large number of alerts
I fumble around a lot
I am very unique in a way I do things
I live on your network for about 300 days
I know what I am after and I look for it, slowly
Your rules will not detect me, I am too smart
I impersonate a legitimate user, but I don’t act like one
Metron can take everything that is known about me and check for it in real time
Metron can model historical behavior of whoever I am impersonating and flag me as I try to deviate
![Page 5: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/5.jpg)
Problems With Existing ToolsSecurity
InformationManagement
System
I am prohibitively expensive
I have vendor lock-in
I can’t deal with big data
I am not open
I am not extensible enough
LegacyPoint Tools
I was built for 1995
I am super specialized
I don’t scale horizontally
I have a proprietary format
You need a PhD to operate me
BehavioralAnalytics
Tools
I am mostly vapor ware
I was built by a small startup
I was modeled after a data set from 1999
I spam you with false positives
![Page 6: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/6.jpg)
Apache Metron Vision
“Apache Metron is a Security Data Analytics Platform (SDAP). As a next
generation security analytics framework, it is designed to
consume and monitor network traffic and machine data within an
enterprise. Apache Metron is extensible and is designed to work at a massive scale. It is not a SIEM but rather the next evolution of a
SIEM.”
Apache Metron provides the following capabilities: Extensible ingest to monitor any telemetry source
Extensible enrichment framework for any telemetry stream
Hadoop-backed storage for telemetry stream with a customizable retention time for cost effective archive
Automated real-time index for telemetry streams enabling real-time search
Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive
ODBC/JDBC compatibility and integration with existing analytics tools
![Page 7: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/7.jpg)
Use Case Setup• On 4/10, a user named Ethan V at Company Foo submits a security ticket complaining about a
potential Phishing Email. • Details provided by the Ethan V in the ticket
• The email states that a signature is required for a new Docu-Sign document for a new Stock Option grant for granted to Ethan from internal Finance employee Sonja Lar
• There is a link in the email to the Docu-Sign Document• Ethan clicks on the link, and login appears• Ethan enters his SSO credentials and submits• On submission, nothing happens• Ethan calls Sonja but Sonja states she didn’t send an email• Ethan is worried and then files help desk security ticket
• A security ticket is created and assigned to the SOC Team• A SOC analyst James picks up the case to investigate it.
![Page 8: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/8.jpg)
Systems Accessed for Threat Scope
Systems Accessed for Forensics
Systems Accessed for Investigation/Context
SIEM
“Scope of Threat”Workflow Steps
• Step 6: Searches SIEM for Fireye and IronPort email events associated with Sonja. The SIEM doesn’t have that info
• Step 6 Result: Need to log into Fireye and IronPort
• Step 7: Log into Fireye Email Threat Prevention Cloud & IronPort to find all emails sent from Sonja from that malicious IP
• Step 7 Result: Have a list of all users that the Phishing email was sent to. Can reset the password for all those users
Maxmind (IP Geo
DB)
AD (Identity Mgmt.)
Asset Mgmt.
Inventory
Soltra (Threat Intel)
Story Unfolding• Step 1 Insight: Anomalous
Event – Corp Gmail was decommissioned on behalf of exchange months back and only few users are currently using it
• Step 2 Insight: Not possible for the same user be logging in from Ireland & Southern Cali at the same time.
• Step 3 Insight: Unauthorized access is occurring from Ireland
• Step 4 Insight: Seems like Sonja is in Southern Cali but someone else pretending to be her is logging in from unidentified Asset
• Step 5 Insight: Sonja’s account has been compromised. Shut it down and Ethan’s credentials have been reset. But what others users are affected like Ethan?
• Step 6 Insight: SIEM doesn’t have all the fireye email events I need to determine scope
• Step 7 Insight: Understand the scope of the threat and can can contain it.
“Forensics”Workflow Steps
• Step 8: Logs into Cisco IronPort to determine when the attacker first compromised Sonja’s Gmail account
• Step 8 Result: On 3/26, a user from Ireleand logged into Sony’s Corp Gmail Account
• Step 8 Insight: Understands when Sonja’s Gmail Account was first compromised
• Step 9: Logs into Intermedia, an email archive system, to understand how the account was compromised
• Step 9 Result: Sees a set of emails where the attacker spoofed someone else email address “warmed up’ her with a few emails and then sent an email with an link that Sonja clicked on which stole her credentials from her chain
• Step 9 Insight: Understand how Sonja’s account got compromised
Systems Accessed for Remediation
Exchange (Primary
Email Service)
Corp Gmail (Secondary
Email Service)
AD & SSO(Identity
Provider & SSO)
Search
FireEye (Email Cloud
Security )
Cisco IronPort(Email
On-Premise Security )
Intermedia (Email Archive)
![Page 9: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/9.jpg)
Do Investigation, Find Scope and Perform Forensics Using only Metron
Systems Accessed for Remediation
Exchange (Primary
Email Service)
Corp Gmail (Secondary
Email Service)
AD & OKTA(Identity
Provider & SSO)
Maxmind (IP Geo
DB)
AD (Identity Mgmt.)
Asset Mgmt.
Inventory
Soltra (Threat Intel)
Systems Accessed for Investigation/Context
Systems Accessed to
Determine Scope
FireEye (Email
Cloud Security )
Cisco IronPort(Email
On-Premise Security )
Intermedia (Email Archive)
Systems Accessed for
Forensics
![Page 10: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/10.jpg)
Challenges that Apache Metron Solves
60%: Percent of breaches that happened in minutes
8 months: Average time an advanced security breach goes unnoticed
$400 million in estimated financial loss in 2015
70%-90%: Percentage of malware in breach unique to organization
2015 Verizon Data Breach Investigations Report
• Too many manual steps in different tools makes investigations slow and expensive
• Too expensive to keep data for enough time to understand history
• Too expensive to collect all the desired data to understand context
• Not sure if can detect a targeted event.• Too many events to review in timely manner• Not enough staff to review events in a timely
manner• Too long to detect breach• Hackers getting more sophisticated
![Page 11: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/11.jpg)
Why Metron? SOC Analyst Perspective
Looking through alerts25%
Collecting contextual data25%
Formulating a Hypothesis5%
Investigate20%
Remediate15%
Update Work-flow5%
Wrte Report5%
Analyst workflow• Alerts Relevancy Engine• Smarter ML alerts• Centralized Alerts Console• Enriched with threat intel data
• Fully enriched messages• Single pane of glass UI• Centralized real-time search• All logs in one place
• Granular access to PCAP• Replay old PCAP against new signatures• Tag behavior for modelling by data scientists• Raw messages used as evidentiary store• Mine investigation history• Asset inventory as an enrichment• User identity as an enrichment
• Workflow engine• Ticket clustering
Everything you need to know in one place
![Page 12: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/12.jpg)
Why Metron? Data Scientist Perspective
Formulating a Hypothesis5%
Finding Data20%
Cleaning Data20%
Munging Data20%
Visualizing Data20%
Modelling Data10%
Validating Model5%
Data Science Workflow• All my data is in the same place• Data exposed through a variety of APIs• Standard Access Control Policies• Quickly see what I have
• Metron normalizes objects• Partial schema validation on ingest• Tagging on ingest
• Automatic data enrichment• Automatic application of class labels• Common Metron Objects• Massively parallel computation framework
• Reusable Zeppelin Dashboards• Real-time search + UI• Integration with Python/R• Integration with analytics tools
Reducing time from hypothesis to model
![Page 13: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/13.jpg)
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Metron Architecture
• Telemetry Parsing
• Enrichment
• Threat Intelligence
• Alert Triage
• Index and Write to Storage
• Getting Started
Agenda
![Page 14: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/14.jpg)
Metron Architecture
![Page 15: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/15.jpg)
Telemetry Parsing
Accept logsNormalize log formats to common Metron event formatVerifies incoming data
Telemetry Parsing
Enrichment
Threat Intel
Alert Triage
Index & Write
Metron Stream Processing Pipeline
![Page 16: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/16.jpg)
Log format to Metron Message Conversion
{"full_hostname":"www.aliexpress.com","code":200,"method":"GET","url":"http:\/\/www.aliexpress.com\/af\/shoes.html?","source.type":"squid","elapsed":832,"ip_dst_addr":"104.116.248.248","original_string":"1475518070.281 832 127.0.0.1 TCP_MISS\/200 448176 GET http:\/\/www.aliexpress.com\/af\/shoes.html? - DIRECT\/104.116.248.248 text\/html","bytes":448176,"domain_without_subdomains":"aliexpress.com","action":"TCP_MISS","ip_src_addr":"127.0.0.1","timestamp":1475518070281}
1475518070.281 832 127.0.0.1 TCP_MISS\/200 448176 GET http:\/\/www.aliexpress.com\/af\/shoes.html? - DIRECT\/104.116.248.248 text\/html
ORIGINAL LOG LINE
METRON JSON MESSAGE
![Page 17: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/17.jpg)
Topic A
Parser Topology ASensor
ANative Format
ApacheKafka
Apache StormEnriched
Metron JSON
Parsing and Normalizing Topology
• Each Telemetry source has:• Kafka topic with original event content• Storm Topology to normalize into common Metron event format
• All telemetry sources feed into single enrichment topic
![Page 18: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/18.jpg)
Telemetry Parsing Storm Topology
Parser Name enrichment
Spout
Bolt
![Page 19: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/19.jpg)
Telemetry Parser Implementation Options
• General Purpose Parsers• Easy to create – no programming• Grok
• Regular expression based parser extracts Metron event values• CSV Parser
• Maps CSV columns to Metron events• Java
• High performance for high throughput sources• Complex formats not easily expressed as Regex• Java class implements MessageParser interface
![Page 20: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/20.jpg)
Sensor A
Sensor B
Sensor N
Topic A
Topic B
Topic (N)
ApacheKafka
PCAPPCAP Probe
Physical Architecture
ParseTopology A
ParserTopology B
ParserTopology N
ApacheStorm
Native Format
Native Format
Native Format
PCAP on HDFS Metron PCAP Service
PCAP Topology
Enrich
Normalized Metron Format Enrichment/
Threat IntelTopology
Out to Index + HDFS
![Page 21: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/21.jpg)
Enrichment
Add extra information to parsed eventAdd context to event to save Security Analyst timeScore event for triage
Telemetry Parsing
Enrichment
Threat Intel
Alert Triage
Index & Write
Metron Stream Processing Pipeline
![Page 22: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/22.jpg)
{"adapter.threatinteladapter.end.ts":"1475595978069","full_hostname":"www.aliexpress.com","code":200,"enrichmentsplitterbolt.splitter.end.ts":"1475595604032","enrichments.geo.ip_dst_addr.city":"Cambridge","enrichments.geo.ip_dst_addr.latitude":"42.3626","enrichmentsplitterbolt.splitter.begin.ts":"1475595604032","enrichments.geo.ip_dst_addr.country":"US","enrichments.geo.ip_dst_addr.locID":"1379","adapter.geoadapter.begin.ts":"1475595604033","enrichments.geo.ip_dst_addr.postalCode":"02142","elapsed":832,"ip_dst_addr":"104.116.248.248”….}
{"full_hostname":"www.aliexpress.com","code":200,"method":"GET","url":"http:\/\/www.aliexpress.com\/af\/shoes.html?","source.type":"squid","elapsed":832,"ip_dst_addr":"104.116.248.248","original_string":"1475518070.281 832 127.0.0.1 TCP_MISS\/200 448176 GET http:\/\/www.aliexpress.com\/af\/shoes.html? - DIRECT\/104.116.248.248 text\/html","bytes":448176,"domain_without_subdomains":"aliexpress.com","action":"TCP_MISS","ip_src_addr":"127.0.0.1","timestamp":1475518070281}
SQUID PARSER MESSAGE
ENRICHED SQUID MESSAGE
![Page 23: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/23.jpg)
Enrichment Topologyenrichments indexing
![Page 24: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/24.jpg)
Enrichment Options
• Geo• Add geo location information for ips (latitude, longitude, city, country, etc)
• Host• Add information from known hosts configuration
• Hbase• Threat intelligence information
• Stellar• Apply Stellar Expressions to event• Flexibility and extensibility
![Page 25: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/25.jpg)
Stellar Enrichments
• DSL for simple computations and transformations on message variables• Capabilities
• Reference event field• Boolean: and, or, not• Real/Integer Arithmetic: *, /, + , -,• Comparison: <, > ,<= ,>= • If else: if var1 < 10 then 'less than 10' else '10 or more’• Check field exists: exists• Functions: MAP_GET, SPLIT, STARTS_WITH, etc
• Documentation• https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common
![Page 26: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/26.jpg)
Enrichment Config File[vagrant@node1 ~]$ cat /usr/metron/0.2.0BETA/config/zookeeper/enrichments/squid.json
"index": "squid", "batchSize": 5, "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "stellar" : { "config" : { "host_info" : { "top_level_domain" : "DOMAIN_TO_TLD(full_hostname)" } } } } },
![Page 27: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/27.jpg)
Event with top_level_domain Stellar Enrichment and geo enrichment
{"adapter.threatinteladapter.end.ts":"1475617327962","full_hostname":"www.aliexpress.com","code":200,"enrichmentsplitterbolt.splitter.end.ts":"1475617327621","top_level_domain":"com","enrichments.geo.ip_dst_addr.city":"Cambridge” …..}
![Page 28: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/28.jpg)
Threat Intelligence
• Threat Indicators• Malicious domain watchlist• Malicious ip watchlist• MD5 signatures
• Triaging• Structured Threat Information eXpression (STIX)
• Threat Intelligence in machine format• May be exchanged by TAXII
• Trusted Automated eXchange of Indicator Information (TAXII)• Describes how TI is exchanged• Automated standard exchange interface of threat intelligence
![Page 29: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/29.jpg)
Enrichment - Threat Intelligence enrichments indexing
![Page 30: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/30.jpg)
Ingesting Threat Intelligence
Threat IntelFeed
Feed Replicator
Taxii Loader
TaxxiRecords
TaxxiRecords
Metron Threat Intelligence
![Page 31: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/31.jpg)
Accessing Threat Intelligence
"enrichment" : { "fieldMap" : { "stellar" : { "config" : { "whois_info" : "ENRICHMENT_GET('whois', domain_without_subdomains, 'enrichment', 't')" } } },
ENRICHMENT_GET(enrichment_type, key, hbase_table, column_family)
![Page 32: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/32.jpg)
Scoring Event
If alert = true, then event is a threatCalculate one or more risk scoresAggregate all scores to get event score
SUM, MEAN, MAX, etc
![Page 33: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/33.jpg)
Scoring Configuration"threatIntel" : { "fieldMap" : { "stellar" : { "config" : { "is_alert" : "whois_info.home_country != 'US'" } } }, "fieldToTypeMap" : { }, "config" : { }, "triageConfig" : { "riskLevelRules" : { "whois_info.home_country != 'US' && IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')" : 50.0, "IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')" : 20.0, "whois_info.home_country != 'US'" : 10.0 }, "aggregator" : "MAX", "aggregationConfig" : { } }
![Page 34: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/34.jpg)
Model as a Service
• Security Analysis Models applied during enrichment and threat intelligence• REST microservices implementing a specified interface• Machine learning or other model
• Train model with event history stored in Hadoop• Register with discovery service • Referenced in Stellar enrichments
• MAAS_GET_ENDPOINT• MAAS_MODEL_APPLY
• System load balances across instances• More Information
https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-maas-service
![Page 35: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/35.jpg)
Model as a Service : Architecture
![Page 36: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/36.jpg)
Indexing and Writing• Store events for future reference• Forensics• Training machine learning models• Reprocess with new threat indicators
Telemetry Parsing
Enrichment
Threat Intel
Alert Triage
Index & Write
Metron Stream Processing Pipeline
![Page 37: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/37.jpg)
Indexing Architecture
indexing
![Page 38: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/38.jpg)
Indexing
• Elastic Search or Solr• Store in HDFS and/or Hive
![Page 39: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/39.jpg)
39
Event Analysis and Machine Learning with Spark and Zeppelin
![Page 40: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/40.jpg)
Getting Started
• Apache Metron Site– http://metron.incubator.apache.org/
• Ask Questions on Hortonworks Community Connection– https://community.hortonworks.com
• Source Code– https://github.com/apache/incubator-metron
• Deploy a quick start cluster– https://github.com/apache/incubator-metron/tree/master/metron
-deployment/vagrant/quick-dev-platform
![Page 41: Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Platform](https://reader035.fdocuments.us/reader035/viewer/2022062503/587c05e31a28ab03768b466b/html5/thumbnails/41.jpg)
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You