Applied Detection and Analysis with Flow Data - SO Con 2014
-
Upload
chrissanders88 -
Category
Technology
-
view
334 -
download
4
description
Transcript of Applied Detection and Analysis with Flow Data - SO Con 2014
![Page 1: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/1.jpg)
Applied Detection and Analysis Using Network
Flow Data
Chris Sanders & Jason SmithSecurity Onion Conference 2014
![Page 2: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/2.jpg)
Chris Sanders
• Christian & Husband• Kentuckian and South
Carolinian• MS, GSE, et al.• Non-Profit Director• BBQ Pit Master
![Page 3: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/3.jpg)
Jason Smith
• Kentuckian• Car Aficionado• Raspberry Pi Enthusiast• Junkyard Engineer
![Page 4: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/4.jpg)
Applied Network Security Monitoring
“This book should be required reading for all intrusion analysts and those looking to develop a security monitoring program.”
“Written by analysts, for analysts.”
- Amazon Reviewers
![Page 5: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/5.jpg)
Agenda
Flow Data!•Why it’s important•How you can collect it•What you can do with it•Tool that’s can help
“[Why, How] to extend Security Onion with Flow Analysis.“
![Page 6: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/6.jpg)
The NSM Cycle
![Page 7: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/7.jpg)
Evolution of NSM Emphasis
![Page 8: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/8.jpg)
We All Want Full PCAP…
•Collection– Easy to Capture / Filter Stream Data
•Detection– Major Detection Tools are PCAP Oriented
•Analysis– Gives us Who, Where, When, and What
NSM Challenges of the Present
![Page 9: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/9.jpg)
But, It’s not Feasible for Every Goal…
•Collection– Not Scalable for Extended Retention
•Detection– Not Ideal for Hunting / Rapid Pivoting
•Analysis– Not a Great Starting Point
NSM Challenges of the Present
![Page 10: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/10.jpg)
• Often Called Flow / Session / NetFlow• Summary of Network Communications• Aggregated Record of Packets• Gives Us Who, Where, When• Based on the 5-tuple + Timing/Data Stats
Enter Flow Data
Source IP Source Port Dest IP Dest Port Protocol
192.168.5.1 48293 8.8.8.8 53 UDP
![Page 11: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/11.jpg)
• Records are Defined by Unique 5-tuples
• Data is added to the 5-tuple Record until a termination condition is met.
Building Flow Records
![Page 12: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/12.jpg)
• Natural Timeout – End of communication per protocol (ex. RST/FIN)
• Idle Timeout– No data received for 30 seconds
• Active Timeout– Thirty minute max timeout (configurable)
Flow Record Termination Conditions
![Page 13: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/13.jpg)
Full PCAP vs. Flow Data
PCAP Data Flow Data
Level of Context
![Page 14: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/14.jpg)
Full PCAP vs. Flow Data
PCAP Data Flow Data
Storage Requirements
![Page 15: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/15.jpg)
• Most Network Devices Generate it Natively• Collectors are Easy to Setup• Data Footprint is Incredibly Small• Easy for Orgs to Keep Years of Flow Data• Useful for Detection and Analysis
Flow Data Benefits
![Page 16: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/16.jpg)
Collection with Flow Data
![Page 17: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/17.jpg)
• Generation– Routers– Sensors
• Fprobe• YAF
• Multiple Types:– NetFlow (v5,v9)– IPFIX– jFlow– More…
Generating Flow Data
![Page 18: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/18.jpg)
• Popular Platforms– Argus
+ Reliable & Fast Collection- Not Well Supported/Documented
– NFDump+ Easy to Setup and Use- Not in Wide Use
– SiLK+ Exceptional Analysis Tools- More Involved Setup
Collecting Flow Data
![Page 19: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/19.jpg)
• The System for Internet-Level Knowledge• CERT NetSA Team• Two Major Components:
– Packing Suite• Collection and parsing of flow data
– Analysis Suite• Filter, display, sort, count, group, mate, and more
• Excellent Documentation & Community– https://tools.netsa.cert.org/silk/docs.html
SiLK
![Page 20: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/20.jpg)
SiLK Collection Architecture
![Page 21: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/21.jpg)
SiLK – What You Need
Flow Sources− Hardware: Routers, Switches− Software: YAF, fprobe
SiLK Server− Rwflowpack− Will also have SiLK analysis suite installed
Analyst Workstation− Access SiLK server directly − Locally mirrored database
![Page 22: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/22.jpg)
SiLK – Packing Suite Config
rwflowpack – Listens and sorts incoming flows, preparing them for the analysis suite.− --sensor-configuration
Defines listener options Defines ipblocks Defines sensor probes
− --site-config-file Matches sensor probes with a naming convention Defines class and type relationships
− --root-directory Location where all binary flat files are stored
Indexed: Type>Year>Month>Day>Hour
![Page 23: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/23.jpg)
SiLK – Analysis Suite
rwfilter - Filters through data based on conditions. rwcut - Converts flow binary data to a human readable format. rwstats - Generates statistics from flow data rwcount - Summarizes total network traffic over time
![Page 24: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/24.jpg)
SiLK Analysis – rwfilter / rwcut (1)
Display all records from the beginning the current day until the current time:rwfilter --type=all --proto=0-255 --pass=stdout | rwcut
![Page 25: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/25.jpg)
SiLK Analysis – rwfilter / rwcut (2)
Display all records of communication to or from Chinese IP addresses over a specific week to one local CIDR range: rwfilter --type=all --start-date=2014/08/01 --end-date=2014/08/07 --any-address=192.168.1.0/24 --any-cc=cn --pass=stdout | rwcut --fields=stime,sip,dip,sport,dport,type
![Page 26: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/26.jpg)
SiLK Analysis – rwstats (1)
Display statistics for the total amount of bytes transferred by protocol (top 10):rwfilter --type=all --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=proto --value=bytes
![Page 27: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/27.jpg)
SiLK Analysis – rwstats (2)
Show the top 10 sip,dip pairs for valid conversations (top 10)rwfilter --type=all --proto=0-255 --packets=4, --pass=stdout | rwstats --top --count=10 --fields=sip,dip --value=bytes
![Page 28: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/28.jpg)
SiLK Analysis – rwstats (3)
Show the top 10 outbound destination country codes by records:rwfilter --type=out,outweb --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=dcc
![Page 29: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/29.jpg)
SiLK Analysis – Zero Access Example (1)
Rwstats to discover potential victimsrwfilter --type=all --dport=16464,16465,16470,16471 --pass=stdout | rwstats --top --fields=sip --value=distinct:dcc --threshold=3
Filter down to only the potential victim machinerwfilter --type=all --start-date=2014/08/02 --end-date=2014/08/03 --saddress=192.168.106.131 --pass=ZA1.rwf
![Page 30: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/30.jpg)
Analyze the data per 10 minute buckets over the course of 24 hours to look for abnormal user data at bizarre times.rwfilter ZA1.rwf --type=all --proto=0-255 --active-time=2014/08/02:00-2014/08/03:00 --pass=stdout | rwcut --bin-size=600
SiLK Analysis – Zero Access Example (2)
![Page 31: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/31.jpg)
• Friendly Intelligence Gathering• Identify Services on the Network• Identify Normal Behaviors of Hosts• Identify “Friends and Family”
– Friends: Who a host communicates with outside the network
– Family: Who a host communicates with inside the network
Collecting Intelligence Data
![Page 32: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/32.jpg)
• Identify SSH Serversrwfilter --type=out --protocol=6 --packets=4- --ack-flag=1 --sport=22 --pass=stdout | rwcut --fields=sip
• Identify Web Serversrwfilter --type=outweb --protocol=6 --packets=4- --ack-flag=1 --sport=80,443,8080 --pass=stdout | rwcut --fields=sip
Identifying Services
![Page 33: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/33.jpg)
Identifying Friends and Family
• Identify Friendsrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dcidr=192.168.0.0/24 --fail=stdout
• Identify Familyrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dipset=local --fail=stdout
![Page 34: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/34.jpg)
DETECTION with Flow Data
![Page 35: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/35.jpg)
Flow for Detection
![Page 36: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/36.jpg)
FlowPlotter
• Generates Visualizations from Output of Flow Tools
• Useful for Detection-Oriented Statistics• Written in BASH – Flexible/Tweakable• Maintained in GitHub• Browser Independent
![Page 37: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/37.jpg)
FlowPlotter - GeoMaprwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout |
./flowplotter.sh geomap dcc bytes > geomap.html
![Page 38: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/38.jpg)
FlowPlotter – Line Chartrwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh linechart
600 bytes > linechart.html
![Page 39: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/39.jpg)
FlowPlotter - TreeMaprwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --
proto=0- --type=all --pass=stdout | ./flowplotter.sh treemap dip records > treemap.html
![Page 40: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/40.jpg)
FlowPlotter - PieChartrwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --
proto=0- --type=all --pass=stdout | ./flowplotter.sh piechart dip bytes > piechart.html
![Page 41: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/41.jpg)
FlowPlotter - Barchart/ColumnChartrwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --
proto=0- --type=all --pass=stdout | ./flowplotter.sh columnchart dip bytes > columnchart.html
![Page 42: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/42.jpg)
FlowPlotter - BubbleChartrwfilter ../Sampledata/sample.rw --type=all --proto=0-255 --pass=stdout
| ./flowplotter.sh bubblechart sip > bubblechart.html
![Page 43: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/43.jpg)
FlowPlotter - Timelinerwfilter ../Sampledata/sample.rw --proto=0- --dcc=us,-- --fail=stdout |
./flowplotter.sh timeline sip dip > timeline.html
![Page 44: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/44.jpg)
FlowPlotter - Force Directedrwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all --
pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 > forcetest.html
![Page 45: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/45.jpg)
FlowPlotter – Asset Discovery
rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout | ./flowplotter.sh assetdiscovery > assettest.html
![Page 46: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/46.jpg)
Analysis with Flow Data
![Page 47: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/47.jpg)
Flow in Analysis – PCAP Only
* Based on the First Hour of Analysis
![Page 48: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/48.jpg)
Flow in Analysis – w/ Flow Data
* Based on the First Hour of Analysis
![Page 49: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/49.jpg)
• Be Prepared to Look at a LOT of Line-Based Data
• Very Command Line Oriented• Not Welcoming to Junior-Level Analysts• Hard to Display/Interpret Data Visually
Flow – Barriers to Entry
![Page 50: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/50.jpg)
SiLK Data Output
![Page 51: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/51.jpg)
![Page 52: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/52.jpg)
• Flow Basic Analysis Tool• Graphical Front-End to SiLK• Easy Two-Step Install on SiLK Capable Box
– Install Locally to SiLK Box– Install Remotely and Interact via SSH w/ Keys
• Rapid Pivoting Between Data• Graphing Ability
![Page 53: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/53.jpg)
Conclusion
• Flow Data is Underused and Underrated• Easy to Collect, Enhances Detection & Analysis• Minimal Barriers to Entry
– SiLK (Easy to Install on SO)– Argus (Already Installed on SO)– Bro (Already Installed on SO)
![Page 54: Applied Detection and Analysis with Flow Data - SO Con 2014](https://reader033.fdocuments.us/reader033/viewer/2022061118/5469eaa0af7959653c8b651e/html5/thumbnails/54.jpg)
Thanks Folks!• Questions?
– Chris Sanders: [email protected]– Jason Smith: [email protected]
• Blog/Book– http://www.appliednsm.com
• FlowPlotter– http://www.github.com/automayt/FlowPlotter
• FlowBAT – Release in October!