Real time analysis and visualization
description
Transcript of Real time analysis and visualization
![Page 1: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/1.jpg)
1
Real time analysis and visualizationANUBISNETWORKS LABSPTCORESEC
![Page 2: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/2.jpg)
2Agenda
Who are we?
AnubisNetworks Stream
Stream Information Processing
Adding Valuable Information to Stream Events
![Page 3: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/3.jpg)
3Who are we?
Tiago Martins AnubisNetworks @Gank_101
João Gouveia AnubisNetworks @jgouv
Tiago Henriques Centralway @Balgan
![Page 4: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/4.jpg)
4Anubis StreamForceEvents (lots and lots of events) Events are “volatile” by nature They exist only if someone is listening
Remember?:
“If a tree falls in a forest and no one is around to hear it, does it make a sound?”
![Page 5: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/5.jpg)
5Anubis StreamForce Enter security Big Data
“a brave new world”
Volume
Variety Velocity
We are here
![Page 6: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/6.jpg)
6Anubis StreamForceProblems (and ambitions) to tackle
The huge amount and variety of data to processMechanisms to share data across multiple systems,
organizations, teams, companies..Common API for dealing with all this (both from a
producer and a consumer perspective)
![Page 7: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/7.jpg)
7Anubis StreamForceEnter the security events CEP - StreamForce
High performance, scalable, Complex Event Processor (CEP) – 1 node (commodity hw) = 50k evt/second
Uses streaming technologyFollows a publish / subscriber model
![Page 8: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/8.jpg)
8Anubis StreamForceData format
Events are published in JSON formatEvents are consumed in JSON format
![Page 9: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/9.jpg)
9Anubis StreamForce Yes, we love JSON
![Page 10: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/10.jpg)
10Anubis StreamForce
Sharing Models
![Page 11: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/11.jpg)
MFE
OpenSource / MailSpike community
DashboardDashboard
Complex Event Processing
SinkholesData-theft Trojans
Real Time Feeds
Real Time Feeds
IP Reputation
Passive DNSTraps /Honeypots
![Page 12: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/12.jpg)
MFE
OpenSource / MailSpike community
DashboardDashboard
Complex Event Processing
SinkholesData-theft Trojans
Real Time Feeds
Real Time Feeds
IP Reputation
Passive DNSTraps /Honeypots
![Page 13: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/13.jpg)
13Anubis CyberFeed
Feed galore!Sinkhole data, traps, IP reputation, etc.
Bespoke feeds (create your own view)Measure, group, correlate, de-
duplicate .. High volume (usually ~6,000 events per
second, more data being added frequently
![Page 14: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/14.jpg)
MFE
OpenSource / MailSpike community
DashboardEvent navigation
Complex Event Processing
SinkholesData-theft Trojans
Real Time Feeds
Real Time Feeds
IP Reputation
Passive DNSTraps /Honeypots
![Page 15: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/15.jpg)
15Anubis CyberFeed
Apps (demo time)
![Page 16: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/16.jpg)
Stream Information Processing
Collecting events from the Stream.
Generating reports.
Real time visualization.
16
![Page 17: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/17.jpg)
Challenge ~6k events/s and at peak over 10k events/s.
Let’s focus on trojans feed (banktrojan).
Peaks @ ~4k events/s
{"_origin":"banktrojan","env":{"server_name":"anam0rph.su","remote_addr":"46.247.141.66","path_info":"\/in.php","request_method":"POST","http_user_agent":"Mozilla\/4.0"},"data":"upqchCg4slzHEexq0JyNLlaDqX40GsCoA3Out1Ah3HaVsQj45YCqGKylXf2Pv81M9JX0","seen":1379956636,"trojanfamily":"Zeus","_provider":"lab","hostn":"lab14","_ts":1379956641}
17
![Page 18: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/18.jpg)
18Challenge
![Page 19: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/19.jpg)
19Challenge
![Page 20: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/20.jpg)
Challenge Let’s use the Stream to help
Group by machine and trojan From peak ~4k/s to peak ~1k/s
Filter fields.
Geo location
We end up with{"env":{"remote_addr":"207.215.48.83"},"trojanfamily":"W32Expiro","_geo_env_remote_addr":{"country_code":"US","country_name":"United States","city":"Los Angeles","latitude":34.0067,"longitude":-118.3455,"asn":7132,"asn_name":"AS for SBIS-AS"}}
20
![Page 21: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/21.jpg)
Challenge How to process and store these events?
21
![Page 22: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/22.jpg)
22Technologies Applications
NodeJS Server-side Javascript Platform. V8 Javascript Engine. http://nodejs.org/
Why? Great for prototyping. Fast and scalable. Modules for (almost) everything.
![Page 23: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/23.jpg)
23Technologies Databases
MongoDB NoSQL Database. Stores JSON-style documents. GridFS http://www.mongodb.org/
Why? JSON from the Stream,
JSON in the database. Fast and scalable.
Redis Key-value storage. In-memory dataset. http://redis.io/
Why? Faster than MongoDB for
certain operations, like keeping track of number of infected machines.
Very fast and scalable.
![Page 24: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/24.jpg)
Data Collection 24
Storage Aggregate information
MongoDB Redis
Worker
Worker
Worker
Processor
Process real time events
Applications Collector Worker Processor
Databases MongoDB Redis
Collector
Stream
![Page 25: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/25.jpg)
Data Collection 25
Storage Aggregate information
MongoDB Redis
Worker
Worker
Worker
Processor
Process real time events
Events comes from the Stream. Collector distributes events to Workers. Workers persist event information. Processor aggregates information and stores it for statistical and
historical analysis.
Collector
Stream
![Page 26: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/26.jpg)
Data Collection 26
Storage Aggregate information
MongoDB Redis
Worker
Worker
Worker
Processor
Process real time events
MongoDB Real time information of infected machines. Historical aggregated information.
Redis Real time counters of infected machines.
Collector
Stream
![Page 27: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/27.jpg)
Data Collection - Collector 27
Collector
Old data is periodically remove, i.e. machines that don’t produce events for more than 24 hours.
Send events to Workers.Workers
Decrements counters of removed information. Send warnings
Country / ASN is no longer infected. Botnet X decreased Y % of its size.
![Page 28: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/28.jpg)
Data Collection - Worker 28
Worker
Create new entries for unseen machines. Adds information about new trojans / domains. Update the last time the machine was seen.
Process events and update the Redis counters accordingly. Needs to check MongoDB to determine if:
New entry – All counters incremented Existing entry – Increment only the counters related to
that Trojan Send warnings
Botnet X increased Y % in its size. New infections seen on Country / ASN.
![Page 29: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/29.jpg)
Data Collection - Processor
Processor
29
Processor retrieves real time counters from Redis. Information is processed by:
Botnet; ASN; Country; Botnet/Country; Botnet/ASN/Country; Total.
Persisting information to MongoDB creates a historic database of counters that can be queried and analyzed.
![Page 30: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/30.jpg)
Data Collection - MongoDB Collection for active machines in the last 24h
{"city" : "Philippine",
"country" : "PH", "region" : "N/A",
"geo" : { "lat" : 16.4499, "lng" : 120.5499 },
"created" : ISODate("2013-09-21T00:19:12.227Z "), "domains" : [
{ "domain" : "hzmksreiuojy.nl", "trojan" : "zeus", "last" : ISODate("2013-09-21T09:42:56.799Z"), "created" : ISODate("2013-09-21T00:19:12.227Z") }
], "host" : "112.202.37.72.pldt.net", "ip" : "112.202.37.72", "ip_numeric" : 1892296008, "asn" : "Philippine Long Distance Telephone Company", "asn_code" : 9299, "last" : ISODate("2013-09-21T09:42:56.799Z"), "trojan" : [ "zeus” ]}
30
![Page 31: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/31.jpg)
Data Collection - MongoDB Collection for aggregated information (the historic counters database){ "_id" : ObjectId("519c0abac1172e813c004ac3"), "0" : 744, "1" : 745, "3" : 748, "4" : 748, "5" : 746, "6" : 745,
... "10" : 745, "11" : 742, "12" : 746, "13" : 750, "14" : 753,
..."metadata" : { "country" : "CH", "date" : "2013-05-22T00:00:00+0000", "trojan" : "conficker_b", "type" : "daily" }}
31
Preallocated entries for each hour when the document is created.
If we don’t, MongoDB will keep extending the documents by adding thousands of entries every hour and it becomes very slow.
![Page 32: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/32.jpg)
Data Collection - MongoDB Collection for 24 hours
4 MongoDB Shard instances >3 Million infected machines ~2 Gb of data ~558 bytes per document.
Indexes by ip – helps inserts and updates. ip_numeric – enables queries by CIDRs. last – Faster removes for expired
machines. host – Hmm, is there any .gov? country, family, asn – Speeds MongoDB
queries and also allows faster custom queries.
Collection for aggregated information Data for 119 days (25 May to 11 July)
> 18 Million entries ~6,5 Gb of data ~366 bytes per object ~56 Mb per day
Indexes by metadata.country metadata.trojan metadata.date Metadata.asn Metadata.type,
metadata.country,metadata.date,met....... (all)
32
![Page 33: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/33.jpg)
33Data Collection - Redis Counters by Trojan / Country "cutwailbt:RO": "1256", "rbot:LA": "3", "tdss:NP": "114", "unknown4adapt:IR": "100", "unknownaff:EE": "0", "cutwail:CM": "20", "unknownhrat3:NZ": "56", "cutwailbt:PR": "191", "shylock:NO": "1", "unknownpws:BO": "3", "unknowndgaxx:CY": "77", "fbhijack:GH": "22", "pushbot:IE": "2", "carufax:US": "424“
Counters by Trojan "unknownwindcrat": "18", "tdss": "79530", "unknownsu2": "2735", "unknowndga9": "15", "unknowndga3": "17", "ircbot": "19874", "jshijack": "35570", "adware": "294341", "zeus": "1032890", "jadtre": "40557", "w32almanahe": "13435", "festi": "1412", "qakbot": "19907", "cutwailbt": "38308“
Counters by Country “BY": "11158", "NA": "314", "BW": "326", "AS": "35", "AG": "94", "GG": "43", "ID": "142648", "MQ": "194", "IQ": "16142", "TH": "105429", "MY": "35410", "MA": "15278", "BG": "15086", "PL": "27384”
![Page 34: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/34.jpg)
Data Collection - Redis Redis performance in our machine
SET: 473036.88 requests per second GET: 456412.59 requests per second INCR: 461787.12 requests per second
Time to get real time data Getting all the data from Familys/ASN/Counters to the NodeJS application and
ready to be processed in around half a second > 120 000 entries in… (very fast..)
Our current usage is ~ 3% CPU (of a 2.0 Ghz core) ~ 480 Mb of RAM
34
![Page 35: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/35.jpg)
Data Collection - API But! There is one more application.. How to easily retrieve stored data
MongoDB Rest API is a bit limited. NodeJS HTTP + MongoDB + Redis
Redis http://<host>/counters_countries ...
MongoDB http://<host>/family_country ...
Custom MongoDB Querys http://<host>/ips?f.ip_numeric=95.68.149.0/22 http://<host>/ips?f.country=PT http://<host>/ips?f.host=\bgov\b
35
API
![Page 36: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/36.jpg)
Data Collection - Limitations Grouping information by machine and trojan doesn’t allow to
study the real number of events per machine. Can be useful to get an idea of the botnet operations or how many
machines are behind a single IP (everyone is behind a router).
Slow MongoDB impacts everything Worker application needs to tolerate a slow MongoDB and discard
some information has a last resort. Beware of slow disks! Data persistence occurs every 60 seconds
(default) and can take too much time, having a real impact on performance..
>10s to persist is usually very bad, something is wrong with hard drives..
36
![Page 37: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/37.jpg)
Data Collection - Evolution Warnings
Which warnings to send? When? Thresholds?
Aggregate data by week, month, year.
Aggregate information in shorter intervals.
Data Mining algorithms applied to all the collected information.
Apply same principles to other feeds of the Stream. Spam Twitter Etc..
37
![Page 38: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/38.jpg)
Reports What’s happening in country X? What about network 192.168.0.1/24? Can send me the report of Y everyday at 7 am? Ohh!! Remember the report I asked last week? Can I get a report for ASN AnubisNetwork?
38
![Page 39: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/39.jpg)
Reports 39
HTTP API Schedule Get Edit Delete List schedules List reports
Check MongoDB for work. Generate CSV report or store the JSON Document for
later querying. Send email with link to files when report is ready.
Server
Generator
![Page 40: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/40.jpg)
Reports – MongoDB CSVs Scheduled Report{ "__v" : 0, "_id" : ObjectId("51d64e6d5e8fd0d145000008"), "active" : true, "asn_code" : "", "country" : "PT", "desc" : "Portugal Trojans", "emails" : "", "range" : "", "repeat" : true, "reports" : [ ObjectId("51d64e7037571bd24500000d"), ObjectId("51d741e8bcb161366600000c"), ObjectId("51d89367bcb161366600005f"), ObjectId("51d9e4f9bcb16136660000ca"), ObjectId("51db3678c3a15fc577000038"), ObjectId("51dc87e216eea97c20000007"), ObjectId("51ddd964a89164643b000001") ], "run_at" : ISODate("2013-07-11T22:00:00Z"), "scheduled_date" : ISODate("2013-07-05T04:41:17.067Z")}
Report{ "__v" : 0, "_id" : ObjectId("51d89367bcb161366600005f"), "date" : ISODate("2013-07-06T22:00:07.015Z"), "files" : [ ObjectId("51d89368bcb1613666000060") ], "work" : ObjectId("51d64e6d5e8fd0d145000008")}
Files Each report has an array of files that
represents the report. Each file is stored in GridFS.
40
![Page 41: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/41.jpg)
Reports – MongoDB JSONs Scheduled Report{ "__v" : 0, "_id" : ObjectId("51d64e6d5e8fd0d145000008"), "active" : true, "asn_code" : "", "country" : "PT", "desc" : "Portugal Trojans", "emails" : "", "range" : "", "repeat" : true, “snapshots" : [ ObjectId("521f761c0a45c3b00b000001"), ObjectId("521fb0848275044d420d392f"), ObjectId("52207c2f7c53a8494f010afa"), ObjectId("5221c9df4910ba3874000001"), ObjectId("522275724910ba3874001f66"), ObjectId("5223c6f24910ba3874003b7a"), ObjectId("522518734910ba3874005763")
], "run_at" : ISODate("2013-07-11T22:00:00Z"), "scheduled_date" : ISODate("2013-07-05T04:41:17.067Z")}
Snapshot{
"_id" : ObjectId("51d89367bcb161366600005f"), "date" : ISODate("2013-07-06T22:00:07.015Z"),
"work" : ObjectId("521f761c0a45c3b00b000001"), count: 123}
Results{ "machine" : { "trojan" : [ “conficker_b“ ], "ip" : "2.80.2.53", "host" : "Bl19-1-13.dsl.telepac.pt",
}, … , "metadata" : { "work" : ObjectId("521f837647b8d3ba7d000001"), "snaptshot" : ObjectId("521f837aa669d0b87d000001"), "date" : ISODate("2013-08-29T00:00:00Z") },}
41
![Page 42: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/42.jpg)
Reports – Evolution
Other reports formats.
Charts?
Other type of reports. (Not only botnets). Need to evolve Collector first.
42
![Page 43: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/43.jpg)
43Globe
How to visualize real time events from the stream? Where are the botnets located? Who’s the most infected? How many infections?
![Page 44: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/44.jpg)
44Globe – Stream
origin = banktrojan Modules
Group trojanfamily _geo_env_remote_addr.count
ry_name grouptime=5000
Geo Filter fields
trojanfamily
Geolocation _geo_env_remote_addr.l*
KPI trojanfamily _geo_env_remote_addr.countr
y_name kpilimit = 10
Stream NodeJS Browser
Request botnets from stream
![Page 45: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/45.jpg)
45Globe – NodeJSStream NodeJS Browse
r NodeJS
HTTP Get JSON from Stream.
Socket.IO Multiple protocol support (to bypass some proxys and
handle old browsers). Redis
Get real time number of infected machines.
![Page 46: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/46.jpg)
46Globe – Browser Stream NodeJS Browse
r
Browser Socket.IO Client
Real time apps. Websockets and other
types of transport.
WebGL ThreeJS Tween
jQuery
WebWorkers Runs in the background. Where to place the red dots? Calculations from geolocation
to 3D point goes here.
![Page 47: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/47.jpg)
47Globe – Evolution Some kind of HUD to get better interaction and
notifications.
Request actions by clicking in the globe. Generate report of infected in that area. Request operations in a specific that area.
Real time warnings New Infections Other types of warnings...
![Page 48: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/48.jpg)
48Adding Valuable Information to Stream Events
How to distribute workload to other machines?
Adding value to the information we already have.
![Page 49: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/49.jpg)
Minions
Typically the operations that would had value are expensive in terms of resources CPU Bandwidth
Master-slave approach that distributes work among distributed slaves we called Minions.
49
Master
Minion
Minion
Minion
Minion
![Page 50: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/50.jpg)
Minions 50
Master receives work from Requesters and store the work in MongoDB.
Minions request work. Requesters receive real time information on the work from the
Master or they can ask for work information at a later time.
Process / Storage Minions
Master MongoDB
DNS
Scan
Minion
Minion
Requesters
Minion
![Page 51: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/51.jpg)
51Minions Master has an API that allows custom Requesters to
ask for work and monitor the work.
Minion have a modular architecture Easily create a custom module.
Information received from the Minions can then be processed by the Requesters and Sent to the Stream Saved on the database Update existing database
Minion
DNS
ScanningData Minin
g
![Page 52: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/52.jpg)
52Extras...
So what else could we possibly do using the Stream? Distributed Portscanning Distributed DNS Resolutions Transmit images Transmit videos Realtime tools Data agnostic. Throw stuff at it and it will deal with it.
![Page 53: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/53.jpg)
53Extras...
So what else could we possibly do using the Stream? Distributed Portscanning Distributed DNS Resolutions Transmit images Transmit videos Realtime tools Data agnostic. Throw stuff at it and it will deal with it.
FOCUS
FOCUS
![Page 54: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/54.jpg)
54Portscanning Portscanning done right…
Its not only about your portscanner being able to throw 1 billion packets per second.
Location = reliability of scans. Distributed system for portscanning is much better. But its not
just about having it distributed. Its about optimizing what it scans.
![Page 55: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/55.jpg)
55Portscanning
![Page 56: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/56.jpg)
56Portscanning
![Page 57: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/57.jpg)
57Portscanning
![Page 58: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/58.jpg)
58Portscanning
IP Australia(intervolve)
China(ChinaVPShosting)
Russia(NQHost)
USA (Ramnode)
Portugal(Zon PT)
41.63.160.0/19 (Angola)
0 hosts up 0 hosts up 0 hosts up 0 hosts up 3 hosts up (sometimes)
5.1.96.0/21 (China)
10 hosts up 70 hosts up 40 hosts up 10 hosts up 40 hosts up
41.78.72.0/22 (Somalia)
0 hosts up 0 hosts up 0 hosts up 0 hosts up 33 hosts up
92.102.229.0/24(Russia)
20 hosts up 100 hosts up 2 hosts up 2 hosts up 150 hosts up
![Page 59: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/59.jpg)
59Portscanning problems...
Doing portscanning correctly brings along certain problems. If you are not HD Moore or Dan Kaminsky, resource wise you are gonna have a bad time
![Page 60: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/60.jpg)
60Portscanning problems...
Doing portscanning correctly brings along certain problems. If you are not HD Moore or Dan Kaminsky, resource wise you are gonna have a bad time
![Page 61: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/61.jpg)
61Portscanning problems...
Doing portscanning correctly brings along certain problems. If you are not HD Moore or Dan Kaminsky, resource wise you are gonna have a bad
time You need lots of minions in different parts of the world Doesn’t actually require an amazing CPU or RAM if you do it correctly. Storing all that data... Querying that data...
Is it possible to have a cheap, distributed portscanning system?
![Page 62: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/62.jpg)
62Portscanning problems...
Minion
![Page 63: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/63.jpg)
63Portscanning
![Page 64: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/64.jpg)
64Data….
![Page 65: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/65.jpg)
65Data
![Page 66: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/66.jpg)
66Internet status...
![Page 67: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/67.jpg)
67Internet status...
![Page 68: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/68.jpg)
68If we‘re doing it... Anyone else can.Evil side?
![Page 69: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/69.jpg)
69Anubis StreamForce Have cool ideas? Contact us
Access for Brucon participants:API Endpoint:
http://brucon.cyberfeed.net:8080/stream?key=brucon2013
Web UI Dashboard maker:http://brucon.cyberfeed.net:8080/webgui
![Page 70: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/70.jpg)
70Lol
Last minute testing
![Page 71: Real time analysis and visualization](https://reader034.fdocuments.us/reader034/viewer/2022051118/568163a0550346895dd49d60/html5/thumbnails/71.jpg)
71Questions?