SHARKFEST ‘10 | Stanford University | June 14–17, 2010

The Shark Distributed MonitoringSystem: Distributing Wireshark DeepPacket Analysis to LAN/WANInfrastructuresJune 16, 2010

Loris DegioanniCTO | CACE Technologies

SHARKFEST ‘10Stanford UniversityJune 14-17, 2010

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Quotes from the conference

“The fundamentals are the same, but the scale of network troubleshooting is much bigger today.”

Hansang Bae, Citi

“One of the main reasons why you need to capture is to get the smoking gun.”

Tim Chung, Google

“No matter what, any company needs to capture all its network traffic. It’s the only final proof when something goes wrong. What’s interesting is how the captured data is used.”

Tim Belcher, NetWitness

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Deployment Overview

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Components

• Probe– 24/7 packet collection– Live metrics– Watches/Alerts– Support for multiple consoles– Ajax control interface– Basic HTML interface for capture settings– Sold as

• Appliance 1G/10G• Kit (1G/10G card + software)• Software only (Q4)

• Console– Windows .net application– Charting– Reporting– Send to wireshark

• Controller (Q4)– Centralized user and license management– Centralized way of pushing views and watches– Centralized alert/event collection and correlation– Automatic reporting– Automatic search for specific IPs/ports/… across multiple probes

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Appliances

• 8 TB Storage• 3 Gbps sustained to disk

• 4 TB Storage• 1 Gbps sustained to disk

• 16 TB Storage• 7 Gbps sustained to disk

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Kit

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Announcing 10 Gbit Appliances and Kit

• Full rate Capture and analysis• 7+ Gbps sustained to disk

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Collection: Capture jobs

• Capture and rotate the packets coming from the network• Uses a dedicated file system on the RAID array• Can run multiple capture ports at the same time.

– E.g. Different capture ports

• A single capture job can capture from multiple ports

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Now what?

I can capture a lot of packets. Now what?

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Indexing

• Writing packets uses a lot of the disk bandwidth• Even if I stop capturing, since I was writing at full speed,

reading the data is going to take around the same time of writing it– Read needs to be localized– I need high level visibility to reach the point I need

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Indexing

• Happens while capturing, you can enable it when you create a capture job

• Contains summary of the network traffic– Volume, talkers and protocol information– “Netflow on steroids”

• Designed to be extremely efficient in terms of disk usage• Coordinated with the packet store

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Packet capture and indexing architecture

Capture threadMoves packets from the live interface into the memory buffer.

Memory buffer

Write-to-disk threadMoves packets from the memory buffer into the packet storage system. Create time indices for fast packet retrieval.

RAID level 0Packet Storage

System

Live network traffic

Content Indexing ThreadCaptures packets off the live interface and creates an index that speeds up “index-friendly” Views.

Index Storage

Pilot Live Views ThreadsPerform live network analysis, including trending and alerting.

View SQL DB

Pilot Offline Views ThreadsPerform retrospective analysis on stored packets.

View SQL DB

Pilot Indexed Views ThreadsReturn summary information about terabytes of traffic in a matter of seconds.

View SQL DB