Defeating Large Scale Attacks:Defeating Large Scale Attacks:Technology and Strategies for Technology and Strategies for Global Network MonitoringGlobal Network Monitoring

The NetViewer Experiment

PAVG in collaboration with Networking Systems

R. Kamath, E. Jang, D. Luckham

2

Project GoalsProject Goals

Detect system misuse on a global levelUser re-configurable and flexibleHierarchical organization of monitorsCorrelation of distributed monitorsMonitor activity from diverse sourcesMonitor at multiple levels of

abstraction

3

Stanford NetViewer Experiment Stanford NetViewer Experiment

Uses Stanford Rapide Toolset Uses Complex Event Processing

technology Uses Talarian’s SmartSocketsTM

middleware for distributed processing

Http://pavg.stanford.edu/rapide

Http://pavg.stanford.edu/cep

FOR MORE INFO...

4

NetViewer Experiment setupNetViewer Experiment setup

NetFlow FlowCollector

Log Files

CiscoNetFlow

FlowCollector

Filter

Map

Map

IntrusionMonitor

CEPLogger

FlowEfficiency

Monitor

Complex EventProcessing

Pass-ThroughMonitor

Monitor Views

5

SUNet Campus NetworkSUNet Campus Network

Undergrad Education

BusinessSchool

AdminHost 1

ComputerCenter 1

ComputerCenter 2

AdminHost 2

StanfordHospital

Grad. Education

RedundancyGateway

RedundancyGateway

RedundancyGateway

RedundancyGateway

CoreGateway

CoreGateway

Internet

Internet

To FlowCollector

6

Complex Event ProcessingComplex Event Processing

Accept network ‘events’ from any source– CISCO NetFlow FlowCollector, tcpdump

Correlates events based on content and temporal relationship between events

Event Processing Agents (EPAs) connected in an Event Processing Network (EPNs)

Both post-mortem and real-time processing

7

Event Processing Agents (EPAs)Event Processing Agents (EPAs)-- Loggers and Filters-- Loggers and Filters Loggers

– Convert external data into events

– E.g. CISCO FlowCollector logs to events

Filters– Select a subset of events based on pattern

– E.g. Only connections from Stanford hosts

8

EPAs-- Maps and ViewersEPAs-- Maps and Viewers Maps

– Search for patterns in input events– Generate appropriate output events– E.g. look for IP scans and generate alarms

Viewers– Graphical display of data in events– Tables, Bar Graphs

9

RapNet User interfaceRapNet User interface

RapNet– Graphical Interface to NetViewer tool– Easy access to EPA and EPN library– Easy re-configuration of EPAs– Easy modification of EPNs– Construct new EPNs using EPAs

10

NetViewer running under NetViewer running under RapNetRapNet

11

Hierarchical monitoringHierarchical monitoring Two types of hierarchy

– Abstraction hierarchy• NetViewer monitors data at different abstraction levels

– Topological hierarchy • NetViewers at different locations

NetViewers at different levels communicate using SmartSockets middleware General case: arbitrary network of monitors

12

Network Abstraction HierarchyNetwork Abstraction Hierarchy Application layer

– Host-based monitoring– Data exchanged by SMTP, TELNET, FTP, HTTP protocols

Transport layer– Data exchanged by TCP/IP suite of protocols

Network layer– Router-based monitoring– IP and UDP packets

13

Topological Hierarchy -- Topological Hierarchy -- multiple gateways examplemultiple gateways example Distributed processing of data Each NetViewer at level 1 monitors data from a different gateway Results (e.g. top 10 IPs) from level 1 NetViewers sent to level2

NetViewers Level 2 NetViewers correlate the results of level 1 NetViewers

– E.g. compute top 10 IPs over all gateways

14

Distributed monitoring on SUNetDistributed monitoring on SUNet

Admin host

Core gateway

Admin host

Admin host

Press gateway

SmartSockets over SUNet

Sender runningNetViewer 1

Sender runningNetViewer 2

Receiver runningNetViewer 3

15

Current Status -- EPAsCurrent Status -- EPAs

Library of Event Processing Agents (EPAs)– Traffic categories

• Web, Mail, DNS, ftp …

– Scan Detectors• IP scan, Port scan

– Policy violation detectors• Access to restricted hosts• Access to restricted ports on hosts

– Traffic event filters • Web, Mail, Hosts, Networks

16

Current Status -- EPNsCurrent Status -- EPNs

Library of Viewers– Tables– Bar graphs– Pie charts

Library of Event Processing Networks (EPNs)– Network of EPAs– Graphical viewers to display results

17

Research DirectionsResearch Directions

Hierarchical monitoring– Data sources from different layers– Correlation of results from multiple NetViewers

Accept more input formats Distributed processing

– Assign individual EPAs within a NetViewer to run on different machines

Expand EPA library– Work on mail spam detection

18

Experiment results on SUNetExperiment results on SUNet NetViewer used to process router logs

– Real-time performance of about 1000 log records/sec

Generated traffic statistics– Top IPs by packets or bytes– Classification of traffic into categories such as internal/external, web/mail/DNS etc.

Intrusion detection– Detected IP and port scans– Well-known attack signature e.g. finger attack

19

Related projects -- CIDFRelated projects -- CIDF

Correlates information from multiple intrusion detectors– Reduces false alarms– Prioritizes network warnings

Part of the DARPA Common Intrusion Detection Framework (CIDF) – Multiple intrusion detectors in cyber battlefield

FOR MORE INFO...

Http://seclab.cs.ucdavis.edu/cidf

20

Overview of the CIDF projectOverview of the CIDF project

GoalExperiment with semantic interoperability of different

components in CIDF

Groups InvolvedGroup A: produces GIDOs, questions, detailed English

description of the events, and the answers to the questions.

Group B: gets 10 scenarios and produces 10 GIDOs describing the scenarios.

Group C: gets the questions and high level scenarios from B and builds the code. Then, gets 10 GIDOs and produces text answers to the questions - Stanford belongs to group C.

21

Processing GIDOs with CEP agents

Make each GIDO an event Use (and fix) our existing cidfLogger Separate event processing agent called “Qagent” Provides flexible way of handling GIDOs

CIDFLoggerBuilds events

InputGIDO

CMEventPoints to C++ GIDO tree

QuestionTarget ID

DescriptionSearch Pattern

Question AgentProcess the C++ GIDO

tree with Question

Answerto user

22

QagentQagent

Finds an answer from a given GIDO and a query pattern.

Qagent traverses the tree to find all the possible paths that can lead to the answer.

The question is fed to the program as a text file with two sections:– The input file may contain a text description– Patterns to be searched from the tree.

The pattern lines are preceded with “@question:”

Implemented in C++ (I.e. not map language)– Easier tree traversal– File input

23

Pattern LanguagePattern Language

Lists of SID separated by comma. Answer is the subtree after the last SIDAttack,AttackSpecifics,IPV4Address

“#true” or “#false” to get the sibling SID rather than child SID of the last SID for the answer.ByMeansOf,Attack#true

‘^’ to indicate that the SID is one of the base SID that applies to all other parts of the pattern^And,^Copy,Outcome,ReturnCode?

success=FileSource,FileName

24

ExamplesExamples

Event1

Brief description:

This is an attack that began on Monday, May 24, at 12:44. What is the certainty of this attack?

@question:

Attack,Certainty

( Attack

( Initiator

( IPV4Address 134.52.160.76 )

)

( Target

( IPV4Address 134.52.160.114 )

)

( AttackSpecifics

( Certainty 100 )

( Severity 50 )

( AttackID 000000020000000f )

)

( When

( BeginTime Mon May 24 12:44:17 1999 PDT )

( EndTime Mon May 24 12:44:18 1999 PDT )

)

)

25

Team MembersTeam Members

Rajesh Kamath (rkamath@pavg) David Luckham (dcl@pavg) Eunhei Jang (ejang@pavg) John Kenney (jjk@pavg) James Vera (vera@pavg)