Network Security Incident Analysis System for Detecting ... · Network Security Incident Analysis...

Network Security Incident Analysis System for Detecting Large-scale Internet Attacks

Dr. Kenji RikitakeSecurity Advancement Group

NICT, JapanSeptember 6, 2005

6-SEP-2005 APEC-OECD Joint WG 2

Our goals

• Collaborative monitoring, centralized network security incident analysis and handling among Japanese Internet Service Providers (ISPs), including:– real-time analysis for early-warning trends– in-depth analysis for detecting new threats– recommendation to the ISPs and users

• Protecting National IT infrastructure


Our partners

• Telecom-ISAC Japan– Wide-area monitoring with probes on ISPs– Incident handling with contingency plans– Clearing house of incident info for ISPs

• Internet Security research communities– Academic network administrators– Virus and malware analysis experts– Datamining and statistics experts


Our project and Telecom-ISACISPs

probes

probes

probes

Honeypot networks

Blackhole networks

Academic network probes

Traffic statistics from partner ISPs and networks

Incident Analysis Center (operated by NICT)

Real-time analysissystem (of primaryshort-term statisticanalysis and incidentdetection)

Integrated databasefor analyzed dataand incidents

In-depth analysissystem (for long-termand detailed analysiswith human experts)

Analysis experts

Telecom-ISAC Japan Operation Center

Incident-informationdissemination via Web

Wide-area monitoringsystem for the ISPs

Incident handlingsystem

Operators

Reports to the gov’t andgeneral users

Recommendations tothe member ISPs

Partner networksand sensor systems


Roles of our analysis center

• Real-time monitoring – from various kinds of network providers– from various types of information sources– for detecting precursors ASAP

• Real-time (automated) analysis• In-depth analysis (with the experts)• Archiving events for future analyses


Required functions (1/2)

• Flow control and synchronization– of different types of monitored data– of different time resolutions and frames

• Parallel analysis of multiple algorithms– for finding out clues of new incident trends

• such as virus or DDoS attack breakouts

• Visualization by multiple methods– for helping the experts to find anomalies


Required functions (2/2)

• Large-scale incident database storage– for archiving massive (tera-to-petabyte)

amount of incident-related data– for fast retrieval by the experts and the in-

depth analyzing tools– for storing non-realtime large statistic data

• Workbench for in-depth analysis– behavioral analysis of quarantined viruses


Configuration schematics ofthe Incident Analysis System

Large-scaleIncidentDatabase

Flow Controland

Synchronization

OfflineMonitored

Data

OnlineMonitored

Data

Real-time Analysis

Process A

Real-time Analysis

Process B

Monitoring ProcessIncident Analysis Experts

In-depth Analysis

Process X

In-depth Analysis

Process Y

Output for visualization, reporting, and recommendation


Monitoring networks and the probes

• Monitoring methods– Capturing packets (raw and digested)– Blackhole networks

• responding only to ICMP echo requests• no actual hosts – only attack packets coming

– TCP first-client-packet monitor• sending a dummy ACK to a SYN request• Effective to obtain HTTP methods for attacks

• Traffic/alert logs (syslog, IDS logs)


A real-time analysis method example: change-point detection• Detecting timing of rapid change of a

time-variant data flow• Faster than repetitive statistical testings

Data flow

Detection Score

TimeChange Point

- Fast real-time learning

- Adaptive to long-term change

- Fast detection

- Low false-alarm rate

- Applicable to DDoS by detecting rapid quantitative change of traffics


A change-point analysis example

Change-pointscore

12/AUG/20045am JST

18/AUG/20041pm JST

time

MS Blaster activities detected

Number of dropped packets for TCP Port 135 Analysis data provided by NEC


Other candidate algorithms for the real-time traffic analysis

• Rare-ratio analysis– determining how rare an event is, by using the

standard/Gaussian distribution model

• Differential analysis– comparing event rate difference between short-

term and long-term time frames

• Those analyses are effective for comparing logs of multiple IDSes of different network traffic characteristics


An example of in-depth analysis: DDoS attacks on a well-known site

• The virus generates simultaneous HTTP requests on specific days of month

• The attacked site can no longer serve normal HTTP requests

• In-depth analysis performed by our engineers– Using actual traffics captured at the victim server – With cooperation of Telecom-ISAC and OCN (ISP

of NTT in Japan) twice on August 2004 and August 2005


In-depth DDoS analysis summary (1/2)

• Preprocessing– per-minute log of captured data– making digests of per-minute logs

• discarding unrelated payload contents• preserving necessary data for analysis• reducing the amount of data to process

– making access history of hosts • for each IP source address


In-depth DDoS analysis summary (2/2)

• Making per-host attack activity ranking– based on the history of each host

• using numbers of transmitted bytes, packets, HTTP requests, and session connection time

• Profiling based on HTTP methods– per-hour summary for each method sent

• Passive operating system estimation– using TCP signatures (p0f)


Digested log values and fields of each DDoS attacking packets

+ TCP- UNIX time() value- Packet length- Source IP address- Destination IP address- IP header flags- TCP header length- “T” for identifying TCP- Source port number- Destination port number- Sequence number- Ack number- TCP flags- TCP payload length- HTTP method (if existed)

+ UDP- UNIX time() value- Source IP address- Destination IP address- IP header flags- “U” for identifying UDP- Source port number- Destination port number- UDP payload length

+ ICMP- UNIX time() value- Source IP address- Destination IP address- IP header flags- “I” for identifying ICMP- Type- Code- ICMP payload length


DDoS activity of July 31, 2005

1

10

100

1000

10000

100000

1000000

15

:32

:00

15

:51

:00

16

:10

:00

16

:29

:00

16

:48

:00

17

:07

:00

17

:26

:00

17

:45

:00

18

:04

:00

18

:23

:00

18

:42

:00

19

:01

:00

19

:20

:00

19

:39

:00

19

:58

:00

20

:17

:00

20

:36

:00

20

:55

:00

21

:14

:00

21

:33

:00

21

:52

:00

22

:11

:00

22

:30

:00

22

:49

:00

23

:08

:00

23

:27

:00

23

:46

:00

Time in JST

Nu

mb

ers

of

pa

ck

ets

GET / HTTP/1.1

GET / HTTP/1.0

POST / HTTP/1.1

POST / HTTP/1.0

POST /cgi-bin/.. HTTP/1.1

POST /cgi-bin/.. HTTP/1.0

GET HTTP/1.1 climbing up


DDoS activity of August 1, 2005

1

10

100

1000

10000

100000

1000000

0:00

:00

0:46

:00

1:32

:00

2:18

:00

3:04

:00

3:50

:00

4:36

:00

5:22

:00

6:08

:00

6:54

:00

7:40

:00

8:26

:00

9:12

:00

9:58

:00

10:4

4:0

0

11:3

0:0

0

12:1

6:0

0

13:0

2:0

0

13:4

8:0

0

14:3

4:0

0

15:2

0:0

0

16:0

6:0

0

16:5

2:0

0

17:3

8:0

0

18:2

4:0

0

19:1

0:0

0

19:5

6:0

0

20:4

2:0

0

21:2

8:0

0

22:1

4:0

0

23:0

0:0

0

23:4

6:0

0

Time in JST

Num

bers

of p

acke

ts

GET / HTTP/1.1GET / HTTP/1.0POST / HTTP/1.1POST / HTTP/1.0POST /cgi-bin/.. HTTP/1.1POST /cgi-bin/.. HTTP/1.0

GET / HTTP/1.1 traffic jumped up

POST /cgi-bin/... HTTP/1.1 traffic slightly decreased

GET / HTTP/1.0 has a similar pattern to GET / HTTP/1.1


DDoS activity of August 2, 2005

1

10

100

1000

10000

100000

1000000

0:00

:00

0:44

:00

1:28

:00

2:12

:00

2:56

:00

3:40

:00

4:24

:00

5:08

:00

5:52

:00

6:36

:00

7:20

:00

8:04

:00

8:48

:00

9:32

:00

10:1

6:00

11:0

0:00

11:4

4:00

12:2

8:00

13:1

2:00

13:5

6:00

14:4

0:00

15:2

4:00

16:0

8:00

16:5

2:00

17:3

6:00

18:2

0:00

19:0

4:00

19:4

8:00

20:3

2:00

21:1

6:00

22:0

0:00

22:4

4:00

23:2

8:00

Time in JST

Num

ber

of p

acke

ts

GET / HTTP/ 1.1

GET / HTTP/ 1.0

POST / HTTP/ 1.1

POST / HTTP/ 1.0

POST / cgi-bin/ .. HTTP/ 1.1

POST / cgi-bin/ .. HTTP/ 1.0

GET / HTTP/1.1back to previousamount of traffic

GET / HTTP/1.0 reduced to almost zero after 7am

POST /cgi-bin/... HTTP/1.1 remained almost the same


Operating systems estimated for the DDoS attacking hosts

Windows XP (RFC1323, w+) [GENERIC]

Windows 2000 SP4, XP SP1 (firewall!)

Windows XP/2000 (RFC1323 no tstamp) [GENERIC]

OpenBSD 3.0 {note: this is probably a Web proxy server OS}Windows 3.11 (Tucows) (firewall!)

Windows XP/2000 [GENERIC]

Windows XP Pro SP1, 2000 SP3 (NAT!)

Windows XP Pro SP1, 2000 SP3

Windows 2000 SP2+, XP SP1 (seldom 98 4.10.2222)

Windows 2000 SP4, XP SP1

(The DDoS virus has been known as Windows-specific)


Trends observed from the monitored DDoS activities

• Increased on the day 1 of the month– two GET activities

• Steady traffics– two POST HTTP/1.1 activities– two POST HTTP/1.0 activities

• While the above three trend groups were the same as in 2004, detailed traffic time variance have been changed


Another candidate algorithms for in-depth analysis and visualization: self-organizing maps

similarity detected on incoming TCP packets and HTTP POST methods

similar patterns for / and /cgi... POST methods

- SOMs are effective to detect similarities between diffrent datasets- The meaning of the resulting figures is non-trivial, though


Schedule and things to do

• Research towards data integration needed– More expertise and research works needed to

understand the relationship between data trends and actual incidents happening on the networks

– More information sources needed• We need to be careful on the legal requirements and

rights of the network users (i.e., privacy of traffics)

• Schedule– December 2005: 1st beta-version demo of

Incident Analysis Center System– Production-level operation on 2007

Network Security Incident Analysis System for Detecting ... · Network Security Incident Analysis...

Documents

Transcript of Network Security Incident Analysis System for Detecting ... · Network Security Incident Analysis...