File000138

104
Module XXV– Log Capturing and Event Correlation

Transcript of File000138

Page 1: File000138

Module XXV– Log Capturing and Event Correlation

Page 2: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

News: Intelligent Log Analysis May Beef up Security

Security logs could help detect and prevent security breaches, but analyzing their reports is so boring that they're underutilized.December 15, 2008

The massive job cuts caused by the recession will pose a huge threat to enterprise security because insider attacks, like disgruntled former employees, account for half of data breaches. Log monitoring and analysis tools provide poor protection from internal breaches because analyzing their reports is a tedious process, experts say.

LogRhythm may have solved this problem by adding the Intelligent IT Search feature to its log management tool. This automatically classifies and tags log entries for easy searching, conducts risk modeling and prioritizes sensitive issues, and puts a universal time stamp on all activities to make them easier to monitor.

Those features will make searches easier, which may help system administrators more rapidly detect breaches through searching the logs. According to the 2008 Verizon (NYSE: VZ) Business Data Breach Investigations Report, which covered a four-year time span, event monitoring or log analysis detected only four percent of breaches.

The technology is sound, and adoption rates have been high for some time, the Verizon report said. "In 82 percent of cases, the victim possessed the ability to discover the breach had they been more diligent in monitoring and analyzing event-related information available to them at the time of the incident. The breakdown is in the process."

And that process is tedious. Few IT administrators have the time to read logs frequently and look for unusual data activity, Prat Moghe, Tizor Systems' founder and chief technology officer, said in an article in Compliance Week. According to him, one retailer had an IT staffer spending six hours a day to look through logs.

Source: http://www.internetnews.com/

Page 3: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Module Objective

• Computer Security Logs• Logs and Legal Issues• Log Management• Centralized Logging and Syslogs• Time Synchronization• Event Correlation• Log Capturing and Analysis Tools

This module will familiarize you with:

Page 4: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Module Flow

Computer Security Logs Log ManagementLogs and Legal Issues

Event CorrelationCentralized Logging

and SyslogsTime Synchronization

Log Capturing and Analysis Tools

Page 5: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Computer Security Logs

Page 6: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Computer Security Logs

Computer security logs contain information of the events occurring within an organization’s systems and networks

Security logs can be categorized as:

• Logs of Operating Systems (OSs) for servers, workstations, and networking devices (e.g., routers, switches)

Operating system logs:

• Logs of applications running on systems and servers such as email server, database server, etc.

Application logs:

• Logs of network and host-based security software

Security software logs:

Page 7: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Operating System Logs

OS logs are most beneficial for identifying or investigating suspicious activities involving a particular host

• Contains information of operational actions performed by OS components

Event Logs:

• Contains security event information such as successful and failed authentication attempts, file accesses, security policy changes, and account changes

Audit Logs:

Page 8: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Application Logs

Application Logs:

Client requests and server responses

Account information

Usage information

Significant operational actions

Application logs consist of all the events logged by the programs

Events that are written to the application log are determined by the developers of the software program

Windows Application Log

A Web Server Application Log

Page 9: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Security Software Logs

Common types of network and host-based security software include:

• Antimalware Software • Intrusion Detection and Intrusion

Prevention Systems • Remote Access Software • Web Proxies • Vulnerability Management Software • Authentication Servers • Routers • Firewalls • Network Quarantine Servers

IDS Log

Antivirus Log

Firewall Log

Page 10: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Router Log Files

Router stores log files in the router cache

Collect the bit stream image of the router cache for investigating log files

It provides detailed information of the network traffic on the Internet

It gives information on the attacks to and from the networks

Page 11: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Honeypot Logs

The honeypot administrator is the only authorized user of honeypot

The logs that are found in honeypot are considered suspicious

These honeypot logs help forensic team to catch the attacker

Page 12: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Linux Process Accounting

Linux Process Accounting tracks the commands that each user executes

The process tracking logfile is found at /var/adm, /var/log or /usr/adm

The tracked files can be viewed with lastcomm command

It enables process tracking by accton command or the startup (/usr/lib/acct/startup)

Page 13: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Logon Event in Window

When the user logs on or off the computer, a logon event is generated

Logon on the security log is generated in the remote server when the user is connected to it

It can determine the attempts to log on interactively at servers

It examines the attacks launched from a particular computer

Page 14: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Windows Log File

• sysevent.evt• secevent.evt• appevent.evt

Windows log files are stored in %systemroot%\system32\config\

Event viewer files can be checked in Control Panel Administrative Tools

• Kiwi Syslog for Windows• Event Reporter

Tools used for auditing these log files:

Page 15: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Configuring Windows Logging

Page 16: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Analyzing Windows Logs

Page 17: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Setting up Remote Logging in Windows

Deleting c:\winnt\system32\config\*.evt could erase the event-tracking logs

Windows does not support remote logging unlike Linux

NTSyslog enables remote logging in Windows

Page 18: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Windows Log File: System Logs

Page 19: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Windows Log File: Application Logs

Page 20: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Logon Events That Appear in the Security Event Log

Page 21: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Logon Events That Appear in the Security Event Log

Page 22: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

IIS Logs

IIS logs all the server visits in log files located at:

• <%systemroot%>\logfiles

If proxies are not used, then IP can be logged

This command lists the log files:

• http://victim.com/scripts/ ..%c0%af../..%c0%af../..%c0%af../..%c0%af../..%c0%af../..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+C:\Winnt\system32\Logfiles\W3SVC1

Page 23: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Maintaining Credible IIS Log Files

Most of the network administrators have encountered serious Web server intrusions that have resulted in legal action

Often, IIS logs are considered as the primary evidence used to track down Web intruders

IIS logs can provide convincing evidence of your argument if their credibility is challenged in court

Protect and maintain the accuracy, authenticity, and accessibility of logs to make them reliable and admissible evidence

Page 24: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Log File Accuracy

Log file accuracy is proving that log file data truly represents the activity on the Web server

Even the smallest inaccuracy can bring into question the validity of the entire set of data

Page 25: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Log Everything

For logging everything , configure your IIS logs to record every available field

While few administrators see value in storing this extra information, every field has some significance in forensic investigation

Gathering information about Web visitors helps establish that an attack came from a specific computer system or logged in user

For example, suppose a defendant claims a hacker had broken into his computer and installed a backdoor proxy server, then used that backdoor proxy to attack other systems; in this case logging every server activity may help investigators in finding the origin of traffic and perpetrator of the crime

Page 26: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Keeping Time

• Key: HKLM\SYSTEM\CurrentControlSet|Services\W32Time\Parameters\Setting: TypeType: REG_SZValue: NTP

• Key: HKLM\SYSTEM\CurrentControlSet|Services\W32Time\Parameters\Setting: NtpServerType: REG_SZValue: ntp.xsecurity.com

On a standalone server, you can synchronize to an external source by setting the following registry entries:

Synchronize your IIS servers to an external time source using the Windows Time Service

If you use a domain, the Time Service will automatically be synchronized to the domain controller

Page 27: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

UTC Time

IIS records logs using UTC time

It helps in synchronization issues , when running servers in multiple time zones

Windows calculates UTC time by offsetting the value of the system clock with the system time zone

The only way to be sure the UTC time is correct is to ensure that the local time zone setting is accurate

If your server is set at UTC -0600, then the first log entries should appear around 18:00 (00:00 - 06:00 = 18:00)

Page 28: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

View the DHCP Logs

The DHCP logs are saved in the C:\WINNT\System32\DHCP folder on DHCP servers

Actual location depends on where Microsoft Windows NT or Microsoft Windows 2000 is installed

Page 29: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

DHCP Logs

Page 30: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

ODBC Logging

ODBC logging is a record of a fixed set of data properties in a database that complies with ODBC, such as Microsoft Access or Microsoft SQL Server

It includes the IP address of the user, user name, request date and time, HTTP status code, bytes received, bytes sent, action carried out, and the target file

It specifies the database to be logged to, and sets up the database to receive the data

Page 31: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Logs and Legal Issues

Page 32: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Legality of Using Logs

First, the logs must be created reasonably and contemporaneously with the event

Log files should not be tampered with

Someone with knowledge of the event must record the information

In this case, the recording is being done by a program; the record therefore reflects the prior knowledge of the programmer and system administrator

Logs must be kept as a regular business practice

Random compilations of data are not admissible

Logging systems instituted after an incident do not qualify under the business records exception

Keep regular logs to use them as evidence later

Page 33: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Legality of Using Logs (cont’d)

A “custodian or other qualified witness” must testify to the accuracy and integrity of the logs

The custodian need not be the programmer who wrote the logging software; however, he or she must be able to offer testimony on what sort of system is used, where the relevant software came from, how and when the records are produced, etc.

It is necessary to offer testimony for the reliability and integrity of the hardware and software platform used, including the logging software

A record of failures or security breaches on the machine creating the logs will tend to impeach the evidence

Log entries of the machine claimed to be penetrated are considered suspicious

Page 34: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Legality of Using Logs (cont’d)

In a civil lawsuit against the attackers, anything in your own records that would tend to exculpate the defendants can be used against you

Your own logging and monitoring software must be made available to them, to permit them to attack the credibility of the records

But under certain circumstances, if you can show that the relevant programs are trade secrets, you may be allowed to keep them secret, or disclose them to the defense, only under a confidentiality order

The original copies of any files are preferred

A printout of a disk or tape record is considered to be an original copy, unless and until judges and jurors come equipped with USB/SCSI interfaces

Page 35: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Records of Regularly Conducted Activity as Evidence

“A memorandum, report, record, or data compilation, in any form, of acts, events, conditions, opinions or diagnoses, made at or near the time by, or from information transmitted by, a person with knowledge, if kept in the course of a regularly conducted business activity, and if it was the regular practice of that business activity to make the memorandum, report, record, or data compilation, all as shown by the testimony of the custodian or other qualified witness, unless the source of information or the method or circumstances of preparation indicate lack of trustworthiness”

Rule 803, Federal Rules of Evidence

Page 36: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Laws and Regulations

• Federal Information Security Management Act of 2002 (FISMA)• Gramm-Leach-Bliley Act (GLBA)• Health Insurance Portability and Accountability Act of 1996 (HIPAA)• Sarbanes-Oxley Act (SOX) of 2002• Payment Card Industry Data Security Standard (PCI DSS)

The following regulations, standards, and guidelines define organizations’ needs for log management:

Page 37: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Log Management

Page 38: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Log Management

• Log Generation• Log Analysis and Storage• Log Monitoring

Log management infrastructure typically comprises the following three tiers:

Log management includes all the processes and techniques used to collect, aggregate, and analyze the computer-generated log messages

Log management systems consist of the hardware, software, network and media used to generate, transmit, store, analyze, and dispose of log data

Page 39: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Functions of Log Management

• Log parsing • Event filtering • Event aggregation • Log rotation • Log archival and retention • Log compression • Log reduction • Log conversion • Log normalization • Log file integrity checking • Event correlation • Log viewing • Log reporting • Log clearing

Log management system performs the following functions:

Page 40: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Challenges in Log Management

Detecting variety of intrusions attempted on your network

Overall Internet bandwidth usage of the enterprise network

Identifying who/when/what activities inside your network

Individual employees’ non-business web usage

Audit and regulatory compliance requirements

Monitoring enterprise policy implementation of access to internal network resources

Threats and user activities at server and SQL applications

Regulatory compliance and audit requirements

Forensic analysis

Troubleshooting

Page 41: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Centralized Logging and Syslogs

Page 42: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Central Logging Design

ConversationalMonitor System Portal

StreamingMedia

JavaApplication

SyslogSyslog

Backup Log Server Mail Apache

Swatch

Page 43: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Centralized Logging Setup

Router

IDS

Host

Firewall

AGENTS

Oracle Database

Reporting Tool: Real-Time Analysis: Forensics Report

NF Engine: Event Aggregation and

Correlation

Page 44: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Steps to Implement Central Logging

1.• Secure the location of log server

2.• Turn off all services that are running for security purpose

3.• Turn off all Internet Daemon services such as Syslog and Secure Shell

4.• Disable Remote Procedure Call (RPC) services

5.• Disable all unnecessary accounts

6.• Specify the time on all devices

Page 45: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Syslog

Syslog is a client/server protocol standard for forwarding log messages across an IP network

The term syslog refers to both the syslog protocol and the application or library sending syslog messages

Syslog sender sends log message to the syslog receiver also known as syslogd, syslog daemon or syslogserver

Syslog messages use UDP and/or TCP

Log messages are sent in cleartext

Page 46: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Syslog in Unix-like Systems

Syslog is a comprehensive logging system that is used to manage information generated by the kernel and system utilities

It allows messages to be sorted by their sources and routed to various destinations

• Examples:• Log files and user’s terminals

It is controlled through the configuration file /etc/syslog.conf

To log all messages to a file, replace the selector and action fields with the wildcard

Configure Syslog to log all authorize messages with a priority of lower or higher to the /var/log/syslog

Page 47: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Steps to Set Up a Syslog Server for Unix Systems

1.• Create a central syslog server that accepts incoming syslog messages

2.• Configure to listen on UDP port 514

3.• Run syslogd with –r option

4.• Configure other servers to log their message to this server

5.• Modify the action field in the syslog.conf file as below

• Auth.* @10.0.0.2

Page 48: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Centralized Syslog Server

• Central Syslog is kept on a different segment for storage security

• Attacker finds it difficult to delete the logs• Log messages allow co-relation of attacks across different

platforms• It has an easier backup policy• Real time alerts are generated by using tools such as Swatch

Advantages of Centralized Syslogging:

Page 49: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Centralized Syslog Server (cont’d)

Routers and Switches Unix/Windows serversFirewall

Central Syslog Server

Log Data Mining

Online Alerting

Log Analysis and Reporting

Page 50: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

IIS Centralized Binary Logging

Centralized binary logging is a process where multiple Web sites send binary and unformatted log data to a single log file

It is a server property, so all the Web sites on that server are configured to write log data to the central log file

It reduces administration burden for Internet Service Providers (ISPs), and helps in collecting and storing the logged data

Page 51: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Extended Logging in IIS Server

Enables extended logging in IIS servers

Page 52: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Time Synchronization

Page 53: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Why Synchronize Computer Times?

A key component of any computer security system is regular review and analysis of both certain standard system log files as well as the log files created by firewalls and intrusion detection systems

If computers are running on different times, it becomes almost impossible to accurately match actions logged on different computers

In case you suffered an intrusion, though your computers have the same time, it might be difficult to correlate logged activities with outside actions if your computer time is wrong

Page 54: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

What is NTP?

An Internet standard protocol (built on top of TCP/IP) that assures accurate synchronization to the millisecond of computer clock times in a network of computers

NTP synchronizes client workstation clocks. Running as a continuous background client program on a computer, NTP sends periodic time requests to servers, obtaining server time stamps to adjust the client's clock

Page 55: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

NTP Stratum Levels

NTP stratum levels define the distance from the reference clock

A reference clock is a stratum-0 device that is assumed to be accurate and has little or no delay associated with it

The reference clock synchronizes to the correct time (UTC) using long wave radio signals, GPS transmissions, CDMA technology or other time signals such as WWV, DCF77, etc.

Stratum-0 servers cannot be used on the network; instead, they are directly connected to computers which then operate as stratum-1 servers

A server that is directly linked to a stratum-0 device is called a stratum-1 server

Higher stratum levels are distanced from the stratum-1 server over a network path

A stratum-2 server gets its time over a network link, via NTP, from a stratum-1 server

A stratum-3 server gets its time over a network link, via NTP, from a stratum-2 server, and so on

Page 56: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Direct Connection(e.g.. RS 232)

Network Connection

NTP

Page 57: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

NIST Time Servers

time-a.nist.gov129.6.15.28 NIST, Gaithersburg, Maryland

time-b.nist.gov129.6.15.29 NIST, Gaithersburg, Maryland

time-a.timefreq.bldrdoc.gov132.163.4.101 NIST, Boulder, Colorado

time-b.timefreq.bldrdoc.gov132.163.4.102 NIST, Boulder, Colorado

time-c.timefreq.bldrdoc.gov132.163.4.103 NIST, Boulder, Colorado

utcnist.colorado.edu128.138.140.44University of Colorado, Boulder

time.nist.gov192.43.244.18NCAR, Boulder, Colorado

Time-nw.nist.gov131.107.13.100Microsoft, Redmond, Washington

Page 58: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

NIST Time Servers (cont’d)

nist1.symmetricom.com69.25.96.13Symmetricom, San Jose, California

nist1-dc.WiTime.net206.246.118.250WiTime, Virginia

nist1-ny.WiTime.net208.184.49.9WiTime, New York City

nist1-sj.WiTime.net64.125.78.85WiTime, San Jose, California

nist1.aol-ca.symmetricom.com207.200.81.113 Symmetricom, AOL facility, Sunnyvale, California

nist1.aol-va.symmetricom.com 64.236.96.53 Symmetricom, AOL facility, Virginia nist1.columbia

countyga.gov 68.216.79.113 Columbia County, Georgia

nist.expertsmi.com 71.13.91.122 Monroe, Michigan

nist.netservicesgroup.com 64.113.32.5 Southfield, Michigan

Page 59: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Configuring the Windows Time Service

• Click Start, click Run, type regedit, and then click OK• Locate and then click the following registry subkey: HKEY LOCAL

MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters• In the right pane, right-click ReliableTimeSource, and then click Modify• In Edit DWORD Value, type 1 in the Value data box, and then click OK• Locate and then click the following registry subkey: HKEY LOCAL

MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters• In the right pane, right-click LocalNTP, and then click Modify• In Edit DWORD Value, type 1 in the Value data box, and then click OK• Quit Registry Editor• At the command prompt, type the following command to restart the Windows Time

Service, and then press ENTER: net stop w32time && net start w32time• Run the following command on all the computers other than the Time Server to reset the

local computer's time against the Time Server: • w32tm -s

To configure Windows Time Service to use an internal hardware clock, follow these steps:

Page 60: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation

Page 61: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation

Event correlation is a procedure, assigned with a new meaning for set of events that occurs in a predefined interval of time

During this process, some events may be added and some events may be deleted

It happens usually inside the log management platform

In general, event correlation process is implemented with the help of simple event correlator software

• Event aggregation• Event masking• Event filtering• Root cause analysis

The four different steps in event correlation:

Page 62: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Types of Event Correlation

• This type of correlation is used when one common OS is used throughout the network in an organization

• Example, organization running Microsoft Windows OS (any version) for all their servers may be required to collect event log entries, do trend analysis diagonally

Same-platform correlation

• This type of correlation is used when different OS and network hardware platforms are used throughout the network in an organization

• Example, clients may use Microsoft Windows, yet they use Linux-based firewall and email gateway

Cross-platform correlation

Page 63: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Prerequisites for Event Correlation

• Transmitting of data from one security device to other until it reaches a consolidation point in the automated system

• To have a secure transmission and to reduce the risk of exposure during transmission of data, the data has to be encrypted and authenticated

Transmission of data

• After the data is gathered, it must be formatted again from different log formats to single or polymorphic log and that can be easily inserted into the database

Normalization

• After collecting the data, repeated data must be removed so that the data can be correlated more efficiently

• Removing of unnecessary data can be done by compressing the data, deleting repeated data, filtering or combining similar events into a single event and sending that to the correlation engine

Data reduction

Page 64: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation Approaches

• This approach constructs graph with each node as a system components and each edge as a dependency among two components

Graph-based approach

• This approach uses a neural network to detect the anomalies in the event stream, root causes of fault events, etc

Neural network-based approach

• In this approach, events are correlated according to set of rules as followed condition -> action

Rule-based approach

• This approach uses codebook to store set of events and correlate them

Codebook-based approach

Page 65: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation Approaches (cont’d)

• A basic approach where specific events are compared with single or multiple fields in the normalized data

Field-based approach

• This method checks and compares all the fields systematically and intentionally for positive and negative correlation with each other to determine the correlation across one or multiple fields

Automated field correlation

• This approach is used for correlating particular packets with other packets• This approach can make a list of possible new attacks by comparing packets with

attack signatures

Packet parameter/payload correlation for network management

Page 66: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation Approaches (cont’d)

• This method is used to identify whether any system is a relay, a formerly compromised host, and/or to detect the same hacker from different locations

• A series of data sets can be gathered from forensic event data such as, isolated OS fingerprints, isolated port scans, finger information, and banner snatching to compare link attack data to other attacker profiles

Profile/fingerprint-based approach

• This approach is used to map IDS events that targets a particular vulnerable host with the help of a vulnerability scanner

• This approach is also used to deduce an attack on particular host in advance and it prioritizes attack data so that trouble spots can be responded to quickly

Vulnerability-based approach

• The open port correlation approach determines the rate of successful attacks by comparing it with the list of open ports available on the host and that are being attacked

Open-port-based correlation

Page 67: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Event Correlation Approaches (cont’d)

• This approach is an advanced correlation method which assumes and predicts what a attacker can do next after the attack by studying the statistics and probability and uses only two variables

Bayesian correlation

• This approach eyes the computers' and computer users' behavior and alerts if some anomalous thing is found

Time (clock time) or role-based approach

• This approach is used to extract the attack route information and uses that information to single out other attack data

Route correlation

Page 68: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Log Capturing and Analysis Tools

Page 69: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Syslog-ng Logging Systemhttp://www.balabit.com/

• Reliable log transfer• Secure logging using SSL/TLS• IETF syslog protocol standards support• Disk-based message buffering• Flexible message filtering and sorting• Direct database access• Flow control• Heterogeneous environments• Agent for Microsoft Windows platforms• Agent for IBM System i platforms• IPv4 and IPv6 support

Features of Syslog-ng:

Syslog-ng is a flexible and scalable audit trail processing tool for organizations of any size

It provides a centralized, securely stored log of all devices on the network

Page 70: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Syslog-ng: Screenshot

Page 71: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

WinSyslog Syslog Serverhttp://www.winsyslog.com/

• Centralized Logging • Interactive Server • Send Syslog Test Message • Standards Compatible • WinSyslog Web Access • Syslog Hierarchy • Email Notifications • Store Messages Persistently • Multiple Instances • Full logging, robust, minimal Resource Usage • Firewall Support • NT Service • Multi-Language Client • Friendly and Customizable User Interface • MWAgent effectively handles for low-memory cases

Features:

WinSyslog is an enhanced syslog server for Windows

It is an integrated, modular and distributed solution for system management

Page 72: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

WinSyslog: Screenshot

Page 73: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Kiwi Syslog Serverhttp://www.kiwisyslog.com/

• Display the message in the scrolling window • Log the message to a text file • Forward the message to another syslog server• Log to an ODBC database • Log to the NT Application Event Log • Email the message to someone via SMTP • Triggering a sound alarm • Run an external program• Send an SNMP Trap message • Page someone using NotePager Pro

Syslog messages can then be processed using events such as:

Kiwi Syslog Server receives syslog messages from network devices, and displays them in real time

Actions can be performed on received messages and messages can be filtered by host name, host IP address, priority, message text or time of day

Page 74: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Kiwi Syslog Server: Screenshot

Page 75: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Tenable Security Centerhttp://www.nessus.org/

•Quickly rediscover your entire network

Asset Discovery

•Present and make sense of your network security information

Reporting

•Aggregate and Correlate your security logs with the optional LCE module

Log Aggregation and Correlation

•Distribute the scan load throughout your whole network

Distributed Scanning

•Audit the configuration of each system on your network and make sure it matches your local security policy

Configuration Auditing

•Track the action of the network administrators

Security Workflow

Tenable Security Center provides continuous, asset-based security and compliance monitoring

Features:

Page 76: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Tenable Security Center: Screenshot

Page 77: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

IISLogger: Development Tool

• Generates additional log information from IIS• Recognizes hacker attacks• Forwards IIS log data to Syslog

It is an addition to the standard Internet Information server logging which:

IISLogger is an ISAPI filter

It is a Dynamic Link Library (.dll) embedded in the IIS environment

Even if the IIS calls an ISAPI filter notification, IISLogger prepares header information and logs this information to syslog in a certain format

Page 78: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

IISLogger: Screenshot

Page 79: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Socklog: IDS Log Analysis Tool

• Selects and de-selects the log entries• Minimizes the code size• Provides modular and reliable network logging• Merges different logs and sorts them in order

Benefits of Socklog:

Socklog is a secure replacement tool for Syslog

It is a small, secure and reliable tool

Page 80: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Microsoft Log Parser: Forensic Analysis Toolhttp://www.microsoft.com/

• Produces the desired information either on the screen, in a file of any desired format into a SQL database

• Allows multiple files to be piped in or out as source or target tables• Generates HTML reports and MS Office objects• Supports conversion between SQL and CSV (Computer System Validation)

Features of Microsoft Log Parser:

It is a command-line program that allows user or administrator to run SQL (Sequential Query Language)-like queries against log files of any format

Output is available from text to XML files and XML files to database storage

Page 81: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Microsoft Log Parser Architecture

SQL Engine

IIS Logs Text Files Event Log FileSystem

RegistryUser

Plug-in

SYSLOGSQL

DatabaseText FilesScreen,Console

Page 82: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Microsoft Log Parser: Screenshot

Page 83: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Firewall Analyzer: Log Analysis Tool

Firewall Analyzer is a web-based firewall monitoring and log analysis tool that collects, analyzes and reports information on enterprise-wide firewalls, proxy servers and radius servers

It helps in tracking intrusion detection, managing user access, auditing traffic and managing network bandwidth efficiently

It uses a built-in syslog server to store the firewall logs and provides comprehensive reports on firewall traffic and security breaches

Page 84: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Firewall Analyzer Architecture

Page 85: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Firewall Analyzer: Screenshot

Page 86: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Adaptive Security Analyzer (ASA) Pro

• Model security specialist expertise• Baseline what is normal for the environment• Identify published threats• Identify activity matching pre-defined criteria• Identify, Measure and Prioritize all anomalous events• Generate root cause insight of threats• Impart new knowledge back into the system

It enables you to:

ASA Pro is a security and threat intelligence application that continuously monitors dynamic, high volume, heterogeneous security-related data, recognizes and quantifies the extent of event abnormality

It provides a flexible mechanism whereby the expert knowledge of the security analyst can be modeled

It reduces the time required to review security-related information

Page 87: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

ASA Pro Implementation Model

Page 88: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

ASA Pro: Screenshot

Page 89: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

GFI EventsManager

Collects data from all devices that use Windows event logs, W3C, and Syslog and applies the best rules and filtering in the industry to identify key data

This allows you to track when staff swipe their fob, pick up the phone to call home, turn on their PC, what they do on their PC and which files they access during their workday

GFI EventsManager also provides you with real-time alerting when critical events arise and suggests remedial action

Page 90: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

How Does GFI EventsManager Work?

• GFI EventsManager will automatically collect Windows event logs W3C and Syslog data from remote log sources

Stage 1 – Event Collection

• GFI EventsManager will process collected events and normalize processed events to a central database

Stage 2 – Event processing and centralization

• During this stage, GFI EventsManager will generate meaningful reports on its findings, trigger email, SMS and network alerts on key events and trigger remedial actions such as the execution of a script or executable file on key events

Stage 3 – Generate output/results

GFI EventsManager breaks down the events management process in 3 automated operational stages, making the product easy to use and configure

Page 91: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Page 92: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

GFI EventsManager

Page 93: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Activeworx Security Center

Activeworx Security Center is a Security Information and Event Management product

Activeworx Security Center monitors security-related events for a variety of devices from one console

Page 94: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Activeworx Security Center Desktop

Page 95: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Ntsyslog

Page 96: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

EventReporter

Centralized logging tool for Windows

EventReporter processes the NT Event Logs, parses them and forwards the results via Syslog protocol to a central Syslog server

Page 97: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

EventLog Analyzer

• Event archiving• Automatic alerting• Pre-defined event reports• Historical trending

Features:

EventLog Analyzer is a web-based systems log analysis tool

It collects, analyzes and reports on application, system, security, file server, and DNS server event logs from enterprise-wide Windows and UNIX systems and routers or switches

Page 98: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

EventLog Analyzer - Screenshot

Page 99: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

FLAG - Forensic and Log Analysis GUIhttp://www.dsd.gov.au/

FLAG was designed to simplify the process of log file analysis and forensic investigations

It uses a database as a backend to assist in managing the large volumes of data, this allows flag to remain responsive and expedite data manipulation operations

It is web-based which enables it to be deployed on a central server and shared with a number of users at the same time

Data is loaded into cases which keeps information separated

It also has a system for reporting the findings of the analysis by extensively using bookmarks

Page 100: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

FLAG Screenshot

Page 101: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Simple Event Correlator (SEC)http://kodu.neti.ee/

SEC is an open source and platform independent event correlation tool

It accepts input from regular files, named pipes, and standard input, and can thus be employed as an event correlator for any application that is able to write its output events to a file stream

The SEC configuration is stored in text files as rules, each rule specifying an event matching condition, an action list, and optionally a Boolean expression whose truth value decides whether the rule can be applied at a given moment

Regular expressions, Perl subroutines, etc. are used for defining event matching conditions

SEC can produce output events by executing user-specified shell scripts or programs (e.g., snmptrap or mail), by writing messages to pipes or files, and by various other means

Page 102: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Summary

Computer security logs contain information on the events occurring within systems and networks

OS logs are most beneficial for identifying or investigating suspicious activity involving a particular host

Syslog allows messages to be sorted by their sources and routed to various destinations

Centralized binary logging reduces administration burden for Internet Service Providers (ISPs), and helps in collecting and storing the logged data

Stratum-0 servers cannot be used on the network; instead, they are directly connected to computers which then operate as stratum-1 servers

Event correlations happen usually inside the log management platform

Page 103: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited

Page 104: File000138

EC-CouncilCopyright © by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited