ii

ABSTRACT

Organizations require security systems that are flexible and adaptable in order to combat

increasing threats from virus attacks and other malicious code, in addition to internal attacks.

Intrusion Detection Systems are increasingly a key part of systems defense. Artificial

intelligence plays a driving role in security services. Monitoring of network involves collections

of information on every connection and analysis of traffic monitors what services are spread

throughout the network and its comparison with the desired and expected activity. Following

these two practices recognizes services which are unauthorized and being used within any

network. This project emphasizes on traffic and network analysis and applying snort rules to the

analyzed traffic to detect threats within the network.

iii

TABLE OF CONTENTS

Abstract.............................................................................................................................. ii

List of Figures.................................................................................................................... v

1 Background and Rationale.................................................................................................1

1.1 Introduction....................................................................................................................1

1.2 Intrusion..........................................................................................................................2

1.3 What is an Intrusion Detection? .....................................................................................5

1.3.1 Anomaly Detection......................................................................................................5

1.3.2 Misuse Detection..........................................................................................................5

1.4 Intrusion Detection System..............................................................................................5

1.4.1 Structure and Architecture of an IDS............................................................................7

1.5Artificial Intelligence.........................................................................................................9

1.5.1Why using A.I?...............................................................................................................9

1.5.2 Intelligent Agent...........................................................................................................10

1.5.3 Agent Environment.......................................................................................................11

2. Narrative.............................................................................................................................14

2.1Network Analysis..............................................................................................................14

2.2Packet Sniffing...................................................................................................................15

2.2.1 Wireshark........................................................................................................................16

2.3 Snort...................................................................................................................................19

iv

2.3.1 Snort Rules....................................................................................................................19

2.3.2 Structure of a Snort Rule...............................................................................................19

3. System Design....................................................................................................................25

3.1Analyzing a Rule...............................................................................................................25

3.2Developed Engine.............................................................................................................27

3.3Process Flow.....................................................................................................................31

4. Testing and Evaluation ......................................................................................................36

4.1 Detection of Threats ........................................................................................................36

4.1.1 Backdoor Attempt..........................................................................................................36

4.1.2 Detection of Chat Rules..................................................................................................37

4.1.3 Detection of Address using Snort Rules..........................................................................39

4.1.4 Suspicious Incoming Packets..........................................................................................40

4.2 Evaluation of the System....................................................................................................41

5. Future work...........................................................................................................................42

6. Conclusion.............................................................................................................................43

Bibliography and References....................................................................................................44

v

LIST OF FIGURES

Figure 1.1 IDS Components.................................................................................................6

Figure 1.2 Intrusion Detection System.................................................................................7

Figure 1.3 Components of an Intrusion Detection System...................................................8

Figure 1.4 Agent Environment.............................................................................................11

Figure 2.1 Wiresshark Snapshot...........................................................................................15

Figure 2.2 Basic Layout of Wireshark..................................................................................16

Figure 2.3 Wireshark Protocols............................................................................................17

Figure 2.4 Layout of the log File Captured Using Wireshark..............................................18

Figure 2.5 Structure of a Snort Rule.....................................................................................20

Figure 2.6 Rule Syntax ........................................................................................................21

Figure 2.7 Rule Header Attributes of a Snort Rule..............................................................22

Figure 2.8 Rule Options.......................................................................................................22

Figure 2.9 Snippet of a Snort Rule.......................................................................................24

Figure 3.1 Process Flow of the Program..............................................................................27

Figure 3.2 Process Flow of the Working of the Program.....................................................31

Figure 3.3 Wireshark Capture..............................................................................................32

Figure 3.4 Wireshark Log File.............................................................................................32

Figure 3.5 Layout of the Frame...........................................................................................33

Figure 3.6 Snortcheck.exe...................................................................................................33

Figure 3.7 Selection of Log File..........................................................................................34

Figure 3.8 Detected Results............................................................................................... 34

Figure 3.9 Results File..........................................................................................................35

vi

Figure 4.1 Screenshot of the Results File.............................................................................36

Figure 4.2 Screenshot of the Log File..................................................................................37

Figure 4.3 Screenshot of a Frame.........................................................................................38

Figure 4.4 Screenshot of a Log Frame.................................................................................38

Figure 4.5 Screenshot of a Results File for Detection of Address......................................39

Figure 4.6 Screenshot of a Frame for Detection of Address...............................................39

Figure 4.7 Screenshot of a Results File for Suspicious Incoming Packet...........................40

Figure 4.8 Screenshot of a Frame for Suspicious Incoming Packet....................................40

ii

1

1. BACKGROUND AND RATIONALE

1.1 Introduction

Research on IDS has gained vital significance as attacks on public or private

networks have been increasing day by day. Corporate world consider information as its

invaluable asset hence the need of protecting it against any threats. IDS have the capacity to

constantly monitor data on computer networks and take corresponding action such as block

or report any anomaly to the administrator. An IDS identifies the data packets on the

networks and reaches their objective by thoroughly purging each packet existing on the

network and there by thwarting any attempts to intrude networks which is more effective

than employing a firewall. Using IDS within the networks is helpful in information

management as they log every application or packet, as it traverses through different nodes in

a network. Before proceeding into artificial intelligent techniques in intrusion detection let us

see what an intrusion is.

Any detecting solution which merely hunts for attacks would be missing the key

aspect: Recognizing any unwanted and unauthorized traffic which is not harmful i.e., does

not contain any malicious code. This kind of traffic is either a result of a host which is giving

access to such unauthorized services which could be accidental or on purpose or could have

been generated by an equipment that has gone wrong. It could also be possible that certain

types of outdoor traffic entering in your secured internal network are a result of an error in

the firewall configuration. Ignorance of these issues could pose as serious threats to your

network and easy targets for hackers. The key to good and secured network is critical

evaluation of unauthorized traffic.

2

In the field of artificial intelligence, an intelligent agent is referred as a system

which acts in its own environment to achieve it desired goals. Intelligent agents use its

knowledge in order to reach its destination. It may also learn from other sources if the

knowledge is insufficient. They may either be simple or too complex. Intelligent agents are

often compared to a computer program because of its functional behavior. They act as an

example in the development of software applications. In the field of computer science, an

intelligent agent is referred to a software agent who possesses some intelligence. Intelligent

agents are a system which has an ability to perform independent and flexible actions in order

to meet its desired goals.

An Intelligent agent is often referred as flexible as it responds quickly to the

changes that occur in its environment. An agent observes it environment and responds to the

changes with respect to the time. They are also proactive as they not only respond to the

changes but also strive to achieve its goals simultaneously. Intelligent agents are social

agents which communicate with each other solving their problem to complete the given task.

1.2 Intrusion

According to Webster‟s, intrusion is an act of trespassing or thrusting oneself in

without invitation or welcome. Similarly intrusion in technically world means that any event

which can enter into private network without authorization thwarting the network security.

The meaning and the significance doesn‟t change when an application differs from a physical

intrusion to the electronic intrusion [Allen 2000]. The conceptual definition for an intrusion

is “Any set of actions that attempt to compromise integrity, confidentiality or availability of a

resource”.

Confidentiality: Information accessibility is given only to the authorized users.

3

Integrity: It deals with alteration of the data that is the trustworthiness of information.

Availability: Information availability is given only to the authorized users.

The above three properties are collectively referred to as CIA which forms the core of

information assurance. If any of the above properties are compromised then it is considered as a

security breach of the system. Intrusions can be of various forms. Most common of them are

smartly engineered worms or viruses etc [Dubrawsky 2001].

Since intrusions into unauthorized computers have been increasing rapidly in the recent

past, system administrators are routinely coping up with computer recovery following three basic

steps- find out how intrusion happened, the percentage of damage incurred and fixing the

vulnerable computer.

First of all, for a system administrator to take on the above steps, he should be able to

find suspect file or process which is called detection point in order to confirm that system has

been attacked. There are many tools available to find out whether a computer is compromised or

not. Trip wire is a tool that can tell you whether a system file is modified or not, network firewall

can give the information whether a process is scanning a port to launch attack or not. Such tools

suffer from one or more limitations [Caswell 2003]. But once an administrator identifies the

detection point in a compromised system, he should take on investigation as to how it did

happen. There are two main sources of information a system administrator can refer to while

investigation – System or Network logs and disk state. These sources may or may not give the

required information and it‟s also difficult for the system administrator to infer what happened

by analyzing the information from the sources. To make this job easier, system administrator can

make use of tools like coroner‟s toolkit, snort etc, which helps in recovering the deleted files and

4

logging network traffic respectively. Unfortunately these tools and available sources of

information suffer from their own limitations. For example, network logs or host logs can show

information specific to a single application like HTTP and login times etc, but they don‟t show

any information related to what occurred on the compromised system. Most of the tools cannot

differentiate between intruder actions and legitimate administrator actions and hence it becomes

difficult to understand the sequence of events that have occurred during system compromise.

1.3 What is Intrusion Detection?

An intrusion is defined as any set of actions that attempt to compromise the integrity,

confidentiality, or availability of a resource. An earlier study done by Allen [Allen 2000] uses

the term “threat" in this same sense and defines it to be the potential possibility of a deliberate

unauthorized attempt to

access information,

manipulate information, or

render a system unreliable or unusable.

An intrusion is a security breach of the set of policies in a system. The goals and requirements

should be satisfied in a system according to the set of rules defined [Bace 2000]. Detecting

intrusions can be categorized as anomaly intrusion detection and misuse intrusion detection.

Conceptually there are two broad categories of intrusion detection techniques:

1.3.1 Anomaly Detection:

System behavior is analyzed to categorize the state of the system into two profiles –

normal activity profile and not normal activity profile. Anomaly detection method can detect any

5

unknown attacks unlike misuse detection. Set of intrusive activities may not be similar to that of

anomalous activities taking the situation into new horizon with interesting possibilities. In this

case any anomalous activity which is not intrusive is flagged as intrusive and any intrusive

activity that bypasses the detection mechanism is flagged false negatives which are considered as

a potential threats. Anomaly detection systems should have a threshold level of overhead so that

any of the above problems is not exponentially amplified [Caswell 2003].

1.3.2 Misuse Detection:

In this method a pattern of an attack is captured and later used to prevent the same

attack in future this is the basis for a learning algorithm in which all the attacks are initially

studied and corresponding signatures are cataloged so that the same attacks can be prevented in

future. But this method is not so promising for naive attacks. The primary issue of this method of

detection is to build a pattern that can detect signatures of all the variations of the attack to avoid

false negatives and also to ensure that any signature that is build does not match with genuine

activities to avoid false positives [Caswell 2003].

1.4 Intrusion Detection System

As network administrators know that, Internet is a bigger network of smaller

computer networks. It evolved due to the interconnectivity of networks around the globe. This

interconnection is a very good feature for communication to take place; it allows the free

exchange of information via the various networking protocols available. But as everything is

two-sided, it also has a very bad feature. Here the other side being “the risk of your

communication being monitored by a third person (or they are usually known as “crackers”) to

gain unauthorized access to your resources or communication [Bace 2000].

6

For example, the availability of computing facilities can be targeted using some kind of attack

called the Denial of Service (DoS) attacks.

Figure 1.1 IDS Components

Intrusion detection systems in the corporate network are very sophisticated and well

understood in the networking world. Generally, IDS is used to establish some sort of security

perimeter around the network with an aims to block or restrict both incoming and outgoing

network traffic. This overcomes problem with firewalls being the ability to maintain information

security, at the same time supporting the free exchange of ideas [Allen 2000]. In an IDS the

incoming and outgoing network signals as well as recognition of possibly malicious

manipulations in a network or system thereby triggers an alert to the administrator so as the

network is safeguarded. Any system or a set of systems possessing the capability to identify and

detect suspicious actions in a system or a network is known as an intrusion detection system.

7

1.4.1 Structure and Architecture of an IDS

In an intrusion detection system, sensor always plays as a vital core element which is

designed to detect intrusions. The decision making rules regarding the intrusions are stored in the

sensor [Boncheva 2006]. The raw data is received by the sensors from the three vital information

sources (Fig 1.2) IDS knowledge database, system log and the audit trails. The system log

includes list of authorized users, file system configurations etc. This information is used for

further decision making procedure.

Figure 1.2 Intrusion Detection System

The sensor is combined with an event generator which is responsible for data collection (Fig1.3).

The collection procedure is determined by the event generator policy which shows filtering of

the event notification information. The set of policy consistent events along with the policy

information is stored either inside the protected system or outside the protected system.

8

Figure 1.3 Components of an Intrusion Detection System

The role of a sensor is to filter all the unwanted data and discard any unrelated data

which is processed from the event within the protected system. The analyzer uses the policy

database to detect malicious activities in this procedure. The latter compromises signature

attacks, required parameters and behavior profiles which are normal. The database also holds the

configuration parameters of the intrusion detection system which includes modes of

communication with the response module. The history of the multiple and complex intrusions are

stored in sensor‟s database [McHugh 2000].

Detection of security violations in information systems is the objective of IDS.

Intrusion detection is considered as a passive method to provide security for the information

system as it alarms or detects after the attack occurs. Privilege abuse or software exploitation is

some of the examples of security violations. Basing on the input information, the IDS are

categorized as host based and network based. As the name suggests host based IDS‟s analyze

host bound audit sources like system logs, audit trails whereas in the network based IDS‟s

packets captured on the network are analyzed.

9

1.5 Artificial Intelligence

Artificial intelligence in the field of computer science is considered to focus on

creating specific machines that are capable to engage on intelligent human behaviors. Artificial

intelligence was termed in 1956 by John McCarthy at the Massachusetts Institute of Technology.

The common motive in all areas of artificial intelligence is to create a machine which can think

as a human [TT 2005]. Artificial intelligence includes

Expert systems: Programming computers in decision making for real time situations and

also provides solution to any problem.

Neural networks: In field of artificial intelligence we use neural networks for problem

solving without establishing any biological system.

Natural language: Programs every system to understand natural human language.

Game playing: Programming computers to play more than one game without difficulty.

Robotics: The design of artificial intelligence to program computers to react to any other

sensory stimuli.

1.5.1 Why using AI?

There are many false positives to deal with during monitoring network traffic the

system manager should always be ware and judgmental of impeccably identify true threats. To

aid system managers in handle such situation AI could be of more help than regular custom

practices. Out of the many reasons AI is preferred, the three most important are

Flexibility (vs. threshold definition)

Adaptability (vs. specific rules)

Pattern recognition (and detection of new patterns)

10

1.5.2 Intelligent Agent

An Intelligent agent is often referred as flexible as it responds quickly to the changes

that occur in its environment. An agent observes it environment and responds to the changes with

respect to the time. They are also proactive as they not only respond to the changes but also

strive to achieve its goals simultaneously. Intelligent agents are social agents which

communicate with each other solving their problem to complete the given task. Intelligent agents

are one of the best contributions made to the field of artificial intelligence. An agent based

system is widely expanding its technology in the field of Software, Industries, Information

Systems, Commercial type of Application, Process Control & Air Traffic Control, Business

Process and Information Management, Entertainment (Game playing, Theaters &Cinemas),

Medical Applications such as (Patient monitoring, Health care).

Intelligent agents are divided into various classes. An agent which uses artificial

character to exhibit its personality to communicate with the other agent to achieve its goals is

categorized into the class of believable agents. Physical agent observes through its sensors and

acts through with its actuators. A temporal agent always uses stored information or data with

respect to time to offer some data to a computer program. A temporal agent often takes time

based decisions in order to complete its tasks [TT 2005].

1.5.3 Agent Environment

An intelligent agent is an entity which observes and acts upon an environment and

directs the paths towards achieving goals. An agent can gather information from its environment

only depending on the characteristics of that particular environment. The properties of an

environment affect the design of the agent. An environment must be observable in order to

11

consider an agent as an intelligent agent .Intelligent agents observes the environment and

respond to the changes occurring in that particular environment with respect to time. An agent

always receives an input from its environment and reacts to it in order to change it to achieve its

desired goals.

Figure 1.4 Agent Environment

Properties of Agent Environment

Accessible Vs Inaccessible

An environment can be accessible if all the sensors of the agent can detect all the

specific actions that are carried out in an environment. In an accessible environment an agent can

get complete accurate information regarding the state of hat environment with respect to time. If

the environment is more accessible, less is the complexity to build an agent to operate in it. An

environment is accessible to an agent its sensors can give access to the complete state of that

environment. An environment like this comfortable because the agent does not have to maintain

the track of internal state. In the same way if the sensors cannot detect the actions then such an

12

environment is defined as inaccessible. For an inaccessible environment it is necessary for the

agents to maintain the internal state. Complex environments are inaccessible.

Deterministic Vs Non-Deterministic

If the current state determines the next state of an environment with respect to the

actions selected by the agents then such an environment is referred as deterministic. If the next

state of the environment is completely dependent on the current state then it is purely

deterministic. A non-deterministic state is always inaccessible. It is also true that a non-

deterministic state is complex and cannot maintain the internal state. Complex systems are non-

deterministic, state which results from an action are not guaranteed even when the system is in

similar state before the action is applied.

Episodic Vs Non-Episodic

An episodic environment is divided into number of episodes and an agent in each

episode observes and then acts in that environment. Hence the actions of the agents are not

independent on the number of discrete episodes with no links in between the performance in

various scenarios. This considers a current environment and is simpler to design. On-episodic

environment is considered to be sequential as it is independent on the episodes and the actions

being performed.

Static Vs Dynamic

A static environment does not change until and unless the result is produced by the

actions of the agents. If the environment does not change while the agent is performing the

actions then it is static. It is easy to deal with static environment because while deciding an

13

action it does not look at the world. Dynamic environment operates by changing the environment

outside the agent‟s control. Dynamic environment requires complex agent design. If the agent

doesn‟t respond with respect to time then a dynamic environment changes [TT 2005].

\

14

2. NARRATIVE

The scope of the project includes analysis of any network log data to determine any

suspicious computer or any threat to our systems. The suspicious activity or IP address sending

threats within the concern network can be determined by analyzing the network log using any

network sniffers like Wireshark, Kismet, TCP dump etc [Gerg 2004].

2.1 Network Analysis

Network Analysis is a process of evaluating organizations network and assessing the

current state of network infrastructure. The analysis proceeds by scanning the entire network, IP

by IP; determining service pack levels, missing patches, open ports, active applications on the

network, key registry entries, extraneous users or groups and any weak passwords. This detail

analysis can be of extreme significance as it uncovers any potential threats on the network and

thereby be a harbinger of our action to take necessary precaution to prevent any attack from

external entities [Kozierok 2005].

A log file that is captured from the network consists of the essential data like – IP

addresses of source and destination, data over the network, TCP/UDP ports, size of each data

packet, timestamps, internet protocol identification bits etc. This data can be used to identify any

threats or suspicious activity over the network. Log files can be obtained by using tools as

mentioned above. Wireshark is proven tool to sniff networks and produce a log file in human

readable format with various control options that facilitate log analysis.

15

The sample wireshark snapshot while capturing network data is as shown below –

Figure 2.1 Wireshark Snapshot

2.2 Packet Sniffing

It is a cruel irony in information security that efficient tools and features that are

supposed to protect a network can be exploited and compromised to thwart the security of the

same. One of such ideas is packet sniffing. Packet sniffing is a practice of system administrator

to monitor the network and analyze the packets legitimately to trouble shoot the network traffic

using the analyzed data the administrator can judge each packet to pin point any erroneous

information and use the data to maintain smooth traffic in avoiding bottlenecks in data

transmission [Wiki 2008]. A packet sniffer captures all the packets that pass through a given

network. It also has the capability of sniffing of whole network traffic in promiscuous mode or it

16

can just capture information within a given subnet which makes it difficult for an attacker to find

a place to launch the malicious packet in the network .If a packet sniffer is placed in a subnet in a

corporate network, the network traffic from any of the corporate machine can be thoroughly

captured and can any trojan or malicious packet can be detected and logged internally to make it

available to the system administrator. A rouge packet on a network cannot be detected easily but

since the sniffer is passive by nature, any such packet can be captured in the network interface

that is because the sniffing packets have no signature for a rouge packet to identify them.

2.2.1 Wireshark

Also known as Ethereal, wireshark is network protocol that can be installed and used on

unix and windows platforms. It has the capability to capture live data from the network and

examines the network traffic. Figure 2.2 shows the basic layout of how wireshark is used.

Figure 2.2 Basic Layout of Wireshark

17

It provides with granular levels of network details that the administrator can use to

study the network data interactively with the wireshark interface. It has some powerful features

like data filters, protocol libraries and has the ability to reconstruct the TCP stream in a given

session. The main protocols that Wireshark can support include - AFS, ANSI ISUP, ANSI MAP,

ASN.1 PER, Bluetooth HCI H4, Bluetooth L2CAP, BSS CFLOW, COPS, Diameter, DICOM,

FF-HSE, ICMPv6, IEC-60870-5-104, IEEE 802.11, Infiniband, IPMI, MIOP, RADIUS, RSVP,

sFlow, SNMP, SMB2, ZIOP etc. Other supported protocols include –

Figure 2.3 Wireshark Protocols

18

Wireshark can also capture live data from Token – Ring, 802.11 wireless LAN (on OS

permissions), ATMs, serial ports etc. The sample layout of the log file captured using Wireshark

is as follows –

Figure 2.4 Layout of the log file captured using Wireshark

19

2.3 Snort

Snort is an open source packet sniffer / packet logger /network intrusion detection

system, which is freely available under GNU Public license. As a packet sniffer it intercepts the

data packets on the screen and displays them on the screen. As a packet logger, it specifies

network traffic either as ASCII text or binary format [Snort 2008]. As a network Intrusion

Detection System, it screens the network traffic for any predetermined set of rules that can

trigger an alert whenever errant packets are detected in the network.

2.3.1 Snort Rules

Like viruses, most intruder activity has some sort of signature. Information about these

signatures is used to create Snort rules. These signatures or attacks may be present in the header

part of a packet or in the payload. Snort's detection system is based on rules [Andrew 2007].

These rules, in turn, are based on intruder signatures. Snort rules can be used to check various

parts of a data packet.

2.3.2 Structure of a Snort Rule

A Snort Rule consists of two main components, the Rule Header and the Rule Body, as

shown in Figure 2.5

The Rule Header: The Rule Header is divided in four main categories that are described as follows:

Rule Actions: A rule action is the first part of a Snort rule. It shows what action will be taken when rule

conditions are met. There are three predefined actions as described below:

Pass: This action tells Snort to ignore the packet. This action plays an important role in speeding up Snort

operation in cases where we don't want to apply checks on certain packets.

20

Figure 2.5 Structure of a Snort Rule

Log: This action tells Snort to log the packet in a manner as specified during the configuration of the

Snort sensor.

Alert: The alert action is used to send an alert message when rule conditions are true for a particular

packet.

Protocols: Protocol is the second part of a Snort rule. The protocol part of a snort rule shows on which

type of packet the rule will be applied. Currently Snort understands the following protocols:

Internet Protocol (IP)

Internet Control Message Protocol (ICMP)

Transmission Control Protocol (TCP)

User Datagram Protocol (UDP)

21

Source Information: This gives the information of the source computer from where the packet

originated. It has two parts, the IP address of the source computer and the port number of the

source computer. The keyword any can be used to apply a rule on all addresses. Similarly, the

keyword any can be used to apply the rule on all packets irrespective of the port number .

Destination Information: This gives the information of the computer to where the packet is

flowing to. It has two parts, the IP address of the destination computer and the port number of the

destination computer. The keyword any can be used to apply a rule on all addresses. Similarly,

the keyword any can be used to apply the rule on all packets irrespective of the port number .

Rule Body: The Rule Body contains various sections enclosed inside a pair of parentheses. Each

section defines an option trailed by the desired option value. There may be one option or many

and the options are separated with a semicolon. When multiple options are used, they form a

logical AND. The action in the rule header is invoked only when all criteria in the options are

true. There are several rule options; the discussion of each of them is beyond the scope of this

report [QOD 2004].

After capturing the traffic from the network, the developed engine compares the traffic data with

the snort rules to determine threats in the network. Snort rules verify and validate if protection is

given against vulnerability in the network [Sturges 2008]. The format of snort rules is as follows-

<rule action> <protocol> <source address & port> -> <destination address & port>

Figure 2.6 Rule syntax

22

Rule Header

Header has the information to identify action of the rule – Dynamic, Alert, Log, Pass

and Activate are some of the defined actions in snort rules. Header contains following fields:

Action (log, Alert), Protocol (TCP, UDP, IP, ICMP), Source IP and Port, Destination IP and

Port, Direction Operator (“->”, “<-“).

Figure 2.7 Rule header attributes of a snort rule

Rule Options: In this part of the rule alert messages are identified.

Figure 2.8 Rule Options

23

Source and Destination Internet Protocol addresses can be – Variables ($HOME_NET),

Classless Inter-Domain Routing Blocks, Individual IP addresses (98.121.122.30, 10.3.147.46).

Ports can be – Individual Ports, port ranges (“:1024”, “80:85”, “1025:”).

Snort Rules body begins with “(“and ends with “)” followed by rule options separated by

“;”. Rule options can specify Payload detection, Non-payload detection, Metadata, Post-

detection, Suppression size. Metadata options provide information about the rule to the log

analyst. Metadata information can be of the following types: “msg” specifies the human-readable

alert message, “reference” includes a URL for more info, “classtype” and “priority” give some

idea about the type of attack and the severity of the event, “sid” and “rev” uniquely identify the

rule (including revisions & edits). “Classtype” implies default priority for each class in the

manual and the “priority” option can be used to override these default properties. Sid range

should be greater than 4,000,000 to avoid conflicts with providers of the snort rules. Payload is

the actual data content also the meat of the packet. The options in the Payload detection include –

“content” (strings of data), “nocase” (case insensitive attribute), “pcre” (allows Perl-compatible

regular expressions), “offset” (bytes length to skip before searching) [Andrew 2007].

TCP is stateful protocol that requires a valid session with the server on the network.

The TCP packets may be discarded if the proper session is not established with valid connection.

TCP data without valid session takes up CPU processing time and may not be fruitful in

delivering the packets to the destination. Hence, rules should be designed in such a way that TCP

establishes valid sessions to log network traffic [Kozierok 2005].

24

The rules are defined by the system administrator to identify any activity in the network which

is not following the pre set rules [QOD 2004]. Hence, these rules can differentiate genuine

activities from malicious activities in the network. This rule is defined by the system

administrator –

Alert tcp $HOME_NET any -> any 6667 (msg:”IRC port in use”; flow:from_client)

The header portion (any -> any 6667) defines the action to examine port 6667 traffic. If a match

happens, then a message „IRC port in use‟ would be generated and IDS would insert a record in

the log that IRC port had been used.

The sample snippet of snort rules is as given below –

Figure 2.9 Snippet of Snort Rules

25

3. SYSTEM DESIGN

The designed system uses artificial intelligence technique to identify any threats in the

network by comparing each frame from the network log with the defined snort rules. Snort rules

are designed based on intruder signatures. All the parts of the data packet can be checked using

these rules. Initial version of Snort can penetrate through layers 3 and 4 to analyze a

frame/packet in the network and later version of snort can even go through application layer.

Depending on the type of the packet, rules are applied to each frame [QOD 2004]. Rules are

defined to generate messages like – alert, log etc., and a rule can allow the system to drop a

packet silently. The snort rule follows easy to understand syntax, most of them written in one

line. Complex rules can be defined as that cannot fit in one line using backslash character at the

end of each line. These rules are generally placed in configuration file (snort.conf).

3.1 Analyzing a Rule

The Syntax of every snort rule looks like this – rule header (rule options). The rule

header consists of the following [Sturges 2008] –

<rule action><protocol><source address & port> -> <destination address & port>

The rule options follow the following syntax:

Keyword : argument, keyword : argument,…

Suppose the following rule is used in the analysis of the network log:

alert tcp $EXTERNAL_NET any -> $HOME_NET 23 (msg:"TELNET SGI telnetd format

bug"; flags: A+; content: "_RLD"; content: "bin/sh"; reference:arachnids,304;

classtype:attempted-admin; sid:711; rev:2;)

26

The meaning of the rule as it suggests is that it should generate an alert and log a message for

any TCP packet encountered in the network that comes from external net (external port or

address space) to the home net (internal net/port 23). The rule options as shown in the above rule

are as follows:

This says to generate an alert (and a log message) for any TCP packet coming from an external

address space (and any port) destined to the local address space (and port 23).

And its rule options are:

msg:"TELNET SGI telnetd format bug": - Specifies the message to printed for log

and laert engines

flags: A+ :- Matches with the TCP Acknowledgement flag and any other set flag

content: "_RLD" :- Matches with the given string in the packets payload (leave to

default)

content: "bin/sh" : Matches with the given string in the packets payload (leave to

default)

reference:arachnids,304 :- packet is forwarded to arachnids which is external attack

identification system with id 304

classtype:attempted-admin :- Sets priority of the alert to high if an attack happens to

gain administrator privilege.

sid:711 :- identifies this rule as #711.

rev:2 associates this rule with a revision of "2".

All the above matches and conditions must be true to trigger the rule and log the alert

message into the results file on any suspicious threat.

27

3.2 Developed Engine

The engine has three main components – Form creator, snort rules comparison and

display results. The process flow of the program is shown in– Figure 3.1

Figure 3.1 Process Flow of the Program

The rules are stored in a C++ structure as given in the following snippet. The attributes of the

rule structure include source address and port, destination address and port, message that rule

contains, content/options in the rule and rule id/name-

struct SRule{

AnsiString rule;

AnsiString proto;

AnsiString src_addr;

AnsiString src_port;

AnsiString dst_addr;

AnsiString dst_port;

AnsiString msg;

AnsiString content; } Rules[1000];

The general steps of the scan and store in the structure is as shown below –

f=fopen(("rules\\"+sr.Name).c_str(),"r"); - statement to open the file to read

28

Rules[r_cnt].proto=S.SubString(1,S.Pos(" ")).UpperCase().Trim(); - Statement to read the

protocol from rule, make all letters to the upper case and trim spaces. Read actual components of

the rule into the structure. The sample snippet for reading source address is as shown and the

similar code is used to read all the other components of the rule –

Rules[r_cnt].src_addr=S.SubString(1,S.Pos(" ")-1); S=S.SubString(S.Pos(" "),S.Length()-S.Pos("

")-1); while(S.Pos(" ")==1) S=S.SubString(2,S.Length()-1);

After these rules are successfully stored in the structure, read the log file captured by wireshark

and compare each frame with the rule to detect any threat. There can be different types of strings

in the log like caption string or actual log. If it is a caption, then there would not be time stamp

for that string in the log which can be checked using the line-

x0=S.Pos("No."); x1=S.Pos("Time");

if(!x0||!x1){

fgets(S.c_str(),500,f);

continue; }

We initialize all the significant components namely IPs and Ports before reading the header of

the frame in the log. Depending on the length of the packet, the loop continues to run to scan and

store the details of the frame. If TCP or IP ports are not in the frame, the frame is passed by and

continues to scan the next frame. Components of the frame are displayed in the form one by one

by the given line –

fputs(("Frame "+IntToStr(fr_cnt++) +" Length "+len+" Proto="+Proto+"

("+SrcIp+":"+SrcPort+"->"+DstIp+":"+DstPort+")").c_str(),f1);

All the port addresses are in aliases format which have to be converted to number

formats e.g. http port has to be translated to port 80, https port has to be translated to port 443,

domain has to be translated to port 53 etc. Thirty nine aliases are translated to their respective

port numbers in this program. Each frame is compared with each record in the rule structure for

29

any inconsistency. We assign a particular address as home network (IP address of the university

and its mask) so that any address other than home net is considered as External network. These

variables (Home net and External net) can be used to substitute in the rules and compare the

frame with the rule. After each frame is processed, the frame details are logged in the results file

with user friendly message that informs whether it is a threat or not. A sample snippet of the

Home net address comparison from the code is as shown –

if(!Uip.Pos(".")) return 0;

_s3=Uip.SubString(1,Uip.Pos(".")-1); Uip=Uip.SubString(Uip.Pos(".")+1,Uip.Length()-

Uip.Pos("."));

msk_val=StrToInt(EUsrMsk1->Text);

loc_val=StrToInt(EUsrIp1->Text);

if((StrToInt(_s3)&msk_val)!=(loc_val&msk_val)) return 0;

_s3=Uip.SubString(1,Uip.Pos(".")-1); Uip=Uip.SubString(Uip.Pos(".")+1,Uip.Length()-

Uip.Pos("."));

msk_val=StrToInt(EUsrMsk2->Text);

loc_val=StrToInt(EUsrIp2->Text);

if((StrToInt(_s3)&msk_val)!=(loc_val&msk_val)) return 0;

_s3=Uip.SubString(1,Uip.Pos(".")-1); Uip=Uip.SubString(Uip.Pos(".")+1,Uip.Length()-

Uip.Pos("."));

msk_val=StrToInt(EUsrMsk3->Text);

loc_val=StrToInt(EUsrIp3->Text);

if((StrToInt(_s3)&msk_val)!=(loc_val&msk_val)) return 0;

_s3=Uip;

msk_val=StrToInt(EUsrMsk4->Text);

loc_val=StrToInt(EUsrIp4->Text);

if((StrToInt(_s3)&msk_val)!=(loc_val&msk_val)) return 0;

This program uses different type of snort rules like Chat rules, Back door rules, Bad-traffic rules

and Attack-response rules.

The sample snippet of the rules is as given below –

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ATTACK-RESPONSES

directory listing"; flow:established; content:"Volume Serial Number"; classtype:bad-unknown;

sid:1292; rev:9;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"ATTACK-

RESPONSES command completed"; flow:established; content:"Command completed"; nocase;

reference:bugtraq,1806; classtype:bad-unknown; sid:494; rev:10;)

30

alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"ATTACK-

RESPONSES command error"; flow:established; content:"Bad command or filename"; nocase;

classtype:bad-unknown; sid:495; rev:8;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"ATTACK-

RESPONSES file copied ok"; flow:established; content:"1 file|28|s|29| copied"; nocase;

reference:bugtraq,1806; reference:cve,2000-0884; classtype:bad-unknown; sid:497; rev:12;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"ATTACK-

RESPONSES Invalid URL"; flow:from_server,established; content:"Invalid URL"; nocase;

reference:url,www.microsoft.com/technet/security/bulletin/MS00-063.mspx;

classtype:attempted-recon; sid:1200; rev:10;)

The results file is the actual log that user/sys admin looks into to detect any suspicious activity.

Frame 11 Length 54 Proto=TCP (10.3.150.136:3339->69.4.231.53:80)

Frame 12 Length 54 Proto=TCP (69.4.231.53:80->10.3.150.136:3339)

80->any Incoming HTTP srv packet

1 threat was found, suspicious source IP: 69.4.231.53

Frame 13 Length 151 Proto=3Com (AskeyCom_de:09:45:no->Cisco_5f:62:a2:no)

The interface for the program is a form coded in C++ which has two edit fields, list box and a

button. The logic behind the button includes – scan frames from the log file and compare them

with the snort rules to generate results file in user readable format.

WINAPI WinMain(HINSTANCE, HINSTANCE, LPSTR, int)

{

try

{

Application->Initialize();

Application->CreateForm(__classid(TForm1), &Form1);

Application->Run();

}

catch (Exception &exception)

{

Application->ShowException(&exception);

}

return 0;

}

31

3.3 Process Flow

The actual process flow of how the program works and all the activities that user does is as

shown in– figure 3.2

Figure 3.2 Process Flow of the Working of the Program

The steps to be followed to use this program are as follows:

a. Install wireshark on any of the computers connected to the network to log the network

activity.

32

Figure 3.3 Wireshark Capture

b. Get the log file and place it on the drive accessible to the administrator to analyze it.

The frame logged by the wireshark shows the list of parameters that are necessary for

detection.

Figure 3.4 Wireshark Log File

33

c. The layout of each frame in the log file that was captured by the wireshark from live

traffic. is as shown in – figure 3.5

Figure 3.5 Layout of the frame

d. Start executing the developed program snortcheck.exe-

Figure 3.6 Snortcheck.exe

34

e. Click on the „open log file‟ button to select the log file captured from wireshark. Also

select to check only the threats that are present in the log file or just the clear

messages.

Figure 3.7 Selection of Log File

f. The program displays the frames and detected results in the Form window and the

detailed description is given in the results file.

Figure 3.8 Detected Results

35

Figure 3.9 Results File

36

4. TESTING AND EVALUATION

Testing was done by analyzing various network logs in the developed program. The

logs have been captured in different environments with many numbers of frames in each log. The

log that was captured was fed to the developed program and analyzed to detect threats. Some of

the cases that we have tested are as follows:

According to the snort rules, Assume that there are two types of IP addresses that belong to

either Home_Net or External_Net.

4.1 Detection of Threats

4.1.1 Backdoor Attempt:

Wireshark was used to log frames in a network with 165.95.10.143 as Home_net and any

other address as External_net.

4.1.1.2 Setup to connect two routers to one computer:

To use your router as an access point only,

Router A is the router used as the router which is connected to the modem. Router B is the router

used as an access point or switch.

Step 1 Do not use the WAN port on router B.

Step 2 Depending on the network setup, change the LAN IP address of router B.

The default is 192.168.0.1. If another router is being used, DHCP or Internet connection sharing

software, change the LAN IP address of the router to an IP in subnet (192.168.0.2). The LAN IP

address must be static.

Step 3 Disable DHCP on router B.

37

A frame was logged in the wireshark log which was detected as suspicious and backdoor attempt

when the rule from back-door rules was activated-

“ alert ip any any -> 216.80.99.202 any (msg:"BACKDOOR fragroute trojan connection

attempt"; content:""; reference:bugtraq,4898; classtype:trojan-activity; sid:1791; rev:2;) “

Figure 4.1 Screenshot of the Results File

Suspicious Frame in the Log file:

Figure 4.2 Screenshot of the Log File

38

4.1.2 Detection of Chat Rules

Suspicious Activity detected during chats according to chat rules. Frames are recorded in the

network log using wireshark during chat session to detect any threats according to the defined

chat rules. One of the chat rules that was activated during the session in testing was –

“ alert tcp $HOME_NET any <> $EXTERNAL_NET any (msg:"HA-AH CHAT MSN

message"; flow:established; content:"@A"; classtype:policy-violation; sid:540; rev:11;) ”

The corresponding frame was –

Figure 4.3 Screenshot of a Frame

After the detecting the threats present in the log that was stored in the computer by the wireshark

is being checked and the corresponding log frame was –

39

Figure 4.4 Screenshot of a Log Frame

4.1.3 Detection of the Address using Snort Rules

Suspicious Address detected according to snort rules in the network. According to snort

rules, Home_Net and External_Net can have only sessions on permissible ports. If the session

takes place on different port or different IP protocols, snort rules detect it as suspicious address –

Figure 4.5 Screenshot of a Results File for Detection of Address

40

Figure 4.6 Screenshot of a Frame for Detection of Address

4.1.4. Suspicious Incoming Packet

The developed program also detects any suspicious source address during sessions or

chats or attack response intervals. We have tested the program for such an attempt by deploying

wireshark to capture such a frame in the network and input the log to the program.

Figure 4.7 Screenshot of a Results File for Suspicious Incoming Packet

41

Figure 4.8 Screenshot of a Frame for Suspicious Incoming Packet

4.2 Evaluation of the System

We have done thorough testing of this program in different environments and evaluated

the systems performance. The program can analyze any number of frames and can process 25 to

30 frames per second. The Network log captured from the wireshark should contain the IP

addresses, TCP/UDP ports and protocol information, frame size and data packets for the program

to detect suspicious activity. We can assign any IP address as Home_Net to detect and evaluate

the systems performance.

42

5. FUTURE WORK

As part of future work for this project, testing the ability of detecting attacks can be

performed on many other Snort rules. With good test criteria, with proper network logs all the

snort rules can be examined and tested in order to determine the performance of the system in

detecting threats. Therefore, this project throws beacon on the scope of security policy design

and network analysis.

43

6. CONCLUSION

This project involved the analysis of network traffic using various tools like

wireshark, packet sniffers etc and identifying suspicious points in the log thereby determining

malicious IP addresses/ ports in the network using snort rules. The developed engine is

intelligent in the sense that it reads the essential data from the network log and compares the

attributes like IP address, ports, packet size, message, cyclic redundancy check etc. with the

predefined snort rules like chat rules, attack- response rules, bad traffic rules, backdoor rules etc.

Thereby surfacing any suspicious activity to the system administrator by creating a

corresponding results file for every log file. Hence we can sum up that the designed system uses

artificial intelligence technique to identify any threats in the network by comparing each frame

from the network log with the defined snort rules.

44

BIBLIOGRAPHY AND REFERENCES

[Allen 2000] Allen ,J.,Christie,A.,Fithen,W.,McHugh,J.,Pickel,J.,and Stoner, E. State of the

Practice of Intrusion Detection Technologies. Technical report, Carnegie Mellon University.

[Bace 2000] Rebecca Gurley Bace. Intrusion Detection. Macmillan Computer Publishing

(MCP), Indianopolis. 2000.

[Boncheva 2006] Boncheva, V., A Short Survey of Intrusion Detection Systems,

Available from www.iit.bas.bg/PECR/58/23-30.pdf (Visited Feb. 12, 2010).

[Bro 2007] Bro.Bro Intrusion Detection System. Lawrence Berkeley National Laboratory.

National Science Foundation (2007) Available from www.bro-ids.org (visited Mar. 15, 2010).

[Caswell 2003] Caswell, B. Snort 2.0 Intrusion Detection. Syngress Publishing, Inc., Rockland,

MA, pp 55-73.

[Chapple 2003] Chapple, M. Evaluating and tuning an intrusion-detection system.

Available from http://searchsecurity.techtarget.com/tip/0,289483,sid14_gci918619,00.html

(visited May. 09, 2010).

[Chou 2007] Chou, T., Ensemble Fuzzy Belief Intrusion Detection Design. Available

from www.proquest.umi.com (Visited Mar. 15, 2010).

[CISSP 2008] CISSP 2008. Examining Different Types of Intrusion Detection Systems.

Wiley Publishing, Inc. (2008). Available from

www.dummies.com/WileyCDA/DummiesArticle/Examining-Different-Types-of-

Intrusion-Detection-Systems.id-5278.html (visited Jan. 18, 2010).

[Dubrawsky 2001] Dubrawsky, I. Freeware Intrusion Detection Tools (2001). Available

from www.samag.com/documents/s=1147/sam0108o/0108o.htm (Visited Feb. 9, 2010).

[Gerg 2004] Gerg, C. and Cox, K. J. Managing Security with Snort and IDS Tools,O‟Reilly

Media, Inc. Sebastopol, (Visited Jan. 19, 2010).

[Innella 2006] Innella, P. An Introduction to Intrusion Detection System. Available from

www.securityfocus.com/infocus/1520 (Visited May. 27, 2010).

[JC 2007] Jupitermedia Corporation. Intrusion Detection System (2007). Available from

http://www.webopedia.com/TERM/I/intrusion_detection_system.html (visited May. 26, 2010).

[Kozierok 2005] Kozierok, M. Charles. TCP/IP Guide. No Strach Press, 2005.

45

[McHugh 2000] McHugh, J. Defending Yourself: The Role of Intrusion Detection

Systems. IEEE Computer Society Press, Los Alamitos, CA (September 2000).

Volume 17, Issue 5, Pages: 42 – 51.

[QOD 2004] QoDwriting. A look into IDS/Snort. Available from

www.freewebs.com/talug/Snort.pdf (visited Apr. 15, 2010)

[Snort 2008] Snort.org. Available from www.snort.org (visited May 12, 2010).

[Sturges 2008] Sturges, S. Writing Snort Rules: How to Write Snort Rules and Keep Your

Sanity. Available from www.snort.org/docs/snort_htmanuals/htmanual_283/snort_manual.html

(visited Jan.09, 2010).

[TT 2005]Tech Target. A Review of Artificial Intelligence (Jun. 2005). Available from

www.searchsecurity.techtarget.com/sDefinition/0,,sid14_gci295031,00.html

[UCR 2008] UCRiverside Security. Security- Glossary of Terms. Available from

www.cnc.ucr.edu/security/index3.php?content=glossary.html (visited May. 23, 2010).

[Wiki 2009] Wikipedia. Packet Analyzer. Available from

http://en.wikipedia.org/wiki/Packet_analyzer (visited Dec. 18, 2009).