Improving the precision of an Intrusion Detection...

Improving the precision of an Intrusion

Detection System using Indicators of

Compromise- a proof of concept -

Lejonqvist,Gisela Larsson,Oskar

Information Security, master's level (120 credits)

2018

Luleå University of Technology

Department of Computer Science, Electrical and Space Engineering

i

Abstract

The goal of this research is to improve an IDS so that the percentage of true positives is high, an organisation can cut time and cost and use its resources in a more optimal way. This research goal was to prove that the precision of an intrusion detection system (IDS), in terms of producing lower rate of false positives or higher rate of true alerts, can be achieved by parsing indicators of compromise (IOC) to gather information, that combined with system-specific knowledge will be a solid base for manual fine-tuning of IDS-rules.

The methodology used is Design Science Research Methodology (DSRM) because it is used for research that aims to answer an existing problem with a new or improved solution. A part of that solution is a proposed process for tuning of an arbitrary intrusion detection system.

The implemented and formalized process Tuned Intrusion Detection System (TIDS) has been designed during this research work, aiding us in presenting and performing validation tests in a structured and robust way. The testbed consisted of a Windows 10 operating system and a NIDS implementation of Snort as an IDS. The work was experimental, evaluated and improved regarding IDS rules and tools over several iterations. With the use of recorded data traffic from the public dataset CTU-13, the difference between the use of tuned versus un-tuned rules in an IDS was presented in terms of precision of the alerts created by the IDS.

Our contributions were that the concept holds; the precision can be improved by adding custom rules based on known parameters in the network and features of the network traffic and disabling rules that were out of scope. The second contribution is the TIDS process, as designed during the thesis work, serving us well during the process.

Keywords: Intrusion Detection System; Indicator Of Compromise; False Positives; Snort.

ii

Table of Contents

Abstract ............................................................................................................................................. i

Table of Contents ............................................................................................................................. ii

Abbreviations and Acronyms ...........................................................................................................v

Tables and Figures ......................................................................................................................... vi

Page ................................................................................................................................................ vi

Chapter 1: Introduction ................................................................................................................... 1

1.1 Introduction ....................................................................................................................... 1

1.2 Research Problem............................................................................................................ 3

1.3 Research question ........................................................................................................... 4

1.4 Proposed solution and research goals ............................................................................ 4

1.5 Research problem domain ............................................................................................... 4

1.6 Research contribution ...................................................................................................... 4

1.7 Research delimitations ..................................................................................................... 5

1.8 Structure of thesis ............................................................................................................ 5

Chapter 2: Background ................................................................................................................... 6

2.1 Introduction ....................................................................................................................... 6

2.2 Indicator of Compromise .................................................................................................. 6

2.2.1 The IOC lifecycle and Tuning process ..................................................................... 7

2.2.2 IOC Framework ......................................................................................................... 8

2.3 IDS .................................................................................................................................. 10

2.3.1 IDS architecture ...................................................................................................... 11

2.3.2 IDS detection methods ........................................................................................... 14

2.3.3 Tuning of IDS .......................................................................................................... 15

2.3.4 IDS deployment ...................................................................................................... 15

2.3.5 IDS comparisons ..................................................................................................... 17

2.3.6 Performance measurement .................................................................................... 17

2.4 Tuning of IDS rules ........................................................................................................ 20

Chapter 3: Literature review ......................................................................................................... 22

3.1 Introduction ..................................................................................................................... 22

3.2 Literature review ............................................................................................................. 22

iii

3.3 Earlier research .............................................................................................................. 23

3.3.1 IOC standards ......................................................................................................... 23

3.3.2 Gathering of IOCs ................................................................................................... 23

3.3.3 Deploying IOCs ....................................................................................................... 24

3.4 Research gap analysis ................................................................................................... 25

Chapter 4: Research method ....................................................................................................... 29

4.1 Introduction ..................................................................................................................... 29

4.2 Design Science Research Methodology ........................................................................ 29

4.2.1 Problem identification and motivation ..................................................................... 30

4.2.2 Define the objectives for a solution ........................................................................ 31

4.2.3 Design and development ........................................................................................ 31

4.2.4 Demonstration ......................................................................................................... 31

4.2.5 Evaluation ............................................................................................................... 32

4.2.6 Communication ....................................................................................................... 32

Chapter 5: Design of TIDS process .............................................................................................. 33

5.1 Introduction ..................................................................................................................... 33

5.2 Design of TIDS process ................................................................................................. 33

5.2.1 Analyze ................................................................................................................... 34

5.2.2 Run .......................................................................................................................... 35

5.2.3 Deploy ..................................................................................................................... 37

5.2.4 Assess ..................................................................................................................... 38

Chapter 6: Implementation and Evaluation .................................................................................. 39

6.1 Introduction ..................................................................................................................... 39

6.2 Testbed ........................................................................................................................... 39

6.3 Datasets ......................................................................................................................... 39

6.3.1 Scenarios CTU-13 .................................................................................................. 40

6.4 Test ................................................................................................................................. 41

6.4.1 Pre-test 1 – DARPA dataset ................................................................................... 42

6.4.2 Pre-test 2 – CTU-13 Blacklisting ............................................................................ 43

6.4.3 Test of alerts on suspicious domain ....................................................................... 44

6.4.4 Test of alerts on any external traffic ....................................................................... 45

6.4.5 Test of vulnerability scan and traffic to unusual ports ............................................ 47

6.4.6 Test of spikes in network traffic .............................................................................. 48

iv

Chapter 7: Result and Discussion ................................................................................................ 49

7.1 Introduction ..................................................................................................................... 49

7.2 Pre-test ........................................................................................................................... 50

7.2.1 Pre-test 1 DARPA dataset results .......................................................................... 50

7.2.2 Pre-test 2 CTU-13 dataset results .......................................................................... 50

7.3 IDS Precision .................................................................................................................. 50

7.3.1 Test result of alerts on suspicious domain ............................................................. 50

7.3.2 Test result of alerts on any external traffic ............................................................. 51

7.3.3 Test result of vulnerability scan and traffic to unusual ports .................................. 55

7.3.4 Test result of spikes in network traffic .................................................................... 57

7.3.5 General results ........................................................................................................ 59

7.4 Design iterations............................................................................................................. 59

7.5 Validity of the design ...................................................................................................... 60

7.6 Answer to research questions........................................................................................ 60

Chapter 8: Conclusions and Further work .................................................................................... 61

References .................................................................................................................................... 62

Appendix ....................................................................................................................................... 65

v

Abbreviations and Acronyms

AMI Advanced Metering Infrastructure

ANN Artificial Neural Network

ASN Autonomous System Number

BSD Berkeley Software Distribution

CIF Collective Intelligence Framework

CPU Central Processing Unit

DAT DATa file. It is widely used for a variety of data content

DLL Dynamic Link Library

DSRM Design Science Research Methodology

HIDS Host-based Intrusion Detection System

IDS Intrusion Detection System

IOC Indicator Of Compromise

IPS Intrusion Prevention System

NGFW Next Generation Fire Wall

NIDS Network Intrusion Detection System

NIST National Institute of Standards and Technology

NSM Network Security Monitoring

OISF Open Information Security Foundation

OTX Open Threat eXchange

TIDS Tuned Intrusion Detection System

URL Uniform Resource Locator

UTM Unified Threat Management

WIDS Wireless Intrusion Detection Systems

vi

Tables and Figures Page

Table 1 IDS evaluating criteria 3

Table 2 Categories of Indicators 7

Table 3 Comparative table of IDS 17

Table 4 Relationship of event categories 20

Table 5 Literature review databases and journal publications 22

Table 6 Literature review references and performance metrics coverage 26

Table 7 Characteristics of the botnet scenarios 40

Table 8 Scenarios in CTU13 41

Table 9 Pre-test 1 42

Table 10 Pre-test 2 43

Table 11 Test of suspicious domain 44

Table 12 Test of external traffic 45

Table 13 Test of unusual port 47

Table 14 Test of spikes in network traffic 48

Table 15 Summary of conclusions 49

Table 16 Iteration 1 of alerts of any external traffic 52



Table 19 Iteration 1 test of vulnerability scan and traffic to unusual ports 55

Table 20 Correlation between spikes in network traffic and alerts 58

Figure 1 Pyramid of pain 1

Figure 2 Cyber Kill Chain, Recorded Future Inc 4

Figure 3 IOC lifecycle 7

Figure 4 Mandiant IOCe 10

Figure 5 IDS and IPS comparison 11

Figure 6 Architecture of Network IDS 12

Figure 7 Components of a Common IDS 13

vii

Figure 8 Snort architecture 13

Figure 9 HIDS and NIDS deployment example 16

Figure 10 Investigative Loop Using IOCs 24

Figure 11 DSRM process 29

Figure 12 Design architecture for POC and formalized process 30

Figure 13 TIDS process 33

Figure 14 TIDS implementation checklist 34

Figure 15 TIDS process step 1 34


Figure 17 Snort rule architecture 36



Figure 20 TIDS process step 1 and 2 39

Figure 21 Registered rules/External IPs 52

Figure 22 Plotted external IP addresses 54

Figure 23 Registered rules/traffic 55

Figure 24 Scenario 1 Botnet capture 57

Figure 25 Scenario 1 Malware-CNC alerts 57

1

Chapter 1: Introduction

1.1 Introduction

There has always been a conflict between usability and security, and if a system should be completely secure no one would want to use it since all implemented security controls and policies would make the system too complex. This balancing has always been an issue when IT-systems have been designed and the optimal security implementation is one that is totally transparent but still protects the users and their data communications from malicious attackers.

A widely used, and well explained mitigation process, is the Pyramid of Pain [1]. It’s a diagram that illustrates the relationship between the types of indicators that may be used to detect an adversary’s activities and how much pain, how troublesome, it will be for the adversary when you, as an IT-security analyst, are able to deny those indicators to them.

Figure 1 Pyramid of Pain. Inspired from [1].

The pyramid is useful whenever new intel regarding an adversary is received. The question:

“is there anything in this intel that can be used to detect the adversary’s activity, and where does this fall on the pyramid”

should then be asked and when all intel is examined. With the conclusion of this question, the indicators of compromise (IOCs) should be created from the pyramid as high up as possible to make the adversary feel as much pain as possible.

The IOC is either created manually, by an IT-professional after a forensic investigation or proactively to meet an uprising threat, or automatically by a gathering software, and its definition is [2]:

“a piece of information that can be used to identify a potentially compromised system”

Therefore, it can be used as a source when rules for IDS are created to identify some type of known malware or utility, either by looking for attributes of the binary, itself, or for some artefact created upon execution, for example, an IDS rule to identify any service-DLL that was not signed, and which was loaded from any directory that did not include the path "windowssystem32".

In this research, the concept of “parsing an IOC”, is used in the formalized process that this research will present, and it can be defined as:

2

“The way to interpret the information given from the IOC and combine it with specific knowledge of current system to write an IDS rule of high quality”

There are several types of codifying the IOCs, and these typically are decided by the objectives of the investigator. Some of the most common use cases are [3]:

• Find malware/utility, is the most common use case. This is an IOC written to find some type of known malware or utility, either by looking for attributes of the binary, itself, or for some artefact created upon execution, such as a prefetch file or registry key.

• Methodology IOCs find things you don't necessarily know about, in order to generate investigative leads. For example, if you wanted to identify any service DLL that wasn't signed, and which was loaded from any directory that did not include the path "windowssystem32", you could write an IOC to describe that condition.

• Bulk, many organizations subscribe to threat intelligence feeds that deliver a list of MD5s or IP addresses. These kinds of IOCs are very generic and are typically only good for an exact match.

• Investigative, when evidence is found of malicious activity such as metadata related to the installation of backdoors, execution of tools, or files being staged for theft, you can track that information in an IOC.

Intrusion Detection is a very well used method in the fight against malicious attackers and have been used in the IT-field since the mid-’80s [4]. This has led to make the concept intrusion detection system a well-established field of research [5]. For example, the work of [6] showed the various opportunities and design decisions that need to be considered for distributing detection devices across an industrial communication network. [7] identified the parameters that need to be monitored in Advanced Metering Infrastructures1 (AMI) for detecting cyberattacks. In their later work, [8] developed a specification-based intrusion detection sensor for AMI.”

As the cyber-attacks gets more advanced over time, the importance of studying IDS gets more and more significant since an IDS has the ability to examine the payload of a data packet, compared to a regular firewall that checks specified fields in the packet header. An Intrusion Detection System (IDS) is a system that can identify malicious network traffic from the normal and legitimate traffic on an individual computer or on a network [9]. It can be deployed in different ways:

• Host-based Intrusion Detection System (HIDS) is located on individual computers installed with the operating system enabling encrypted traffic to be analysed.

• Network Intrusion Detection System (NIDS) is located on a network either as a dedicated device or loaded onto other computers [10].

There are several types of different IDS-architectures to implement. In this research, Snort® will be used and there are three main detection methods[11]:

• signature-based, the oldest form of intrusion detection. It searches for matches and specified patterns, like IP-addresses, text strings, or more complex patterns like a specific string in combination with execution of a specific protocol.

1 Advanced metering infrastructure is an architecture for automated, two-way communication between a smart utility meter with an IP address and a utility company.

https://searchwindevelopment.techtarget.com/definition/IP-address

3

• policy-based, detection is based on a security policy which means that an alarm is triggered

when there is out of policy traffic in the network. An example can be use of non-secure

protocols like Telnet or FTP.

• anomaly-based detection, relies upon observing network occurrences and discerning anomalous traffic through heuristics and statistics. This detection method has the ability to recognize attack patterns that deviate from normal network behaviour, which is very powerful but more difficult to implement.

1.2 Research Problem

If an IDS implements rules created from bulk-IOCs or IOCs gathered from a poorly tuned automated solution, the IDS may create a large number of false positives which impacts both network resources as well as IT-personnel when screening large logfiles to determine which threats are real or not. The issue that creates this problem is that many of these bulk-IOCs are generic, in the sense that they shall suit any system that implements them, and therefore can be a bit wider when, for example, determine which ports are allowed or not for specific network traffic.

When you measure the performance of an IDS, you can look at several IDS evaluation criteria’s [12],[11]:

Table 1 IDS evaluation criteria.

FPR False Positive Rate FRP = FP/(FP+TN) Normal traffic identified as an attack

FNR False Negative Rate FNR = FN/(TP+FN) Attack traffic identified as normal

TPR True Positive Rate TPR = TP/(TP+FN) An attack correctly identified

TNR True Negative Rate TNR = TN/(TN+FP) Normal traffic correctly identified

Accuracy = (TP+TN)/(TP+TN+FN+FP)

Precision = TP/(TP+FP)

To be able to talk about an IDS precision, it’s important to identify the true positives and false positives. These different criteria’s must be measured and calculated and to be able to do so, it’s crucial to state what normal traffic is, which is different from organization to organization.

Since every network has unique features, in regards of infrastructure and data flows, that need to be considered when using IOCs. We will examine if we can improve, by parsing the IOCs to IDS rules, the precision of indication of malicious activity, either when performing forensic analysis or implementation of the rules in an IDS.

One of the problems when managing an IDS in an organisation is to have time to process all the alerts created. Is it a true positive and how serious is it? Triaging and analysing takes a lot of time and effort. That is why the percentage of true alerts of the total amount of alerts, i.e. the precision, is such an important metric.

4

1.3 Research question

Research question 1: How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise?

Research question 2: How can activities and procedures be formalized into a process for managing rules for an IDS?

1.4 Proposed solution and research goals

This research will focus on the impact of an IDS implementing rules with higher precision, parsed from Indicators Of Compromise and will result in an assessment whether:

• Use of rules from parsed IOCs raises an IDS precision

• It is cost-effective to parse IOCs

Apart from the above assessment, the research will also present a formalized process of how to implement and manage rules for an IDS.

1.5 Research problem domain

This research’s problem domain is as we researchers see it, a small to medium organisation network, with sensitive data transmissions that needs to be protected against malicious adversaries. An IDS can be implemented both as a HIDS as well as a NIDS, but this research will only focus on a NIDS implementation scanning recorded data, but both concepts will be explained further in chapter 2.3.4.

If the research provides findings that it is efficient to tune IOCs and IDS rulesets, it should be a valid contribution to every type of network, adjusted to their specific demands.

1.6 Research contribution

This research contribution is connected to the organizational aspect that all organisations have limited resources, and the subject is important to study because the outcome will help organisations to allocate their resources in a more optimal way.

By having real-time insight and context on a particular system, the IDS can be fine-tuned. This enhance the precision and enables the organization to “move left” in the Cyber Kill Chain and thwart an attack at an earlier stage, limiting its impact and cost. To achieve this, we will present a formalized process, named TIDS, that will guide an IT-security professional in tuning and implementing rules in an IDS for the purpose of higher security in a system.

Figure 2 Cyber Kill Chain. Inspired from [13].

5

1.7 Research delimitations

The study covers improving the precision of an IDS by parsing IOCs to write IDS-rules. Since there are several different IOC frameworks and this research is limited in time and resources, it will only cover OpenIOC, which is one of the most widespread and has been used in previous research [14].

There are several sources for recorded network traffic but since this thesis is limited regarding time aspect, we will only use one dataset, the CTU-13, created by [15] in 2011, which in sense of open dataset for research use, is fairly modern.

We will use Snort as IDS since it is the most commonly used IDS today and therefore a useful choice [16], [17].

1.8 Structure of thesis

This thesis is structured in the following manner:

Chapter 1 Introduction, contains an introduction to the thesis including research problem and questions, proposed solution and research goals, contribution and delimitations.

Chapter 2 Background, describes the background of intrusion detection systems, their architecture and rules in the form of indicators of compromise.

Chapter 3 Literature review, covers the literature review that forms the theoretical basis for this thesis.

Chapter 4 Research method, describes the selected method, Design Science Research Methodology (DSRM) connected to the research question.

Chapter 5 Design of TIDS process, the tools used and the configuration of them in order to improve IDS precision are described in this chapter.

Chapter 6 Implementation and Evaluation, contains the setup for experiments

Chapter 7 Result and Discussion, contains the results obtained by running the experiments.

Chapter 8 Conclusions and Further work, Summary, analysis and conclusions of the conducted experiments and discussion about limitations and possible areas for further research.

Chapter 9 References

Appendices, contains appendices to this thesis.

6

Chapter 2: Background

2.1 Introduction

This chapter contains the theoretical background and clarifies different concepts regarding IOCs and IDS.

2.2 Indicator of Compromise

NIST states in their publication NISTIR 7298, Revision 2, that the term compromise stands for:

“Disclosure of information to unauthorized persons, or a violation of the security policy of a system in

which unauthorized intentional or unintentional disclosure, modification, destruction, or loss of an

object may have occurred.”

and that the term indicator stands for:

“a sign that an incident may have occurred or may be currently occurring.”

If we combine these two terms we can endorse the definition of indicator of compromise as stated by Rudman and Irwin [2]:

“a piece of information that can be used to identify a potentially compromised system. It could include

suspicious IP addresses, domain names, email addresses, file hashes or a file mutex”

With this understanding of the concept we can examine a system and log files for indicators in order to protect our data. This can be performed during different stages of the security work: proactively, retrospectively, and during an ongoing attack. Koen [18] describes the different stages as:

• Proactively: Having the knowledge of what IOCs are out there can help develop defense methodologies to prevent new malware infections. These can be used to develop signatures, set firewall rules, and improve defensive mechanisms. A system can be protected from malicious locations and prevent malware communications with command and control by proactively blocking network locations.

• Retrospectively: information from IOCs can be used to deny potential threats the ability to gather any information during the reconnaissance phase of the planned attack. This information also helps us to identify threats during the malware delivery, exploitation, and installation phases, and provides important opportunities to deny attackers the ongoing access they desire.

• Incident Response: Incident responders can use IOC information to aid in threat triage and remediation. IOCs help to identify specific strains of malware and provide invaluable information for responders. This information enables responders to better understand how threats were able to infiltrate their systems. By knowing what IOCs to look for, we can quickly and efficiently locate and resolve any damage the threat may have caused to our system and our users.

7

From its presence or specific attribute, IOCs can be divided into groups. For example:

• Atomic IOC, are elements of data that can’t be broken down further, like a host name, IP-address, email-address, file name etc.

• Computed IOC, are md5 hash, regular expressions and other computed data.

• Behavioural IOC, are the signature of an attack. It can be comprised of several atomic IOC as well as behavioural IOCs.

To further categorize the different types of IOCs, they can be divided in Host-based and Network-based. Certainly, there may be an overlap between the two categories and that should be noted as a correlation.

Table 2 Categories of Indicators.

Host Based Indicators Network Based Indicators

Registry key IP address

File name Domain name

Text string (CC, SSN) Text string (SQLi, CC, SSN)

Process name X500 Certificate hash

Mutex Communication protocol

File hash File name

User account URL

Directory path

2.2.1 The IOC lifecycle and Tuning process

The IOC lifecycle consists of a number of steps according to [26]. First, there is an Initial Trigger in the form of an attack, a change of policy, change in the network or something else. It triggers an Analysis of data and then a Creation and/or Parsing of IOCs. Within the Creation or Tuning of IOCs there are several steps.

Figure 3 IOC lifecycle. Inspired from [26].

8

Creation or Tuning

• Putting the IOC into a context.

o A collection of public IOCs does not in itself give a conclusive information on how they will affect a particular network. An IOC sometimes needs to be translated into a more useful format. For example, an IOC can consist of ten randomly named new *.cab files in a temp directory. It will not be an efficient rule to list the ten individual file names as rules. It is better to rephrase the IOC to look for any new *.cab files in the temp directory.

o For e-mail attacks, fields like “from” “delivery method” and “e-mail content” can be recognized as IOCs but they can easily be manipulated and randomized by an attacker and it is not wise to rely solely on those rules.

o Blocking individual IP numbers in a black list file may not be efficient and should rather be applied by a policy block list from a trusted source, since infected IPs are subject to constant change.

Deploying

• Some IOCs with information valid only for applications or functions that is not used in a particular system are not relevant to apply and are only slowing down the IDS performance.

Identifying

• When the alarms start to go off, affected systems are identified, data is collected preferably in a centralized log function and an analysis can start in order to create or tune the IOCs. Here is also where the chain of IOC tools needs to be analysed and optimized. There might be a need for changing and updating tools, integrations, settings etc.

In an ideal world there would also be a Share data or Share IOCs in the IOC lifecycle. One of the problems is the reluctance of organisations to share their datasets with others for confidentiality and proprietary reasons and IOC sharing is also difficult due to a lack of standards.

2.2.2 IOC Framework

There is no accepted standard format for the IOC yet [2]. There are several different frameworks like STIX, Yara and OpenIOC etc. This research is performed by the use of the OpenIOC-framework.

2.2.2.1 OpenIOC

OpenIOC is created by the company Mandiant and released to the public as an Open Source project under the Apache 2 license in November of 2011, but OpenIOC had been used internally at Mandiant for several years prior. OpenIOC provides the ability to use logical comparison of Indicators in an IOC, providing more flexibility and discrimination than simple lists of artifacts. An extensive library of Indicator Terms allows for a variety of information from hosts and networks to be recorded, and if something is not covered in the existing terms, additional terms may be added.

In their release of OpenIOC at GitHub, [19] states the following:

“OpenIOC is usually used to generate Xpath expressions to query data sources, so OpenIOC 1.1 operators behave as they would in Xpath 2.0. OpenIOC now supports regular expressions with the

"matches" operator.”

9

In version 1.1 of OpenIOC, the section Parameters has been added to the schema. This section is a rather big enhancement since it gives the editor of the ioc the possibility to, for example to apply application or organization specific logic that can then be passed along with the IOC.

A brief description of an IOC under OpenIOC 1.1 identifies three distinct sections [19]:

1. Metadata - the traditional metadata header that contains metadata about the entire Indicator

2. Criteria - the "matching" section, a boolean logical evaluation that determines whether or not you have found malicious event or artifact, as defined by this specific indicator.

3. Parameters - are assignable metadata, that can be applied to any element in the Criteria section of the IOC. They may be used for linking one indicator to another or to be used to test an idea before altering the xml-schema.

Examples of the three sections:

<?xml version="1.0" encoding="utf-8"?>

<OpenIOC xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-

instance" id="04f522c1-efc9-492a-a07f-25501f395255" last-modified="2013-07-27T23:13:42Z" published-

date="0001-01-01T00:00:00" xmlns="http://OpenIOC.org/schemas/OpenIOC_1.1">

<metadata>

<short_description>WideCharTextExample</short_description>

<authored_date>2013-07-27T23:10:56Z</authored_date>

<links />

</metadata>

<criteria>

<Indicator operator="OR" id="dad3e9d8-940d-4e48-9b2c-471170472476">

<IndicatorItem id="4e0a03af-25bb-4451-8662-8db20c45e475" condition="contains" preserve-

case="true" negate="false">

<Context document="Yara" search="Yara/TextString" type="yara" />

<Content type="string">Borland</Content>

</IndicatorItem>

</Indicator>

</criteria>

<parameters>

<param ref-id="4e0a03af-25bb-4451-8662-8db20c45e475" id="29bfde48-a800-4f5f-9169-

6225bf6df59b" name="yara/wide">

<value type="bool">True</value>

</param>

<param ref-id="4e0a03af-25bb-4451-8662-8db20c45e475" id="09c52038-e5d8-4042-8a47-

1a38de364043" name="yara/ascii">

<value type="bool">True</value>

</param>

</parameters>

</OpenIOC>

10

The company Mandiant, later acquisited by FireEye2, have created a tool, IOCe, which is an editor that provides a user interface which makes it easy to read or edit the IOC. You can easily add AND or OR sections to further evolve the IOC and the image below shows an example.

Figure 4 Mandiant IOCe.

2.3 IDS

Intrusion detection systems are usually classified in two ways, the selected detection method(s) and deployment(20). This chapter describes Intrusion Detection Systems in terms of difference from Intrusion Prevention System, architecture, performance and deployment. It also describes Indicators Of Compromise in general with OpenIOC as an example.

2 https://www.fireeye.com/

11

2.3.1 IDS architecture

An Intrusion Detection System (IDS) is a system that can identify malicious network traffic from the normal and legitimate traffic on an individual computer or on a network. It can identify attacks by analysing network traffic and respond to them, for example by creating an alarm. An Intrusion Prevention System (IPS) can identify attacks and respond by creating alarms and by stopping the attacks. Both types of systems use a database of well-known threats towards which the network traffic is compared. Both can be deployed as network based or host based. One could think that an IPS is better than and IDS since it stops attacks, but in fact the two systems complement each other. An IDS can be used to verify that an IPS is functioning and the IDS can create warnings about suspected malicious traffic that can be acted upon, without having the IPS to stop all the traffic. An IPS can also be a bottleneck in regards of capacity if all the traffic is directed through it, while an IDS is more passively listening in [9].

Figure 5 IDS and IPS comparison [20].

This is of course a generalized description of the systems. Although unusual, an IDS could be placed in-line and have a blocking functionality while an IPS could be set in a non-blocking mode. An IDS usually needs other supporting equipment like switches and firewall in order to create a robust network security solution.

The volumes of data packets in a network can be huge but decreases when passing through the different levels of an IDS, as indicated by the green arrows in the picture below [21]. Since only a small portion of the packets are malicious, detection mechanisms and algorithms need to be applied to the IDS and a hierarchical approach is used where packets from certain services are detected and

12

selected by a sniffer and filter function. Examples of open source filters are Tcpdump3 or Wireshark4 that work on the packet level.

The level above sniffer/filter is where the event engine checks the packet headers, for example checks that the IP header checksums are correct. An example of a security-related event can be a failed login, if it has been defined as such in the event definitions.

Figure 6 Architecture of Network IDS. Inspired from [21].

Finally, in this hierarchical approach, the detection rules control what events will generate alerts. Detection rules can be expressed in the form of Indicators of Compromise.

A complete security solution like an Intrusion detection system work on the application layer of the TCP/IP stack (and the corresponding levels of the OSI model) since it handles session management and applications that are executed. It is, however, capable of detecting malicious traffic according to specific implemented rules on all the different levels of the protocol stack.

From the alert management perspective, an Intrusion Detection System can be described as follows: An IDS has a Sensors part that detects suspicious events. The data from the sensors are sent to Analyzers, Incident Storage and a Response Unit. When an alert is sent to the Analyzers and is deemed to be a sign of a malicious activity, incident information is sent to the Incident Storage for update and the Response Unit for further actions in the form of Reactions to Events [22].

3 https://www.tcpdump.org/ 4 https://www.wireshark.org/

13

Figure 7 Components of a Common IDS. Inspired from [22].

2.3.1.1 Snort architecture

Snort is selected for this thesis because it has the largest market share, it started as an open source tool and it is still free of charge even if some extra features come at a cost. It is a user-friendly IDS tool and has support for different host platforms and can run as both HIDS and NIDS [16]. It can be run in three modes, sniffer mode, packet logger mode and network intrusion detection mode [21].

The Snort architecture consist of a number of modules. Libpcap for Linux or WinPcap for Windows is used for capturing network traffic. The packet decoder engine converts the traffic to data structures and identifies what protocols are used (UDP or TCP). If there is something wrong with the packet header or length, an alarm will be created. The pre-process tries to find potentially malicious activity. The detection engine uses the detection plugins which in turn contains the rule files. The rule fines are text files with the detection rules stated according to a specific syntax. The rule files are updated regularly to keep track of the current threats [21].

Figure 8 Snort architecture. Inspired from [21].

14

2.3.2 IDS detection methods

There are three main detection methods used in IDSs; signature-based, policy-based and anomaly-based detection.

• Signature-based detection means that the IDS reacts on matches against strings of bytes contained in a database and kept update by a system administrator.

• Policy-based detection is based on a security policy which means that an alarm is triggered when there is out of policy traffic in the network. An example can be use of non-secure protocols like Telnet or FTP.

• Anomaly-based or behaviour-based detection can be used in two ways. One is statistical anomaly detection where the traffic is recorded over a period of time and the traffic is compared to the normal baseline traffic. There is also non-statistical anomaly detection where a system administrator can define beforehand what normal traffic is. These methods are hard to use in networks with great variation in its traffic. They will generate a lot of false positives. Anomaly based detection methods are in general weak in defence against previously unknown vulnerabilities [23].

Today, there is rarely just one IDS in a network. It is rather a system of distributed IDSs that needs to collect and fusion data from multiple systems. Alert fusion is therefore an emerging field [24]. Fusion can be made by combining the data from multiple distributed IDSs in a network. It can also be done by combining different detection methods like signature-based detection and anomaly-based detection, for example by fusing Snort alerts with Packet Header Anomaly Detection (PHAD).

Distributed IDS are very complex and that is why random theory and fuzzy logic are being applied. Fuzzy logic is based on evidence theory that combines historic data with data from different sources in order to calculate the probabilities of an attack. Support for fuzzy logic can be found in common open-source IDSs like Snort®. [24] have shown that alert fusion has a strong positive effect on intrusion detection performance.

IDSs are challenged when subject to yet unknown and unforeseen threats. Deep learning approaches are increasingly used for countering those treats and implementation in a NIDS can be made in three steps,

1) Learning features from unlabelled training data

2) Training with labelled data

3) Testing with real network data

Typical metrics used for assessing the performance of self-learning IDSs are Accuracy and Precision as defined earlier, but also Recall, which is the % ratio of True Positives divided by the sum of True Positives and False Negatives.

Recall = TP/(TP+FN) [25]

Artificial Neural Networks can be found to identify significant intrusion threats [26]. The intrusion detection systems within ANN use machine learning algorithms. They have different kinds of learning systems [27].

• Supervised learning means that the system is provided with a training set of data that is labelled.

15

• Unsupervised learning means that the system gets an input of data and creates its own labels from the data.

• Reinforcement Machine Learning means that in the absence of training data, the system

chooses an action, at a given data point and trains itself through trial and error.

Algorithms can be combined for higher efficiency and performance. It is positive for ANNs if the input data comes from of a variety of attacks. Machine learning is a large research field of its own and will not be covered in this research [26].

Other emerging fields are modern data mining which can be used for analyzing large amounts of network and attack data in order to find patterns and make forecasts. Game theory is another relevant field. It is a mathematical method that has been used for many years and in many research areas within information security. The aim is to predict the behaviour of the agents (attackers and defenders) who influence each other reciprocally. In practice game theory could be used, for example, to optimize security resource allocation in an organisation [28].

2.3.3 Tuning of IDS

Apart from parsing the IOCs to create rulesets that the IDS can implement, you could also tune the way the IDS should perform. The same technology that is used to develop IOCs could be used by the system to identify which rules are relevant to the current environment and therefore enable/disable rules as necessary. An example would be if no BSD systems were in the environment, then the system may disable BSD related signatures. This reduces the system overhead and lowers the chance for false positives that may be triggered based on an unnecessary signature. Due to changes that occur in the network over time, a function like this should be run with recurring intervals to ensure that a current network profile is used as the basis for the operating ruleset [29].

2.3.4 IDS deployment

An IDS can be deployed in different ways and that there are different implementations for different types of networks. To clarify this topic, we can divide it into two main aspects where the first addresses the common placing of an IDS or its sniffers into the network infrastructure and the second addresses the different implementation-types for different network with different purposes, like critical infrastructure, smart grids, WiFi, or cloud environment.

A common placing of an IDS or its sniffers is often defined in two main categories:

16

Figure 9 HIDS and NIDS deployment example. Inspired from [9].

Host-based Intrusion Detection System (HIDS) is located on individual computers installed with the operating system enabling encrypted traffic to be analysed. The drawbacks are high cost for purchase and maintenance and if the computer breaks down, the attack database will be gone. A Network Intrusion Detection System (NIDS) is located on a network either as a dedicated device or loaded onto other computers. It is less expensive and less demanding in regards of maintenance compared to a HIDS, on the other hand it will be unable to analyse what goes on within the operating systems of the clients [10]. Today, network based and host-based solutions regarding intrusion detection and prevention are commonly deployed together in the same network. In the following figure an example is shown where one can see that a NIDS is typically placed in a more passive position on the side while the HIDS is placed at each investigated computer.

After we’ve addressed the difference between NIDS and HIDS we need to understand that networks with different purposes needs different IDS implementations. Deployment will differ depending on the type of network, for example, Critical Infrastructures, Smart Grids, WiFi, and Cloud environment. What all different network types have in common is the structure regarding the placement of the IDS and other sensor systems. It always depends on where the highest risk of attacks will come from. Is it mostly inter-LAN traffic that is of interest or is it attacks from the outside via Web, DNS and e-mail servers that are prioritized?

Other aspects to take into consideration are:

• Classification level of monitored resource

• Network design

• System throughput

• Personnel time (for management and analysis)

• Resource availability

This research will focus on IDS in small or medium generic networks due to limited research time and laboratory environment. We will implement and evaluate the NIDS deployment since we have chosen to work with Snort which is most common to run as a NIDS. In a complete security solution there would be both NIDS and HIDS deployed since there are differences between how they work. A HIDS is often a collection of applications installed on a host, and together controls activity within the operating system, while a NIDS is only scanning data traffic to or from the host in the network.

17

2.3.5 IDS comparisons

Summary of features in current IDS [30]:

Table 3 Comparative table of IDS [30].

Parameter Anomaly-/Heuristic-/Behavioural based

Signature-/ Knowledge-/ Pattern based

Host based Packet-/ Network based

Example Firestorm, Bro,

Dragon

Suricata, Prelude Dragon Squire, Real Secure

Network Flight Recorder, Snort, Cisco secure ID, Tripwire

Information sources from computers

Statistics, Expert systems, Neural Networks, Compute Immunology

Expert systems, Signature analysis

Accounting, Syslog, C2 security audit

SNMP information, Network packets

Installation and deployment

Typical Mixed Immediate Easy

Throughput Max Moderate Max Moderate

IPS capability Yes No No Yes

User friendliness Less Yes Less Yes

Performance OS independent OS dependent OS dependent OS independent

Identification of unknown attack

Yes No No No

Maintenance required

Comparatively less Needs frequent updating

Comparatively less Comparatively less

From this table one can see that the IDSs have different strengths and weaknesses. For example, Signature-/Knowledge-/Pattern-based IDS have a very low network usage but requires more resources in regards of updating and offers no protection against unknown attacks.

2.3.6 Performance measurement

The performance of IDS in a wider sense depends on many factors. Like [31] argues, if the security level in general is low in an environment and the appropriate defense-in-depth strategy with multiple layers in security are not applied, an IDS optimized for performance will not be of help regardless of how sophisticated the solution is. Similarly, if the alarms generated by the IDS are ignored or if you detect and act on all anomalies, false as well as true positives, the use of the IDS will not be optimal. Current research mentions these background factors but concentrate mainly on the performance of the IDS itself.

There are research work focusing on the performance of different Intrusion Detection Systems, comparing different modern systems like Snort, Suricata and OSSEC [16], [32], [22], [33], [34], [30].

K. Thongkanchorn et al. [35] have tested the different IDS on the same machine platforms, CentOS v,5, used four different network traffic generators tools and have used an identical ruleset for Snort and Suricata. Bro has a different architecture and could not use an identical ruleset. They have replayed the tcpdumps at different traffic rates, from 50 to 2000 pps. used normal and malicious traffic, eight types of attacks (DNS, DoS/DDoS, FTP, ICMP, POP3/IMAP, SCAN, SNMP, Telnet/SSH) and different rulesets. They have shown that certain types of attacks, especially TCP, give low packet loss and low CPU usage on all IDSs. High traffic rate has a strong effect on packet loss, generated alerts and CPU usage. The set of rules also effects the number of alerts significantly. According to the Snort ® User’s Manual 2.9.1.1[36] Snort is optimised for the TCP, UDP, IP or ICMP

18

protocols so it is perhaps not so surprising that it shows better performance analysing such packets compared to packets of other protocols.

V. Shah et al. [24] have compared Snort and Suricata on a fixed 10Gbps network speed using learning algorithms and fuzzy logic. The network used CentOS 6.5 machines and Kali Linux Metasploit Framework. The result showed that Suricata had a 10-15% higher CPU usage than Snort at that speed due to the multi-threaded architecture. In the beginning there was a high false positive rate for Snort. When using legitimate and malicious traffic the FPR for Snort was 55,2% and for Suricata 73,4%. The FNR for Snort was 6,7% and for Suricata 16,7%.

The figures were improved when using a hybrid approach with Snort adaptive plugin and learning algorithm “Firefly” and the false positive rate was reduced to 8.6% and the false negative rate was 2.2% which the authors thought was a good result.

V. Nagadevara [17] has compared Snort, Suricata and OSSEC on a virtualized environment using Ubuntu over Virtual box on a Macbook MD101 hardware with packets per second ranging from 100 to 50 000. The studied metrics are CPU usage, packets lost, and alerts created. The research showed that the packet loss with normal traffic is insignificant for packets per second <= 2000. One of the tests focused on the number of alerts and precision per studied attack type (DOS, FTP, ICMP and SCAN). The results indicated that FTP attacks in general had the highest precision rate (around 90%) while ICMP and DOS had the lowest (65-80%). OSSEC performed the best, on second place Suricata and then Snort. The question is if comparisons with OSSEC are possible to make because OSSEC is host based only and usually as such has a high performance compared to network-based IDS.

W. Park and S. Ahn [16] have made a comparative study between Snort and Suricata where tests run over Security Onion comparing single core and multi-core. On a single thread, Snort had 27% full and partial detection while Suricata had 77%. On multi thread, Snort had 16% full and partial detection while Suricata had 73% full and partial detection. How partial detection is defined is not clearly explained in the research. The authors claim that Snort is a good choice if CPU resources are scarce. When looking at the %-age of full detection, however, there is no difference at all between the single thread and multi-thread results. On a single CPU, Snort had a 5% higher CPU share than Suricata. On multiple CPUs, the situation was the opposite where Suricata benefits from multi-threading which consumes more CPU resources.

J. S. White et al. [32] have tested the performance of Snort vs Suricata in a hardware environment consisting of an AMD 8439 Opteron class processor. They have detailed descriptions of the server specifications including RAM, CPU, cache, disc etc. The number of CPU cores and workload were varied during the tests. The research shows that if Snort is deployed in a multi-instance configuration, Suricata beats Snort by 9 299 packets per second at 24 cores and by 64 420 at 6 cores. They claim to have tested Snort and Suricata with their out-of-the-box settings. All installations, however, require adaptions to the host on which it is run, choices in parameter settings, changes to configuration files, import of rules database etc. so it is debatable whether there actually exists a single out-of–the-box installation. The choice of ruleset was the ET-Open (Emerging Threats Open Ruleset which is compatible with both IDSs. The ruleset comes in a Free and a Pro version and both have been evaluated. The researchers have found that the choice of the ET Pro rules gives a higher capacity in Kilo-Packets Per Second than the Free rules. The Free rules were good, but the Pro rules were better since they are optimized for high capacity. Writing of custom rules requires high skilled staff to avoid poorly written rules that put unnecessary strain on the system. Performance in terms of precision was not covered in this research.

19

When comparing different IDS, some criteria are typically mentioned by R. Dewanjee and R. Vyas (20):

• Installation and deployment – if it allows standard/custom installation, if it is easy or complex

• Throughput – anomaly and host-based IDS have better throughput than signature based (example Suricata) and packet/network based (example Snort) IDS.

• Network usage – how much of the network capacity is consumed by the IDS. Network IDSs typically create higher load on the network.

• IPS capability – If it is capable of prevention as well as detection.

• User friendliness – if the IDS is easy to use. Host based IDSs are deemed less user friendly.

• Identification of unknown attack – if it also protects from new and unknown intrusions

• Maintenance required – less or more, frequent or infrequent

• Performance – Packet/Network-based and Anomaly based IDSs are independent of the OS on which it runs, in regards of performance.

The performance criteria can be divided into more subcategories, such as (20), [37]:

• Performance – technical capacity like number of alerts, packets lost and CPU usage

• Scalability – how well the IDS works in different sized networks with varying workload

• Features – unique functionality in the IDS solution

• Precision – how well the IDS identifies intrusions, known as well as unknown, with as low rate of false positives and false negatives as possible and as high rate of true positives and true negatives as possible.

The performance criterion used in this research is precision and other criteria are not covered in this research. The IDS comparisons in current research have focused mainly on varying the datasets (live recordings, public databases) workload (amount of data, speed) and/or configuration of the IDSs, but few researchers have continued beyond the out of the box settings of the rulesets or IOCs. Research has shown that changing the set of rules may impact the performance of the IDS in a major way. Especially non-optimized and poorly written ones have a potentially negative impact on performance [37]. The ability to classify the network traffic and the precision of the rulesets or IOCs are two of the most important factors influencing the performance of IDSs [38].

20

2.4 Tuning of IDS rules

IDSs have differences in performance, but they can be tuned in order to achieve higher precision measured in terms of true and false positives and negatives.

Table 4 Relationship of event categories [39].

Positive Negative True True Positive:

Alerted on intrusion attempt

True Negative: Not alerted on benign activity

False False Positive: Alerted on benign activity

False Negative: Not alerted on intrusion attempt

It may seem like a contradiction why it is a problem with many false positives since the IDS of today have a high capacity of processing data. If there are many false positives, they tend to be ignored in the long run because management of positives require a lot of resources. The intended function is to have a high rate of true positives and true negatives while false positives and false negatives are a sign of a broken function. To achieve a high percentage of true positives and true negatives is not something that is done once and for all, instead it is an ongoing task of finetuning the IDS rule sets with input from real network traffic from the system itself and from external sources [39].

Whether the tuning should be automatic, or manual depends on what the tuning purpose are. Examples of automated tuning could be to remove stopwords after IOCs have been collected. A stopword is a term or token with no semantic meaning like “a”, “an”, “the” etc. Another measure that suits to be automated is called stemming and it minimize the size of the feature set by removing misspelled words or words with the same stem. There are algorithms called stemmer, which removes words with the same stem and keeps the stem or the most common of them as feature. For example, the words "download", "downloads", "downloading" and "downloaded" can be replaced with "download" [40].

Another possibly automated way to keep rulesets up to date is to implement the set of signatures that is updated regularly and provided by many of the IDS vendors as part of their solution. These signatures typically are broken into categories of signatures based on the behaviour they are designed to detect. Each signature has some form of a validity rating that identifies how likely it will generate a false positive result. New signatures are typically released when a new attack or vulnerability emerges, and the vendor’s research team develops a high-quality signature. High quality signatures are typically automatically enabled when updated and low-quality signatures require manual activation [29].

As well as IOCs mostly is gathered automatically, they can be developed or tuned internally. To be able to do so, it demands knowledge of the local network and connected hosts. [26], suggests looking at a number of internal sources of IOCs. Logs like web proxy and DNS logs should be examined. Irregularities such as unexpected spikes in network traffic, port scans and connection attempts using unusual protocols or ports should raise alerts. By knowing the normal behaviour and use it as a baseline, the abnormal and malicious behaviour can be detected more easily.

To implement this, the IOC can be tuned into a localized and more effective IDS rule by, for example, adding different thresholds. Information about the system or network that does not change very often, like IP-addresses and open ports, can also be used for tuning IDS rules. Some network topologies tend to generate more alerts than others, which should also be taken into consideration when tuning the rules [41].

21

One of the most demanding activities in incident response is the process to determine if an incident truly has occurred, and if it is proven to be a true incident, determine its category, scope, and root cause. [39] states what makes this process so difficult is a blending of three elements:

• The detection of incidents occurs through a variety of means, with differing levels of detail and reliability. Detection can be automated or occur through manual means. Automated detection is comprised of such devices as host and network-based intrusion detection and prevention systems (IDSPs), log analyser’s, and antivirus software. Manual means of detection would include a user reporting a problem, or an analyst that is manually reviewing system log data and discovering some anomaly.

• The volume of possible indicators of incidents is normally tremendous for a company. It is not unusual for these companies to have thousands or even millions of intrusion-detection sensor alarms each day.

• The thorough and accurate evaluation of incident-related information requires a deep, expert-level, technical knowledge of all involved technology [12].

22

Chapter 3: Literature review

3.1 Introduction

In this literature review information about the specific domain is described. Aspects of IDS performance including using Indicators of Compromise are covered. A comparative analysis of current research is also included.

3.2 Literature review

Literature review methodology is used in this research in order to form a theoretical base, distinguish what needs to be studied from what has been already been studied and to synthesize previous research with the results of this research in the analysis.

The literature review includes introducing the topic and showing the relevance. It also shows the relationship between this research and the whole field. Finally, it justifies the need for the research and states it purpose.

In the review, the sources have been selected from the criteria of relevance, age and authenticity.

Table 5 Literature review databases and journal publications

Databases

Google Scholar

IEEE Xplore

Publications

IEEE Network

IEEE Security & Privacy Magazine

IEEE Transactions on Smart Grid

Information Systems Research

Journal of the Association for Information Systems

Journal of Management Information Systems

MIS Quarterly

NIST Special Publication

SANS Institute InfoSec Reading Room

The age of the published academic work has been limited to the last three years except for some cases where the work forms a conceptual base for later research, has been frequently referenced by other authors and similar.

23

Keywords that have been used for the searches are "Intrusion Detection"; "Indicator Of Compromise"; "IOC"; "False Positives"; "False Negatives"; "OpenIOC"; "Cyber Threat"; "IDS"; "IDS benchmark"; "high precision IDS"; "Cost of performance IDS".

A number of tech blogs and IOC and IDS vendor sites have also been used as sources, primarily for the design and development phase and onwards in this research work.

3.3 Earlier research

3.3.1 IOC standards

There is no standard format for IOCs today. It would be helpful if it existed, because then IOC sharing would be easier, less time consuming and conversion between formats would not be necessary. Examples of common formats today are OpenIOC, CybOX and STIX. New platforms have, however, emerged where IOCs of different standards can be shared. Examples are AlienVault Open Threat Exchange (OTX)5 and Malware Information Sharing Platform (MISP)6. Similar solutions are needed to be studied and invented in order to enable flexible and automatic sharing of IOCs between organisations as [2] stated. Some standards are said to be easier to convert than others, but the problem remains with incompatibility causing manual handling and problems with IOC sharing according to [42]. The selection depends on what features are most important for an organisation. If you can accept being dependant on a certain commercial product family, then OpenIOC from Mandiant could be a good choice. If not, CybOX which is vendor neutral, could be a better choice [42].

3.3.2 Gathering of IOCs

[14] states that cyber threat intelligence has, over time, been more aggressively collected and increasingly exchanged across organizations, often in the form of indicators of compromise. They also identify the challenge to effective gathering indicators of compromise from large amount of data since automatic collection of indicators of compromise may create false positives.

IOCs can be collected in different ways. From commercial and industry sources, free and open source databases, online tools for IOC sharing, googling and internal generation from analysis on private network. The collection can be done manually or with use of automation(56) [31].

[43] identifies several sources, both free and pay-subscriptions, for threat data and IOCs, for example:

• OTX - Open Threat Exchange is a publicly available service created by Alien Vault® for sharing enterprise threat data. OTX aggregates, validates and publishes threat data streaming in from a broad range of security devices across the community of Alien Vault® deployments.

• McAfee® – ‘pull’ service for subscribers, virus definition, or DAT files contain up-to-date virus signatures and other information that McAfee anti-virus products use to protect a

5 OTX - Open Threat Exchange is a publicly available service created by Alien Vault® for sharing enterprise threat data (https://www.alienvault.com/open-threat-exchange) 6 MISP - The MISP threat sharing platform is a free and open source software helping information sharing of threat intelligence including cyber security indicators (http://www.misp-project.org/)

24

Linux, Windows or Mac computer against harmful software in circulation. New threats appear each day and McAfee constantly release new DAT files.

• Wildfire® – a paid service offered for Palo Alto® firewall/IPS devices automatically delivers protections to all subscribers upon detection of illicit DNS usage, Botnets and other activity.

• CIF – Collective Intelligence Framework a client/server system for sharing enterprise threat intelligence data. CIF was developed out of the Research and Education Network and dictionary of hardware, operating systems, and applications. Information Sharing and Analysis Center (REN-ISAC). CIF includes a server component that collects and stores threat intelligence data. Data can be IP addresses, ASN numbers, email addresses, domain names and uniform resource locators (URLs) and other attributes.

Automated gathering of indicators has, however, clear advantages compared to manual gathering and human detection. One of the biggest advantages is the vast amount of data that an automatic gathering system can process. Another is the precision, since an automatic collector of indicators will not vary in its performance, it will be consistent in the judgement, unfortunately also consistently wrong, if it is wrong.

There are a number of studies performed within behavioural decision-making research that indicate problems with human judgement under uncertainty [44]. They have also found that people tend to be overconfident in their ability to detect phishing emails. The more familiar the person is with the business entities - the higher the risk for overconfidence in one’s own ability. The precision of the performance was also highly variable.

The automatic scanning of cyber threat intelligence is not just limited to tech blogs with IOCs, social media like Twitter streams are also increasingly being analysed and researched. have proposed a self-learning framework that can be used for forecasting cyber-attacks by analyzing Twitter. The logic in the form of IOCs that can be implemented in actual IDSs remain to be more clearly described.

3.3.3 Deploying IOCs

When planning for deployment of IOCs, the chain of tools to use as well as the IOCs themselves must be considered. According to [31], the best way of deciding what tools and methodology to use in a certain environment is to compare and experiment with different alternatives and configurations. An optimized solution does not come out of the box.

The IOC life cycle follows an Investigative loop according to [45]. The first step of the cycle includes searching for indicators as described in chapter 3.1.2 of this thesis. The next step of the cycle includes investigation of the compromised systems by extracting data and metadata from live systems and hosts. All compromised systems, accounts, backdoors, and ports should be identified and the corrective actions towards the vulnerabilities should be taken promptly under a concentrated period of time. After the remediation, the search for IOCs should continue to verify that the corrective actions have had the desired effect. The final step of the loop is about refining and developing new IOCs based on the existing IOCs and the analysis of the new evidence. The investigative loop is never ending and starts over again with a search for new indicators.

Figure 10 Investigative Loop Using IOCs. Inspired from [48].

25

IOC tools follow the same principle as other software, the more configurable, the more complex and time consuming to learn. Some IOC tools give assistance regarding the IOC process. For example, Redline®7 which is used for IOC searches, has a GUI with wizard function while IOC Finder is a strict command line based tool [31].

Some of the drawbacks with the OpenIOC framework, which is used for collecting and sharing of IOCs, are that it provides no guidance in how to apply it (tactics, process etc.) [42]. You can have a set of powerful tools, but if you do not know what you are doing and do not apply the appropriate methodology the end result will be poor.

3.4 Research gap analysis

Much of previous research have either been performed with focus on the automation of gathering indicators of compromise [14], [43], or with regard of manual methods [44]. A few have combined different methods. In our research we will use the automated gathered cyber threat intelligence from different sources like the ones that Rhodes [43] propose and combine that data with our manual editing to examine if it’s possible to improve an IOC, when editing, compared to a generic automated gathered IOC.

Another gap that we have identified is regarding the deployment of IOCs and the descriptions of tactics, techniques, procedures and processes. Both in regards of the collection and deployment of IOCs as well as the configuration and setting up the chain of tools in an organisation. This research aims to show that it is possible to describe a best practice starting from the need of an organisation to the specific implementation.

Further, we identified the gap that there is lack of research regarding the importance of optimized IOCs for IDS performance, a conclusion that is supported by [46]. Much of the earlier research has focused on differences in technical performance in IDSs, while testing different configurations, workload, speed etc. In our research we will focus on studying how optimizing standard IOCs using knowledge about the systems and normal workload and type of traffic in the network and how this affects the performance. Earlier research has mentioned this as having a potentially big impact [32], but few have studied this in a focused way. One could suspect that capacity increases like CPU power and multi-threading techniques would make IDSs less affected by high volumes. On the other hand, the amount of malicious traffic has increased and so has the Internet and network load in general. The IDS software is also more complex and consume more capacity.

A comparison of previous research is shown in the table 5 below, listing the references with the IDSs studied and the different aspect around IOC and proposed solutions, and some of the research referenced in the table are highlighted as follows:

7 Redline®, FireEye's premier free endpoint security tool, provides host investigative capabilities to users to find signs of malicious activity through memory and file analysis and the development of a threat assessment profile.

26

.Table 6 Literature review references with IOC related metrics and proposed solutions coverage.

Ref.

IDS Dataset Number of alerts created

False pos/ neg

IOC Connection/ IOC Gathering

Ruleset Performance of the rules/filters

Tuning of IOC

Precision

[35]

Snort v2.9.1, Suricata v1.1, Bro v1.5

Traffic generated with Ostinato, NMAP, HOIC and LOIC

Suricata created most alerts, then Bro and finally Snort

Not investigated

Not covered Emerging Threats full ruleset and specific rulesets for different attack types

Full ruleset performed best

Selection of rules to fit different attack types

Not covered. Need for further study

[33]

Snort, Suricata, "as is" and improved by learning algorithms for a hybrid approach

Open source network traffic generators, Ostinato, NMAP, and NPING. NSA Snort IDS Alert Logs DARPA NSL-KDD

Not described Snort: 55,2% FPR Suricata: 74,3% FPR Snort and Suricata’s TPR was 99%

Learning algorithm

Default The best result was with a Support Vector Machine on Snort with 8,6% FPR and 2,2% FNR.

Automatic Improved via an adaptive plug-in working in parallel with the IDS

97,3% for Decision Tree algorithm

[32]

Snort, Suricata

1. 2010 iCTF Conference “Attacks Against Litya” network capture 2. Own created using tcpdump to capture runs of specific PytBull traffic

Not compared Not investigated

Not investigated Sourcefire Vulnerability Research Team (VRT) and Emerging Threats (ET) and ET-Pro, the commercial ruleset

Not so much differences as the researchers had expected, between default rulesets and the others. Performance in terms of packets per second is analysed.

Not performed

Not analysed

[47]

Not

covered

Definition, comparison and exa mples of datasets DARPA

Not covered Not covered Real or Synthetic traffic recorded in production or laboratory network

Not covered Not covered Not covered Not covered

27

Ref.

IDS Dataset Number of alerts created

False pos/ neg

IOC Connection/ IOC Gathering

Ruleset Performance of the rules/filters

Tuning of IOC

Precision

(+NSL-KDD KDD99) CTU-13

[34] Juniper Palo Alto Check Point Suricata

CTU-13 Not described Not analysed Manual Proprietary, Emerging Threats signatures (for Suricata)

All IDSs detected threats in 50% of the datasets

No 25% (in one out of four datasets all attacks were detected)

[41] Snort Simulated synthetic environment within CRATE, cyber range of the Swedish Defence Research Agency (FOI)

The rules create fewer alerts, thus decreasing Precision

There were

more false

negatives

with the

added filters,

compared to

baseline

Manual Baseline Snort IDS rules with 18 new filters

Several filters performed worse than baseline. Baseline Precision 1,4% Average Precision with filter: 1,13% Baseline Recall: 44,2% Average Recall with filter: 21%

Manual,

Vulnerability

based

Increased from 1,4% to 2,9%

[48] Host based IDS OSSEC combined with network-based IDS Snort, versus Administrator(manual)

Simulated synthetic environment within CRATE, cyber range of the Swedish Defence Research Agency (FOI)

IDS: 20107 Admin: 70

False

negative rate

IDS: 30,2%

Admin:

41,2%

Alarms from simulated synthetic traffic. Admin monitored the network via web based UI.

Not described Bayesian detection rate IDS: 11% Admin: 57%

Manual Admin responsible for setup and tuning

Expressed in terms of probability of detection, IDS 69% and administrator 58%

28

IDSs have been evaluated in many studies from a technical performance perspective, covering metrics like number of CPU usage, alerts created, packet loss, throughput and traffic rates, network speed and IDS scalability [35], [33], [17], [16].

K. Thongkanchorn et al. [35] vary the rulesets by running portions of them targeted towards particular attacks. The result is that a full ruleset has better performance in terms of number of alarms created, compared to a partial ruleset.

Rulesets from different sources have been compared by J. S. White et al. [32] and D. Hedemalm [34] with the conclusions that the standard rulesets, free or proprietary, do not show any significant differences in regards of alert creation.

S. A. R. Shah and B. Issac [33] have shown that hybrid approaches with automatic tuning of IOCs via self-learning systems can dramatically improve the true positives rate in Snort.

All of the studies have been based on public datasets, generated datasets or combinations of them. None of them use live recordings from real production networks for proprietary and confidentiality reasons as well as for transparency reasons. M. Malowidzki et al. [47] have elaborated on the strengths and weaknesses with different types of datasets. They conclude that many public datasets are obsolete, both regarding background traffic and attack types. Synthetic datasets are controllable but are difficult to create so that they resemble authentic traffic. They recommend CTU-13 as an example of a good dataset.

T. Sommestad and U. Franke [41] and T. Sommestad and A. Hunstad [48] have approached the subject of IDS performance from a network information and expertise of the system administrator perspective. The hypothesis is that a signature-based IDS can be enhanced by utilizing the more static information that exists about a certain network. Examples are IP addresses, network topology, what ports are open, and the known vulnerabilities within the supervised network. After adding these vulnerability rules or filters the precision and recall percentages dropped in their test. The authors suggest that additional studies should be performed on datasets with more background traffic and over a longer time period. They suggest that a more thorough vulnerability scan could lead to creation of better filters and higher precision and recall figures.

T. Sommestad and A. Hunstad [48]also stress the importance of competence of the system administrator when using the IDS. A system administrator that knows the network well, is skilled in networking and security in general, knows what assets are the most valuable and understands the capability of external adversaries could perform as well as, or even outperform, an IDS. The skills of the system administrators are claimed to be possible to implement in an IDS, as a filters or modified rulesets.

The outcome regarding precision from previous research cannot be compared since they differ in ways of testing, categorizing of true vs false positives, and different starting points for improvement. Nevertheless, most of them have achieved an improvement of precision.

A research gap has been identified in that a number of researchers highlight the importance of IOC and ruleset tuning, but little guidance is given to how this should be done. The technical alternatives are described, but not the process steps involved in the work. It is recommended according to current research to look for unusual activities and static information about the network in order to find the parameters and IOCs to tune. What those parameters are and what to look for in the context of the monitored network and how to actually parse an IOC into an effective IDS rule in order to achieve higher precision is not yet so thoroughly researched.

This research gap is the motivation for our research question stated in chapter 1.3.

29

Chapter 4: Research method

4.1 Introduction

Design Science Research Methodology is chosen because the research question is about a new or improved solution to known problem.

4.2 Design Science Research Methodology

K. Peffers et al. [49] have proposed a Design Science Research Methodology (DSRM) for Information Systems Research. The method was developed as a response to a lack of generally accepted methods within Information Systems research that have a goal to create and/or improve artefacts.

An artefact has been defined as [50]:

"those bundles of material and cultural properties packaged in some socially recognizable form such as hardware and/or software".

The artefacts are represented in this research by the Indicators of Compromise which are the forensic signs of intrusions.

The DSRM methodology is intended to meet three goals; consistency with previous research, providing a Design Science research process model and also providing a model for presenting and evaluating the research.

The process steps are described in the following picture. The output of the design science process is an artefact solving a problem that is of interest for both organisations and academia. The entry point of this research is the Design & Development.

Figure 11 DSRM process. Inspired from [49].

30

The following picture shows the architecture for creation of this research’s two artefacts in the design and development phase. Artefact 1 is a proof of concept for heighten precision in an IDS and this research will show it is achieved by gathering IOCs which are then parsed to create IDS rules that is implemented in an IDS. This is performed by following the steps in the formalized process that is artefact 2 from this research. The artefacts are tested and validated in the phases demonstration and evaluation.

Figure 12 Design architecture for POC and formalized process.

Each research question has been mapped to each DSRM-phase:

4.2.1 Problem identification and motivation

Research question 1: “How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise?”

• The problem identified is the number of false positives that an IDS can create if the rules implemented are to general written.

o The motivation to address this problem is because it causes a lot of unwanted work for IT-professionals that have to go through logfiles and assess whether the threat is real or not. This unwanted work could be shifted to tasks of more value to the IT-professionals and the organization that they work for.

• The problem identified is how complex it is to write accurate IOCs.

o The motivation to do research about this problem is to find out whether an IT-professional gain, time and effort, when tuning IOCs or handling the false positives.

31


• The problem identified is the complexity to create and implement trustworthy rules in an IDS.

o The motivation to address this problem is because we believe the complexity is better managed when addressing the problem in a formalized way.

4.2.2 Define the objectives for a solution

Research question 1: “How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise?”

• To write rules with a higher essentiality for a specific system will enable the IDS implementing these rules to raise its precision.

o The solution will be a methodology of best-practises and a contribution of know-how to all IT-professionals that work with implementing or editing rules from parsed IOCs.


• Identify steps and activities that must be performed to cover all aspects when writing and managing rules for an IDS.

o The solution will be a formalized process which will guide personnel creating and managing rules for an IDS.

4.2.3 Design and development

Research question 1: “How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise”

• To write improved rules from parsed IOCs, the OpenIOC-framework will be used since it provides the ability to use logical comparison of Indicators in an IOC, providing more flexibility and discrimination than simple lists of artefacts.


• A body of methods, rules and qualifications that will guide an IT-security professional in tuning IOCs for the purpose of higher precision of an IDS, will be the key components of the design and form the method TIDS.

4.2.4 Demonstration


• The research question will be answered in a laboratory environment at Basalt AB with both virtual and physical devices. We will use both Linux and Windows operating system and the virtual machines will be created in VirtualBox. For our IDS implementation, we will use Snort, installed as a NIDS on a single host.

32

• With the use of recorded data traffic from CTU-13, the difference between the use of tuned versus un-tuned rules in an IDS will be presented in terms of measuring the precision of the IDS.


• During the demonstration phase, our process TIDS will be implemented and used. This will both show the different steps as well as validate them when the demonstration of research question 2 is used to answer research question 1.

4.2.5 Evaluation



The results from the demonstration phase will be evaluated and reviewed when we implement our formalized process TIDS that have been designed during this research work, it aids us in presenting and performing validation tests in a structured and robust way and with the use of recorded data traffic from the dataset CTU-13, the difference between the use of tuned versus un-tuned rules in an IDS is presented in terms of measuring the precision of the IDS in the testbed.

When the results from our tests are reviewed, we can come to the conclusion that it is possible to tune a local rule in another way, for example, by adding conditions, or removing obsolete rulesets. This action is not connected to any measurement criteria and the most important reason to stop tune an IDS is often time, since in an organization someone is often paying you for that time you consume.

• This step can be an iterative phase depending on if the outcome from the demonstration phase creates results that can be further examined and will in that case be “moved back” to the design phase.

• If an iteration is needed, the parameters that may be changed are activities or steps within the TIDS process steps.

4.2.6 Communication



• The research will be presented to our supervisor and fellow students at Lulea University of Technology and the companies Basalt AB, Saab AB and FOI. The communication will be in the form of an artefact, presentations and seminars and by publishing of the research paper.

33

Chapter 5: Design of TIDS process

5.1 Introduction

The process that this research will establish is named TIDS, Tuned Intrusion Detection System. It consists of several activities that performed in a proper manor will create a solid foundation for a secure and well implemented set of rules for an IDS.

5.2 Design of TIDS process

TIDS is divided into four main activities with sub-events within. Depending on the organization implementing the process, choices can be made to ignore certain parts but it’s very important to notice that this process is iterative. Even though all seems working fine, the different steps should be performed at certain intervals to maintain a proper level of security.

It is crucial to understand that prior to implement TIDS, the responsible personnel in the organization need to perform analyses and risk assessments to have a clear understanding on what the IDS should secure and which data that needs to be protected. If this pre-check step is not performed, the IDS will only become a costly and hard to manage implementation.

Figure 13 TIDS process.

The process should be looked upon as checking tool consisting of four corner stones that we call steps. It defines what to do, and not explicitly, how to do it. We mention a few examples when we describe the different steps, but to implement TIDS for rule-management demands an understanding of IDS and if an implementing organization have specific needs or regulations, local activities can be added to the best suiting step.

To make sure that no step is forgotten or skipped due to limited time for implementation, a TIDS implementation checklist can be used. This checklist contains amongst other a column for the date that a specific activity is performed which is very useful information when you plan your implementation iterations.

34

Figure 14 TIDS implementation checklist.

The different main activities and their sub-events are further described in the sub-chapters below and to demonstrate and validate TIDS, it is implemented during our test and result activities in chapter 6 and 7.

5.2.1 Analyze

First step in this iteratively process is to gather IOCs and evaluate which is suitable as information source for the IDS-rules that shall be created.

5.2.1.1 Gather

The IOCs can be gathered both manually and automatically. The different sources for finding IOCs can be blogs, vendor sites, open source communities, googling and internal generation from analysis on private network etc. To find the sources that suits your specific organization can be a matter of test and evaluation, but even though a couple of reliable sources have been found, the searching should continue since the cyber threat evolves over time, so can other IOC sources. There are two types of gathering:

Figure 15 TIDS process step 1.

35

• The IOCs is either created manually, by an IT-professional after a forensic investigation or proactively to meet an uprising threat.

• The IOCs is gathered automatically. Which tool to choose for automatic gathering is up to every organization preference, there numerous products, both open source solutions as well as commercial. For example:

o Open source

▪ MISP8 - A threat intelligence platform for gathering, sharing, storing and

correlating Indicators of Compromise of targeted attacks, threat intelligence,

financial fraud information, vulnerability information or even counter-

terrorism information.

▪ iACE9 - an innovation solution for fully automated IOC extraction. The

approach is based on the observation that the IOCs in technical articles are

often described in a predictable way.

o Commercial

▪ Cisco AMP Threat Grid® - combines static and dynamic malware analysis with threat intelligence into one unified solution and provides an extensive selection of IOCs that can be leveraged to protect the organization.

5.2.1.2 Evaluate

With information from Pre-check, earlier performed analysis and risk assessments, the gathered IOCs should be scanned and assessed whether they suit the domain or data that the organization wants to protect. If there are IOCs that don’t suit the organizations purpose, they should be removed. The reason for this thinning is that a lower number of rules lowers the total processing time for the IDS, and as we defined earlier, precision is one of the performance criterions, the implemented rules should be as suiting as possible for the current domain.

5.2.2 Run

The second step to perform when following TIDS is to parse the gathered IOCs to collect information and create IDS-rules. In this step, it’s also important to tune the implemented tools.

5.2.2.1 Parse IOCs

When it comes to parsing the IOCs in their originally format there are several tools that can be used. Mandiant® have created a tool, IOCe, which is an editor that provides a user interface which makes it easy to read or edit the IOC. You can easily add AND or OR sections to further evolve the IOC.

8 http://www.misp-project.org/ 9 [14] https://www.informatics.indiana.edu/xw7/papers/liao2016acing.pdf


36

One way to create IDS-rules from information found in IOCs would be to open the IOC in IOCe to interpret and combine the indicators that together can form the rules suited for the desired IDS setup. The actual creation of IDS-rules can then be made in what text editor that is available, for example Notepad++10.

Figure 17 Snort rule architecture.

There are many researchers that study the subject how to write a perfect rule and that is something you can’t address in a master thesis due to the limited amount of time there is to perform research [51]. But there are best-practices to follow and apart from technical blogs and communities, Snort has a chapter in their user manual, “Writing good rules”, that highlight the aspects of content matching, which is written as a condition in the “rule options” section in the figure above. They also address that the rule writer should focus on the vulnerability, instead of the exploit and other useful hints. Events to look for [31]:

• Unusual spikes in ingress or egress traffic

• Unexpected access from IP-addresses from new geographic locations

• Logs (web proxy, DNS)

• Connections on unusual ports or protocols

• High rate of failed logons

• Unusual database activity

• Cross security zone access

• Hosts that are constantly failing in connecting to a target that they should be able to connect to

Before the created rules are deployed they may very well be tested against an organizational recorded dataset to check for weaknesses and bad logical content matching. If errors are found, the parsing and rule writing continues for as many iterations as needed, before we continue to next step, Deploy.

10 https://notepad-plus-plus.org

37

5.2.2.2 Tune tools

To perform this activity, information about the network topology from the pre-check needs to be revised. It must be clear which tools are involved in the IDS solution before we can determine how to tune them and we can look at it from different views, for example, an IDS can be tuned by optimizing its rules as well as by patching the binaries with the latest updates. In summary, the tuning can be made in different ways:

• Make sure that the latest, or best suited rules for the specific domain are implemented.

• Make sure no obsolete, or unessential, rules are implemented

• Verify that involved tools are updated and patched.

• Verify that the right tool is chosen for a specific task, for example, Snort demands less network cost compared to Suricata, which on the other hand can handle several current threads. Other examples of tools with specific functionality is Pulled Pork and Oinkmaster. With those tools, updates can be made easier and manual changes and additions like custom made rules are content managed. Updates can be automatized and rules can be turned on or off by using the tools [52]. Both Snort and Suricata can be set to reload the rule sets without having to be restarted. This means that new rules can be deployed without any service window [53].

5.2.3 Deploy

The third step is to identify and make sure that all implemented artefacts that the IDS should secure is accounted for.

5.2.3.1 Identify affected systems

This step is very much depending on proper performed Pre-check where amongst other, the questions about network topology and sensitive data should be answered. To try to secure a network that you don’t know it’s topology or which kind of data resides where, is almost an impossible task.

How this network-survey is performed is up to every organization. There is a lot of network-scanning tools available, both free and commercial.

5.2.3.2 Collect data

After the IDS is implemented the results from configurated logfiles should be collected and reviewed. The scanning of logfiles can be performed in Wireshark, Splunk11, Notepad++ etc. In this step, there is several ways to store logs and collect data. You could use a syslog server and there are several open source tools available to parse the data. However, which log-solution you choose is up to every organization to decide since, for example, different solutions needs different operating systems to run on.

11 https://www.splunk.com/


38

5.2.4 Assess

The final step in this method involves assessment whether the implemented tools and IDS-rules have delivered a satisfying outcome. If not, the weaknesses must be identified and measures to adjust the implementation should be performed.

5.2.4.1 Implementation

Methods and processes regarding security often declares that it’s important to update rulesets and virus definitions, but we feel that it’s equally important to update and verify the different tools you have implemented. It can’t be stated enough that if a security solution shall work, it’s important that the administrators or otherwise responsible personnel have the complete picture of the specific organization very clear. If you don’t understand the business, you don’t know what to protect. This is where this step gets realized. You must assess at certain interval that the security measures implemented is the best implementation for the business. If not, then the Pre-check should be performed again and TIDS should be started all over.

5.2.4.2 Level of satisfaction

After a process iteration of TIDS, all steps performed should be reviewed and the implemented security solution should be measured against the Pre-check questions to verify if the implemented solution has performed as the organization thought it would.


39

Chapter 6: Implementation and Evaluation

6.1 Introduction

In this chapter, which corresponds to the Design and Development phase in DSRM, we will present tests that where performed to provide information to this research which enable us to reflect on, and answer, research question 1. When we performed our tests, we implemented step 1 and 2 and their activities from our process TIDS. This created a structured approach and showed that TIDS is a valid and well-grounded outcome from our research. Step 3 and 4 will not be used since they are only applicable when you implement the process in a production environment. The result of the tests will be presented and discussed in chapter 7.

6.2 Testbed

Since our research domain is a small or medium organization, we have focused on a cost-effective solution for an organization with a network topology consisting of windows machines. That lead to the specification of our testbed which consists of a standalone Windows 10 Enterprise installation12 combined with Snort version 2.9.11.1, Wireshark and Microsoft Excel. The test data comes from the dataset CTU-13 which is a dataset of botnet traffic that was captured in the CTU University, Czech Republic, in 2011. The dataset consists of thirteen captures, called scenarios, of different botnet samples [15].

To create a baseline, the CTU-13 dataset scenarios are analysed with Snort rules, version 29111. The alert files are created .csv-file format, for further processing in MS Excel and the log files and .pcap files are analyzed in Wireshark and Notepad++.

Scenario 10 in the dataset suffered from disc space outrun during the capture and therefore has an incomplete ending causing our Snort scans to never complete. This scenario is therefore excluded from further analysis.

6.3 Datasets

This thesis is run on pre-recorded datasets. If the IDS was run on real traffic in real time the performance of Snort and the rules would be influenced by the workload (packets/second) that is sent in the network and the technical performance of the system on which it was run. With a high load Snort would be starting to drop packets, which in turn would generate false negatives. With the pre-recorded datasets we expect to have no dropped packets because the IDS can process the data packets in its own pace.

KDD99 is the most commonly used dataset for IDS simulation [54]. It has become a de facto standard in research regarding IDS performance. [47] claim that the major weakness with KDD99 is

12 Dell optiplex 9020 with a i7 processor and 32GB Ram.

Figure 20 TIDS process step 1 and 2.

40

that it is not recent, instead rather outdated. This could raise a question mark whether the dataset is realistic, diverse and representative enough.

CTU-13 is a dataset developed at the Sratosphere Lab as part of the Malware Capture Facility Project in the CVUT University, Prague, Czech Republic, in 2011. The dataset is well documented, labeled and consists of background traffic, normal traffic and botnet traffic. The dataset consists of 13 botnet scenarios using Neris, Rbot, Virut, Menti, Sogu, Murlo and NSIS.ay. All scenarios include at least one infected Windows XP host and a Web, DNS and Matlab server. All are virtual machines run via Virtualbox on Linux. Scenarios 9 and 10 have ten inflected hosts and Scenarios 10 and 11 have three. This dataset simulates a Neris botnet running for 6.15 hours in a small University network. The scenarios are intended to represent how a botnet attack could progress, from a single infected host to port scanning of the remaining hosts, infection of more hosts to Command and Control (C&C) from an external IP address to the hosts attacking external IP addresses with UDP and ICMP DDOS attacks.

Table 7 Characteristics of the botnet scenarios [15].

Id IRC SPAM CF PS DDoS FF P2P US HTTP Note 1 X X X 2 X X X 3 X X X 4 X X X UDP and ICMP DDoS. 5 X X X Scan web proxies. 6 X Proprietary C&C. RDP. 7 X Chinese hosts. 8 X Proprietary C&C. RDP. 9 X X X X

10 x X X UDP DDoS 11 X X X ICMP DDoS 12 X Synchronization 13 X X X Captcha. Web mail

What is not very well documented about the CTU-13 is the network topology, the user is provided with a list of IPs and whether the host is infected or normal. One of the hosts, “Saruman”, is said to be a Windows XP machine, English version. By analyzing the alert log from Scenario 9 a vulnerability typical for Windows XP SP3 is discovered on one of the hosts, “Saruman 2”, so it is likely that all hosts are of that version, but we do not know that for a fact.

6.3.1 Scenarios CTU-13

In this research, we want to show that specific knowledge of a network can be of importance when it comes to create rules for an IDS. We will perform tests that will validate the different activities we suggest in our methodology to tune an IDS. We mention the term “knowledge”, and in our fictive scenarios it comes from both organisational implemented policies and from senior expertise.

The total of data packets in the CTU-13 data set is 855 805 603 and when [15] performed their research and created the dataset, they also labelled the packets in four categories:

• Background flows

• Botnet flows

• C&C flows

• Normal flows

In this research we will use the prior performed work of labelling data packets as an indisputable reference of which data packet belongs to which label category.

41

The CTU-13 dataset consists of 13 scenarios that combined cover a botnet attack using several C&C channels, attempting to send SPAM, sending SPAM and performing click-fraud using some advertisement services.

Table 8 scenarios in CTU13.

Scenario # Description

1 A bot sending spam, connected to an HTTP CC, using HTTP to do ClickFraud. One infected host.

2 Neris malware attack. One infected host.

3 The virtual machine is infected with rbot.exe, that can be controlled with IRC. One infected host.

4 DOS attack using rbot.exe. One infected host.

5 VM infected with fast-flux malware. One infected host.

6 Donbot attack. One infected host.

7 Sogou botnet sending spam, connected to IRC, using HTTP to do ClickFraud. One infected host.

8 Murlo botnet, C&C to China hosts. One infected host.

9 Mimic of a real Neris infection from client to client. 10 infected hosts.

10 Corrupt/incomplete and not used.

11 ICMP DOS attack to a computer by an IRC botnet controlled from the lab. 3 infected hosts.

12 NSIS.ay botnet. 3 infected hosts.

13 Infection of VM with a fast-flux malware. One infected host.

6.4 Test

The purpose of this research is to answer our two research questions:

1. How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise?

2. How can activities and procedures be formalized into a process for managing rules for, and implementation of, an IDS?

To answer research question 1, our tests is performed to validate that there are more alerts created by adding new rules, than it is by using a standard snort ruleset, and as long as the majority are true positives the precision is increased as stated by the formula:

Precision = TP/(TP+FP)

What parameters to change to improve the precision cannot be stated, it’s a question from time to time to gather information from IOCs and using your own specific network knowledge to iteratively test tuned rules, preferably in an offline environment, to achieve the quality of the result you want, before deploying them in an on-line environment.

42

To answer research question 2, our test will be performed by following the identified activities in our formalized process TIDS. The two first tests are merely to assess which dataset to use and whether blacklisting of a specific IP address is a to blunt condition to use in a rule.

The tests that is performed during this research are presented in chapters 6.3.1 to 6.3.6. The overall purpose is to verify if collected information from forensic analysis of the recorded dataset can be added to IDS-rules and create a greater number of true positives and hence, heighten the precision.

6.4.1 Pre-test 1 – DARPA dataset

The purpose of this test is to verify that the DARPA dataset is a suitable dataset to use for the remainder of the thesis.

Table 9 Pre-test 1.

Pre-check Data/Outcome Comment:

In this research, the pre-check is equal to us using a pre-recorded and labelled dataset

1999 DARPA Intrusion Detection Evaluation Data Set, a work week containing attacks.

Outside sniffing data (Tcpdump format)

Inside sniffing data (Tcpdump format)

Analyze - Gather IOCs Data/Outcome

Comment:

The DARPA 1999 dataset is scanned by Snort with the ruleset version 2983.

Monday Outside Snort alert file, containing 51757 data packets. Monday Inside Snort alert file, containing 48837 data packets. Tuesday Outside Snort alert file, containing 20630 data packets. Tuesday Inside Snort alert file, containing 20964 data packets. Wednesday Outside Snort alert file, containing 11153 data packets. Wednesday Inside Snort alert file, containing 11395 data packets. Thursday Outside Snort alert file, containing 20293 data packets. Thursday Inside Snort alert file, containing 20530 data packets. Friday Outside Snort alert file, containing 30956 data packets. Friday Inside Snort alert file, containing 48837 data packets.

Analyze - Evaluate Data/Outcome Comment: A pre-test is conducted to evaluate the suitability of the DARPA dataset for the thesis.

The alerts created are mainly for protocol mismatch and content length errors. There are also some sensitive data alerts for exposure of e-mail addresses, portscan/portsweep and a few ftp bounce attempts. We get a low or no detection rate for other attacks that exist in the dataset according to documentation, for example, DOS (Back. Land, Neptune, Smurf, Teardrop). Taking the Friday Inside Snort file as an example, it contains the following alerts:

• 40000 spp_ssh, TCP protocol mismatch • 6025 http_inspect, content length errors • 1027 sensitive_data e-mailadresses • 796 Spp_sdf, combination alert • 151 Consecutive TCP small segments • 129 Unescaped space in http • 80 simple request • 25 UDP portsweep • 2 ftp_telnet • Others

This tcpdumpfile is a representative example for all the examined files. Run - Parse IOCs Data/Outcome Comment: N/A

Run - Tune tools Data/Outcome Comment: N/A

43

6.4.2 Pre-test 2 – CTU-13 Blacklisting

The purpose of this test is to verify that the CTU-13 dataset is a suitable dataset to use for the thesis and to test if blacklisting of certain IPs is a suitable function to use in order to raise the precision of alerts.

Table 10 Pre-test 2.

Pre-check Data/Outcome

Comment:


CTU-13 Dataset


Comment:

The CTU-13 dataset is scanned by Snort with the ruleset version 2983.

#1 Snort alert file, containing 857 data packets. #2 Snort alert file, containing 1782 data packets. #3 Snort alert file, containing 5538 data packets. #4 Snort alert file, containing 5 data packets. #5 Snort alert file, containing 100 data packets. #6 Snort alert file, containing 88 data packets. #7 Snort alert file, containing 11 data packets. #8 Snort alert file, containing 441 data packets. #9 Snort alert file, containing 4314 data packets. #11 Snort alert file, containing 123 data packets. #12 Snort alert file, containing 151 data packets. #13 Snort alert file, containing 2009 data packets.

Analyze - Evaluate Data/Outcome

Comment:

CTU-13 scenarios are processed with snort ruleset 2983. As part of the attack scenario, a host in the network is infected with bot malware. As alerts are analyzed and the attack is deemed to originate from one host in the network, an approach could be to use the blacklist function.

The host with IP 147.32.84.165 is added to the blacklist rules file.


Run - Parse IOCs Data/Outcome

Comment:

What can be noted is that the first scenarios have a much higher alert rate when using the blacklist function compared to the other. This suggests that the blacklist function labels all traffic as malicious when instead a big portion probably is benign.

The later scenarios on the other hand have a very low alert rate compared to the botnet flow. One explanation could be that in the latter scenarios multiple hosts have been infected so the blacklist function used with one single blacklisted IP address only captures a small portion of the malicious flows. In the comparison, scenario 4 is excluded because it showed an alert rate of over 99% when using blacklisting, which we have no explanation for.

N/A

44

Scenario 10 was too large and corrupt and caused Snort to crash which resulted in different results every time. A lesson could be not to use too large .pcap files in Snort, not to record too long at a time or try to divide them before analysis. It seemed like CTU-13 could be an appropriate dataset to use, except for scenario 10.

Run - Tune tools Data/Outcome

Comment:

N/A

6.4.3 Test of alerts on suspicious domain

The purpose of this test is to verify that an IOC of a suspicious domain can be parsed into an effective IDS rule, with good precision.

Table 11 Test of suspicious domain.


Comment:


CTU-13 dataset, scenario 13, with a total of 440625 data packets infected with a Fast-Flux malware with a MD5 hash of 85f9a5247afbe51e 64794193f1dd72eb


Comment:

The CTU-13 dataset is scanned by Snort with the ruleset version 2983.

#13 Snort alert file, containing 2010 data packets.

Analyze - Evaluate Data/Outcome

Comment:

The .pcap file was opened in Wireshark and manually examined, we found several occurrences of the infected windows machine making DNS-requests to the domain, tasb.ru, which examined by VirusTotal13 showed communication with a large number of files, connected to identified malware.

N/A

Run - Iteration 1 Parse IOCs Data/Outcome Comment: A local rule was created to look for any external communication to the IP address that communicated with the suspicious domain.

alert tcp $EXTERNAL_NET any -> 147.32.80.9 53 (msg:" SUSPICIOUS DOMAIN:tasb.ru"; reference:domain,virustotal.com /#/domain/jot.tasb.ru; sid:1000007; rev:1;)

Run - Iteration 2 Parse IOCs Data/Outcome Comment: A local rule was created to look for any internal communication to the IP address that communicated with the suspicious domain.

alert tcp $HOME_NET any -> 147.32.80.9 53 (msg:" SUSPICIOUS DOMAIN:tasb.ru"; reference:domain,virustotal.com /#/domain/jot.tasb.ru; sid:1000008; rev:1;)

13 https://www.virustotal.com. VirusTotal inspects items with over 70 antivirus scanners and URL/domain blacklisting services, in addition to a myriad of tools to extract signals from the studied content

45

Run - Iteration 3 Parse IOCs Data/Outcome

Comment:

A local rule was created to look for any communication between the IP address that communicated with the suspicious domain and the machine that was stated to be infected from the beginning.

alert tcp 147.32.84.165 any <> 147.32.80.9 53 (msg:" SUSPICIOUS COMMUNICATION:tasb.ru"; reference:domain,virustotal.com /#/domain/jot.tasb.ru; sid:1000009; rev:1;)


Comment:

As a final test to this .pcap file we removed rules that didn’t had any connection towards DNS events. As a total we removed 5407 rules by disabling the files in the Snort configuration file

#13 Snort alert file, containing 2010 data packets.


Comment:

N/A

N/A

6.4.4 Test of alerts on any external traffic

The purpose of this test is to test the applicability of the simple generalization that all external traffic is unknown and therefore potentially malign [48], and if it increases the precision to apply rules that trigger on external IP addresses.

Table 12 Test of external traffic.


Comment:


CTU-13 dataset, scenario 1-13, with a total of 50888256 data packets


Comment:

Every scenario in the CTU-13 dataset is scanned by Snort with the ruleset version 2983.


Analyze - Evaluate Data/Outcome Comment: A basic rule of thumb for system administrators may be to simply assume that all network traffic originating from outside the network is potentially malign. According to [48] “Intrusion detection and the role of the system administrator”, the combination of a peak of alarms and external IPs can be used to identify attacks.

N/A

46


Comment:

A local rule was created to capture the traffic not originating from the internal network. The rule searches for specific content in the file.

alert tcp ![147.32.84.0/24,147.32.87.0 /24,147.32.80.0/24] any -> [147.32.84.0/24,147.32.87.0 /24,147.32.80.0/24] 111 (content: "|00 01 86 a5|"; msg: "external mountd access"; sid:1000004;)


Comment:

The local rule that was created to capture the traffic not originating from the internal network was modified so it would not trigger on specific content, instead concentrating on traffic to specific ports.

alert tcp ![147.32.84.0/24,147.32.87.0/24,147.32.80.0/24] any -> [147.32.84.0/24,147.32.87.0/24,147.32.80.0/24] 21,22,80,443 (msg: "external mountd access"; sid:1000004;)


Comment:

Due to of a lot of false positives because of an internal IP address (147.32.96.45) regarded as external, the rule was changed.

alert tcp ![147.32.84.0/24,147.32.87.0/24,147.32.80.0/24,147.32.96.0/24] any -> [147.32.84.0/24,147.32.87.0/24,147.32.80.0/24,147.32.96.0/24] 21,22,80,443 (msg: "external mountd access"; sid:1000004;)


Comment:

NetworkMiner 1.6.1 was added as a forensics tool

Graphical User Interface that complemented the analysis

47

6.4.5 Test of vulnerability scan and traffic to unusual ports

The purpose of this test is to see whether it is useful to track traffic to unusual ports based on known weaknesses of those ports on certain operative systems. The reason is that known weaknesses are more prone to external attacks and a rule targeting those weaknesses creates alarms raising the attention towards the most important parts of the network.

Table 13 Test of unusual port.

Pre-check Data/Outcome Comment: In this research, the pre-check is equal to us using a pre-recorded and labelled dataset


Analyze - Gather IOCs Data/Outcome Comment: Every scenario in the CTU-13 dataset is scanned by Snort with the ruleset version 2983.


Analyze - Evaluate Data/Outcome Comment:

According to the description of Scenario 9, there is a lot of incoming traffic from an external IP address using TCP on port 3389 towards one of the hosts in the internal network.

With access to the real network a network mapping, port scan and a vulnerability scan would in this stage have been conducted in order to map the network structure and nodes, what ports are open and what vulnerabilities there are. We know in this case that one of the hosts have Windows XP. We make an assumption that all hosts have Windows XP. When searching for vulnerabilities of Windows XP, we find a vulnerability of port 3389 which is a vulnerability in the RDP protocol, CVE-2012-2526, on Windows XP SP2.


Run - Iteration 1 Parse IOCs Data/Outcome Comment: This alert rule is created to find incoming TCP traffic to the 3389 ports of the affected host.

alert tcp any any -> 147.32.84.0/24 3389 (msg: TCP traffic to Windows XP hosts to unusual port 3389"; sid:1000008;)

Run - Iteration 2 Parse IOCs Data/Outcome Comment: The alert is supplemented with all the internal IP numbers assumed to run Windows XP.

alert tcp any any -> [147.32.84.0/24,147.32.87.0/24,147.32.80.0/24] 3389 (msg: TCP traffic to Windows XP hosts to unusual port 3389"; sid:1000009;)


N/A

48

6.4.6 Test of spikes in network traffic

The purpose of this test is to examine whether it is a useful indicator of compromise, if and when, there are unusual spikes in the network traffic. If botnet attacks generate a lot of network activity, it might be possible to see a correlation between malware alarms and traffic spikes.

Table 14 Test of spikes in network traffic.

Pre-check Data/Outcome Comment:



Analyze - Gather IOCs Data/Outcome Comment:

Every scenario in the CTU-13 dataset is scanned by Snort with the ruleset version 2983.


Analyze - Evaluate Data/Outcome Comment: The purpose of this test is to see whether it is a useful indicator of compromise if and when there are unusual spikes in the network traffic. Since botnets generate a lot of traffic, we anticipate seeing some kind of reflection in the network traffic, packets/second.

The .pcap files are opened in Wireshark and analysed for I/O statistics for all packets and with time stamps to be able to locate the packets more easily in log and alert files etc. An analysis of the alert files will also be performed to see if there are alerts generated that correlate with the spikes.

Run - Parse IOCs Data/Outcome Comment: N/A

N/A


N/A

49

Chapter 7: Result and Discussion

7.1 Introduction

Our test where conducted to verify if the theories we have concluded, from the knowledge gathered during this research regarding an IDS precision are valid. The results are far from indisputable due to the fact that the number of tests is few and the dataset used to simulate traffic is somewhat questionable since its stated network topology have a few odd features. However, the gathered results are pointing towards the satisfying state which validate that IDS rules from parsed IOCs produces a higher precision than an untuned ruleset. In the table below, we summarize our results which is further elaborated in the following chapters 7.1.1 – 7.2.4.

Table 15 Summary of conclusions.

Tests Conclusions

Pre-tests 1 & 2 When using a dataset for tuning of an IDS, use a labelled and well-documented dataset.

Pre-tests 1 & 2 It is better to use a dataset from about the same time period as the IDS software and ruleset versions. For that reason, CTU-13 is better than the DARPA dataset for this research.

Pre-tests 1 & 2 The blacklisting function should be used with caution. It is easy to generate a lot of false alarms if the rule is set up in the wrong way.

Test of alerts on a suspicious domain

Forensic analysis of suspicious domains discovered in the traffic can be worth the effort if the cost of the time spent is less than the presumed cost of a successful attack.

Test of alerts on a suspicious domain

One of the many ways to tune an IDS is to disable rules that are out of scope for the specific network and data traffic.

Test of alerts on any external traffic

A rule that filers out external traffic generates some new alerts that are true positives, but not many. The IDS standard ruleset catches the majority of suspicious external traffic.

Test of vulnerability scan and traffic to unusual ports

Adding custom rules based on known vulnerabilities in the monitored network and hosts, is an effective way of getting important true positives and raising the precision of alerts.

Test of spikes in network traffic

The results are varied and inconclusive. In this research there seems to be no correlation between the occurrence of Malware-CNC alerts and spikes in network traffic in the different scenarios.

General When tuning rules against pre-recorded capture files in this way, there was no packet loss. In a live scenario there would be packets and alerts lost due to high traffic rates and/or insufficient IDS capacity.

General Separating the false positives from the true positives was more difficult than expected.

Test of TIDS process (ways of working during the tests)

The TIDS process has shown to be useful during the Analyse and Run steps.

50

7.2 Pre-test

7.2.1 Pre-test 1 DARPA dataset results

The pre-test on the DARPA dataset showed that the detection rate of the botnet attacks is low. If an IDS implement rules created today, and test it on a 19 years old dataset, we draw the conclusion that many attack types of today will not be detected because they were not invented then. Snort does have a repository for older rulesets and older versions of the program [52], but not more than a few years back. They strongly recommend not using old and unsupported versions. Another issue is that Snort did not even exist at the time of the DARPA dataset collection [51].

In Snort, many older rules are still available and only commented out because the creators have deemed them not to be relevant anymore [51]. It is not documented when they were commented out or why, but a systems administrator could easily re-activate them. Still, what would be the purpose of using old versions of an IDS and rules to optimize for the past, when in a real-life scenario, you are forward looking, focusing on the present and the near future threats?

Some of the strengths of the DARPA datasets and its derivatives are that it is widely used and that is labelled and very well documented in regards of host filesystem listings and the network topology and setup. The advantages did not compensate for the disadvantages of using an old dataset, so we decided to look for more recent datasets to work with. Literature review and recommendations led us to the CTU-13 and ISCX datasets [47], [51].

We conclude, that for the best tuning of an IDS should be done with a dataset that is from the same time as the IDS version and ruleset. It should also be supplemented with a complete documentation and contain relevant traffic.

7.2.2 Pre-test 2 CTU-13 dataset results

The CTU-13 dataset is a more contemporary dataset than DARPA. It is claimed to cover or mimic a number of modern attacks, in this case botnet attacks. It triggered a various number of alarms. Even though Scenario 10 was corrupt we decided to continue using this dataset. Since the dataset consists of a series of capture files of suitable size and content, we decided to use the whole dataset for our work. The verification can preferably be done on a separate dataset that also contains botnet traffic.

Using the blacklist function can be a very blunt instrument, especially where the blacklisted IP exists within the analyzed network, thus generating a lot of false positives. A blacklisting rule must be refined and elaborated much more than we did in this experiment.

During the pre-tests no packets were dropped, as expected.

7.3 IDS Precision

7.3.1 Test result of alerts on suspicious domain

The test started with a forensic investigation of the .pcap-file for scenario 13 in the dataset to gather IOCs that in step 2 of our implemented process TIDS where parsed to create rules for the IDS. We conclude that this manual gathering of IOCs is a very crucial step if the purpose is to achieve higher precision from the IDS, and to perform this well you need a very thorough understanding of the traffic on the specific network. The downside of performing forensic analysis

51

is that it is time consuming and it is up to every organization to make the decision whether the organizations data is worth the cost, both in terms of personnel as well as economy. The cost of the work should be lower than the anticipated loss in case of an attack.

7.3.1.1 Iteration 1

The first iteration tested whether any external artefact communicated with the DNS server that where found communicating with the fast-flux infected Windows machine in scenario 13. No alerts where created by the local rule and it validate that only internal traffic where communicated. To follow our TIDS process, we continued with another iteration.

7.3.1.2 Iteration 2

In the second iteration, we continued the test by adding another rule that was written to look for occurrences where any artefact inside the local network was communicating with the identified ip-address. This rule gave us 10 hits and when we examined them further we saw that the recently discovered ip-address 147.32.80.9 communicated with the ip-address infected from botnet traffic, 147.32.84.165.

7.3.1.3 Iteration 3

To examine the result from iteration 2, we created a rule with a bidirectional operator to explore how the machines had communicated with each other. The rule created 15 hits containing of communication between the infected machine and the DNS server, no other artefacts were involved.

7.3.1.4 Iteration 4

For the fourth iteration, we disabled 5407rules by altering the snort configuration file. We find this as a very important step when it comes to tune the IDS since you should never implement rules that is out of scope for the specific network and data traffic. The run time was lowered but the IDS still generated the same number of alerts.

One issue regarding disabling rules is the risk of disabling a rule that is out of the scope today but may be topical tomorrow. Therefor we stress that disabling rules should be preceded by a thorough analyze of what domain the disabled rule will affect.

7.3.2 Test result of alerts on any external traffic

7.3.2.1 Iteration 1

As can be seen in the following comparison with the original Snort registered rules, adding the alert for traffic from outside the internal network increases the alert rate (% of total number of packets) slightly in most of the scenarios. In Scenario 13, however, adding the external IP alert causes the alert rate to decrease.

Scenarios 4 and 11 have a very low alert rate in both cases. In these scenarios the malicious traffic is originating from the internal network towards external IPs.

52

Figure 21 Registered rules/External IPs

Table 16 Iteration 1 of alerts of any external traffic. Scenarios are greyed if the new rule generates no true positives.

Scenario Number of alerts Number of alerts generated by rule

True Positives Precision of alerts generated by rule (TP/FP+TP)

1 901 0 0 0,00%

2 2246 336 211 62,80%

3 5688 0 0 0,00%

4 7 0 0 0,00%

5 110 10 0 0,00%

6 88 0 0 0,00%

7 11 0 0 0,00%

8 494 53 3 5,66%

9 4864 550 19 3,45%

11 4 3 3 100,00%

12 153 2 0 0,00%

13 2752 743 4 0,54%

The added rule did not generate any direct alerts connected to the specific “sid”. The rule was designed to look at specific content within the traffic and there were no matches. It caused, however, additional alerts of other types to be created, for example (portscan) TCP and UDP portsweep, malware_cnc and htttp_inspect.

7.3.2.2 Iteration 2

During iteration 2, it was clear that many alerts originated from an internal host of the network, not documented in the network description. The rule led to a lot of false alerts from the inside IP address. This highlights the importance of having the entire internal network well documented.

0

0,5

1

1,5

2

2,5Alerts based on Registered Rules vs adding External IPs

Registered rules External IP

53

A decision was made to skip the scan for specific content since it did not trigger any alerts with the selected “sid”. To limit the number of alerts, the rule was narrowed down to specific ports 21,22,80 and 443.




1 1120 219 1 0,46%

2 1970 60 1 1,67%

3 6344 656 39 5,95%

4 35 28 20 71,43%

5 110 0 0 0,00%

6 135 47 0 0,00%

7 11 0 0 0,00%

8 648 154 3 1,95%

9 4878 14 12 85,71%

11 34 30 30 100,00%

12 155 2 0 0,00%

13 2980 228 9 3,95%

7.3.2.3 Iteration 3

During iteration 3, the newly detected internal IP address was included in the rule, which improved the precision.




1 902 1 1 100,00%

2 1911 1 1 100,00%

3 5728 40 40 100,00%

4 27 20 20 100,00%

5 110 0 0 0,00%

6 88 0 0 0,00%

7 11 0 0 0,00%

8 497 3 3 100,00%

9 4864 12 12 100,00%

11 34 30 30 100,00%

12 153 0 0 0,00%

13 2760 8 8 100,00%

54

The precision of the new alerts has been assessed against the intention of the rule, to identify any external traffic. In a live scenario, external traffic cannot be judged as malign per se, it must be further analyzed before the system administrator can make a judgement on that. Just analyzing all IP numbers involved in sending packets is not very effective. For example, in Scenario 13, all the IP addresses in the pcap file are plotted on the following map, created in Wireshark:

Figure 22 Plotted external IP addresses.

Some of the external IP-addresses come from harmless IP mapping services and some are from suspicious domains. The interesting IP addresses are filtered and identified through the IDS processing.

In all but one of the scenarios, the efforts of adding the extra rule created only a few more alarms. In Scenario 13, the malicious traffic was primarily generated from within the network towards external hosts and that is probably why the rule did not work very well.

A reflection of this test is that refining a rule for high precision can require iterations and can be time consuming, while the end result is meager in comparison to the number of alerts already generated by the standard IDS ruleset. On the other hand, it is the seriousness of the alert that matters the most, not the quantity.

The conclusion of this test is that adding a rule that creates alerts for external IP addresses adds some true positives, which in turn improves the precision somewhat, but not much.

55

7.3.3 Test result of vulnerability scan and traffic to unusual ports

7.3.3.1 Iteration 1

Scenarios 3,6 and 8 show a higher alert rate compared to the other scenarios and the Snort registered rules.

Figure 23 Registered rules/traffic.

Table 19 Iteration 1 test of vulnerability scan and traffic to unusual ports. Scenarios are greyed if the new rule generates

no true positives.


True positives Precision of alerts generated by rule (TP/FP+TP)

1 901 0 0 0,00%

2 1901 0 0 0,00%

3 89672 83984 83984 100,00%

4 7 0 0 0,00%

5 110 0 0 0,00%

6 1110 1022 1022 100,00%

7 11 0 0 0,00%

8 4613 4119 4119 100,00%

9 5587 1023 1023 100,00%

11 4 0 0 0,00%

12 153 0 0 0,00%

13 5456 2704 2704 100,00%

0

5

10

15

20

Alerts based on Registered Rules vs traffic to port 3389 with weakness

Registered rules Weakness in Port 3389

56

In this test there were extensive traffic to port 3389 in scenarios 3, 6, 8, 9 and 13. Most of the traffic to that port originates from IP addresses in China, then the U.S and Czech Republic and a few from Germany, Luxembourg and Hungary. Because the port carries a weakness and there should not be any traffic to that port, all alarms are regarded as true positives. A deeper analysis of the companies behind the IP addresses might reveal that some of them are benign, but with the current knowledge we regard them as malign.

7.3.3.2 Iteration 2

In this iteration we ran the test with a rule including all the hosts claimed to and assumed to be Windows XP hosts to get a more complete picture of the possible attacks to the weakness in port 3389.

The result was the same as in iteration 1. The only host subject to suspicious traffic on port 3389 is the host 147.32.84.165.

Result and discussion

In this dataset and network, with traffic towards an unusual port on a host with a known weakness, creating an IDS rule for that seems to be good practice since it generated true positives in five of the scenarios. A similar approach could probably be effective in other networks as well.

One could assume that a logical response to a known weakness would not to be to put a lot of effort into creating alerts for attacks, but rather to fix the weakness itself. In this case mitigate by upgrading from Windows XP to a newer and supported operating system. There may be software dependencies towards specific operating systems or other reasons why it is not possible to mitigate the weakness at that time. In such cases a risk analysis would give input to the decision.

To summarize, if a system administrator notices suspicious traffic towards a specific port, it is probably worthwhile to write a local IDS rule that triggers alarms for that activity.

57

7.3.4 Test result of spikes in network traffic

The test is performed by creating an Input and Output graph from Wireshark from the dataset .pcap files. A check is also made against the alert file created by Snort by using the standard ruleset. A typical alert for botnet attacks is called Malware-CNC. Here we test if there is any correlation in time between traffic spikes and Malware-CNC alerts.

Scenario 1: A bot sending spam, connected to an HTTP CC, using HTTP to do ClickFraud.

Timeline: 1 hour of background traffic, then attack up until 5 minutes before the end of capture.

Figure 24 Scenario 1 Botnet-capture.

The test continues with checking for occurrence of Malware-CNC alerts in the dataset.

Figure 25 Scenario 1 Malware-CNC alerts

First occurrence of Malware-CNC alert: 11:06:43

Last occurrence of Malware-CNC alert: 15:48:02

2

3

2

1

3

1

2

44

2

1

33

2

4

333

4444

3

4

5

3

2

444

1

3

2

4

22

5

2

3

11

3

5

3

2

4

2

33

4

33

2

3

4

2

4

3

44

22

44

2

4

1111111111111

3

2

3

11111

2

1111

22

4

11111111

3

1

2

1

3

2

3

1

2

111

2

11

2

3

2

111

3

2

1

3

2

4

22

44

33

2

3

4

3

44

2

44

2

33

4

2

1111

2

11111

0

1

2

3

4

5

6

:06:19:24:29:34:39:44:49:10:27:32:37:42:47:02:13:27:42:57:05:14:28:35:44:52:58:03:08:13:18:29:42

11 12 13 14 15

Nu

mb

er

of

Ale

rt o

ccu

rre

nce

s

Time of day

58

The test continues with the remaining scenarios. The result is summarized in the following table

and the graphs are available in the Appendix of this thesis.

Table 20 Correlation between spikes in network traffic and alerts. Scenarios are greyed if there are no spikes and/or

alerts.

Scenario Description Spikes in network

traffic

Malware-CNC Alerts

Correlation between Spikes

and Alerts 1 A bot sending spam, connected to an HTTP

CC, using HTTP to do ClickFraud. Yes Yes No

2 Neris malware attack Yes Yes Yes 3 The virtual machine is infected with rbot.exe,

that can be controlled with IRC Yes No N/A

4 DOS attack using rbot.exe Yes No N/A 5 VM infected with fast-flux malware Yes Yes No 6 Donbot attack No No N/A 7 Sogou botnet sending spam, connected to IRC,

using HTTP to do ClickFraud. Yes No N/A

8 Murlo botnet, C&C to China hosts Yes Yes Yes 9 Mimic of a real Neris infection from client to

client Yes Yes No

11 ICMP DOS to a computer by an IRC botnet controlled from the lab

Yes No N/A

12 NSIS.ay botnet Yes No N/A 13 Infection of VMs with a fast-flux malware Yes Yes No

The reason for conducting this test was the idea that knowledge of the normal traffic can be used to distinguish malicious and irregular traffic from normal traffic. In all the test scenarios except number 6, there are spikes in the network traffic. This could indicate that botnet traffic leaves traces in the packets/seconds statistics. If the spikes are related to the botnet traffic it should be possible to create a rule that raises an alert when the traffic load exceeds a certain percent of the average normal load. What that load and the tolerance is, will vary from network to network. If the spikes in the network traffic instead are normal variations, it will be much harder to draw any definite conclusions.

In all the scenarios but number 6, there are clearly identifiable spikes in the network traffic. In the scenarios 3, 6, 7, 11 and 12, no Malware-CNC alerts were created. Scenarios 1, 2, 5, 8, 9 and 12 both had spikes in network traffic and Malware-CNC Alerts. Those scenarios were studied to see whether there exists any correlation between the two.

The results vary and are inconclusive. There seems to be no correlation between the occurrence of Malware-CNC alerts and spikes in network traffic in the different scenarios. In Scenario 1, in fact the pattern is almost the reverse. In Scenario 2 there might be a correlation.

It is difficult to draw any conclusions from the traffic since the capture files of the dataset have not been analyzed in detail. We do not know the pattern of the normal traffic in the network.

The cyclic or periodic traffic in Scenario 8 is interesting since it seems to be reflected also in the Malware-CNC alerts. It could be investigated further if this is typical for other botnet attacks and other datasets as well. Perhaps also different botnets have unique patterns or signatures that could help in identifying them. To summarize, an analysis of this dataset alone gives no clear indication that spikes in network traffic is a sign of a botnet attack. Especially since we do not know the normal pattern of the background traffic. It is probably better to analyze the individual alerts and their contents in more detail.

59

7.3.5 General results

One result that is consistent throughout all the tests is that there was no packet loss. In our testbed the IDS processed the capture files in its own pace and was never forced to drop any packets. In a live scenario with high traffic load and/or insufficient capacity of the IDS, there would be a risk for false negatives due to packet loss. If the tuned IOC rules are poorly written, too many etc. so that they require a lot of IDS resources, the risk for false negatives increases.

When analyzing alerts created by the added rules, there were alarms that were definitely true positives or false positives. There was also a portion of alarms that were not so easy to categorize. For example, an alarm regarding benign traffic from a suspicious domain, should it be categorized as a true or false positive? Traffic from certain countries could be regarded as true or false positives depending on which country the monitored network is situated in. There could be a localization factor in IOC management that may need consideration. In short, the categorization of alerts as true or false positives was harder than we initially thought.

7.4 Design iterations

During this research, several different testbeds have been explored. For example,

• IDS installation on several different operation systems, like Linux Ubuntu, Fedora, Kali, SecurityOnion and Windows 7and Windows 10.

• IDS scanning on different data sets with recorded traffic, DARPA, CTU-13

• Interpretation of logs in Excel, Wireshark, Notepad++ and Syslog server, Orange. During the test phase, the graphical tool NetworkMiner was added.

The reason for how our final testbed is configured is because we have more knowledge of Windows environment and apart from minor changes in the Snort configuration file, the installation was straight forward compared to Linux.

A disadvantage with a windows installation is that many of the externally developed add-ons or tools that are compatible with the Snort architecture are developed for Linux, which made the analysis phase more cumbersome.

The choice of datasets was subject to iterations. In the beginning of the thesis work the ambition was to use the DARPA dataset, but CTU-13 was chosen instead because it is more contemporary. The main reasons for using publicly available datasets was that they are labelled and documented, and the tests would be repeatable by someone else. The plan was also to verify the results against a separate dataset, possibly a live recording. The live recording was discarded for legal and confidentiality reasons. One option could have been to create a simulation environment, but this was also discarded due to lack of time. A possible alternative is to use another public dataset with botnet traffic, for example the ISCX Botnet dataset from the Canadian Institute for Cybersecurity.

In the beginning of the thesis project, the focus was towards IOCs. As the project evolved with the selection of Snort IDS and building of the testbed, the focus shifted towards installation and different IDS optimisation methods which was to wide. We started designing TIDS for a broader scope than initially intended. In the end, we once again focused on the core issues, IOCs and IDS rule tuning and precision.

60

7.5 Validity of the design

To validate the improvement of the IDS precision one dataset could be used to test and optimize against and another to validate with. A possible selection is the ISCX Botnet dataset from the Canadian Institute for Cybersecurity. It contains similar traffic as the CTU-13, but is claimed to include signatures from a wide variety of botnets with realistic background traffic and with the representativeness of a production network [55].

What could be done in a production environment is to record normal traffic over a period of time and optimize the rules according to it. Maybe also, if possible, disconnect from the Internet for a while in order to capture pure internal traffic to use as baseline and training data. Finally, also to implement the solution on the live production network.

TIDS process – the process should be implemented in a real case scenario to validate whether the steps and activities created are solid and covers the whole tuning cycle.

7.6 Answer to research questions

Research question 1: How can an intrusion detection system’s precision be improved using rules from parsed indicators of compromise?

The precision was improved by utilizing knowledge of the monitored network, its context and network traffic by:

• Adding custom rules based on known vulnerabilities in the network

• Adding custom rules based on traffic from suspicious domains

• Disabling rules that are out of scope for the particular network and data traffic.

A minor precision improvement was achieved by creating a custom rule that creates alerts for external traffic.

Correlating spikes in network traffic to alarms did not improve the precision.


The research question is implicitly answered by the creation of the TIDS process, as it has been designed and implemented during chapters five and six.

61

Chapter 8: Conclusions and Further work

The threat landscape of today is changing quickly. Snort was created as an open source IDS/IPS with the purpose of monitoring and blocking malicious traffic in a network. The rulesets available are updated quickly and very comprehensive. Because they have a broad-spectrum character they need to be tuned in order to perform well and have a high precision.

In this thesis, we have used the Snort in IDS mode. Snort is a lightweight yet highly configurable IDS. It is easy to install but requires deep knowledge of the system administrator in order to utilize its functions in an optimal way. Learning the tools and is crucial for the result and takes a lot of time and effort. The official documentation for Snort is not very detailed, rather more on a checklist level. To be a qualified systems administrator you need to have deep knowledge of every function in Snort, know what every line of code does, so the relevant functions can be turned on or off.

We have realized that the craft of writing optimized rules is a profession you learn over a lifetime and nothing that you master after only a few months (of thesis) work. Our ambition was nevertheless to study if it was possible to tune the rulesets from knowledge of certain indicators of compromise and to document a process for tuning.

We conclude that Snort is a very competent tool in itself and has good ability to identify malicious traffic even with its "out-of-the-box" settings. It is, however, possible to improve the precision using custom local rules, without creating a lot of false positives. It takes skill and an iterative approach to optimize the new rules. It also takes skill to interpret the alerts and define what a true positive is.

Limitations of our work are that the tests are made on one dataset with one type of attack. Maybe the results would be different if the attack spectrum was wider or if the tests were made on multiple datasets. Another limitation is that verification on a separate dataset or on a production network was not within the scope of this thesis.

The main obstacles in our work is that this was a new field for us to examine. It took time to build the necessary understanding of the subject and tools that we used.

Working in pair gave us the opportunity to draw from two persons’ experiences, to discuss ideas and to feel commitment to an agreed time schedule. On the other hand, reaching consensus is easier when working solo.

If we started all over again with the knowledge that we have now, we would be able to create a more comprehensive tool chain using Snort and the ecosystem of software available around it. We would also make a faster and more accurate selection of datasets since we have better knowledge of the features that make a dataset appropriate for its purpose. We would also spend more time on statistical data analysis and visualization in order to retrieve more information from the data.

Our suggestions for further research is to continue to analyze more datasets, but those datasets need to have a comprehensive description of the network topology, the nodes, the hosts including operating systems versions and software used, plus information about how the traffic is routed. With that information, the prerequisites for parsing IOCs into useful IDS rules are much better. Another suggestion is to study how automatic parsing of IOCs and self-learning mechanisms could be combined with manual tuning. Further research could also compare solutions for different deployment domains, to see if their characteristics require different types of tuning.

There is so much test and research possible to perform. The question is how we could ever say that we know something for a fact since the binaries get updated, the rules get tuned, and the malware constantly evolves. What we know today is old knowledge tomorrow. In other words, the opportunities for further research are great.

62

References

[1] D. Bianco, “The Pyramid of Pain,” Enterprise Detection & Response, 17-Jan-2014. Private

Blog.

[2] L. Rudman and B. Irwin, “Dridex: Analysis of the traffic and automatic generation of

IOCs,” presented at the 2016 Information Security for South Africa (ISSA), Johannesburg,

South Africa, 2016, pp. 77–84.

[3] W. Gibb and D. Kerr, “OpenIOC: Back to the Basics,” OpenIOC: Back to the Basics, 01-

Oct-2013. Commercial Blog.

[4] B. Mukherjee, L. T. Heberlein, and K. N. Levitt, “Network intrusion detection,” IEEE

Netw., vol. 8, no. 3, pp. 26–41, May 1994.

[5] B. Genge, P. Haller, C.-D. Dumitru, and C. Enachescu, “Designing Optimal and Resilient

Intrusion Detection Architectures for Smart Grids,” IEEE Trans. Smart Grid, vol. 8, no. 5,

pp. 2440–2451, Sep. 2017.

[6] M. Horkan, “Challenges for IDS/IPS Deployment in Industrial Control Systems,” Master,

SANS Institute InfoSec Reading Room, 2015.

[7] R. Berthier, W. H. Sanders, and H. Khurana, “Intrusion detection for advanced metering

infrastructures: Requirements and architectural directions,” presented at the IEEE

International Smart Grid Communications, Gaithersburg, USA, 2010, pp. 350–355.

[8] R. Berthier and W. H. Sanders, “Specification-Based Intrusion Detection for Advanced

Metering Infrastructures,” presented at the Dependable Computing (PRDC), 2011 IEEE

17th Pacific Rim International Symposium on, Pasadena, CA, USA, 2011, pp. 184–193.

[9] K. Barker, K. Wallace, and M. D. Taylor, CompTIA Network+ N10-006 cert guide.

Computing Technology Industry Association, 2015.

[10] D. L. Prowse, CompTIA security+ SY0-401 authorized cert guide, Deluxe edition. Third

edition. Indianapolis, Indiana, USA: Pearson, 2015.

[11] C. Sanders and J. Smith, Applied network security monitoring collection, detection, and

analysis. Waltham, MA: Syngress, an imprint of Elsevier, 2014.

[12] K. Goeschel, “Reducing false positives in intrusion detection systems using data-mining

techniques utilizing support vector machines, decision trees, and naive Bayes for off-line

analysis,” presented at the SoutheastCon 2016, Norfolk, VA, USA, 2016, pp. 1–6.

[13] G. Dupont, “Threat Intelligence and SIEM (Part 4) — An Active Cyber Defense

Approach,” Threat Intelligence and SIEM, Apr-2016. Commercial Blog.

[14] X. Liao, K. Yuan, X. Wang, Z. Li, L. Xing, and R. Beyah, “Acing the IOC Game: Toward

Automatic Discovery and Analysis of Open-Source Cyber Threat Intelligence,” presented

at the 2016 ACM SIGSAC Conference on Computer and Communications Security,

Vienna, Austria, 2016, vol. 2016, pp. 755–766.

[15] S. García, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet

detection methods,” Comput. Secur., vol. 45, pp. 100–123, Sep. 2014.

[16] W. Park and S. Ahn, “Performance Comparison and Detection Analysis in Snort and

Suricata Environment,” Wirel. Pers. Commun., vol. 94, no. 2, pp. 241–252, May 2017.

[17] V. Nagadevara, “Evaluation of Intrusion Detection Systems under Denial of Service Attack

in virtual Environment,” Faculty of Computing, Blekinge Institute of Technology,

Karlskrona Sweden, 2017.

[18] E. Koen, “Indicators of Compromise and where to find them,” Security, 10-Feb-2017.

Commercial Blog.

63

[19] D. Ross, J. Shiffer, T. Dell, W. Gibb, and D. Wilson, “OpenIOC 1.1 Readme,” OpenIOC

1.1 Draft, Aug-2013. Commercial Blog.

[20] P. Catherine, “Network Security Using Cisco IOS IPS,” Network Security Using Cisco IOS

IPS, 08-Jun-2009. Commercial Blog.

[21] A.-S. K. Pathan, Ed., The state of the art in intrusion prevention and detection. Boca Raton,

Florida: CRC Press/Taylor & Francis Group, 2014.

[22] N. Venkatesh, “Evaluation of Intrusion Detection Systems under Denial of Service Attack

in virtual Environment,” Master, Blekinge Institute of Technology, Karlskrona, 2017.

[23] K. Barker, Comptia network+ n10-006 cert guide, 1st edition. Indianapolis, IN: Pearson IT

Certification, 2015.

[24] V. Shah, A. Aggarwal, and N. Chaubey, “Alert Fusion of Intrusion Detection systems using

Fuzzy Dempster shafer Theory,” J. Eng. Sci. Technol. Rev., vol. 10, no. 3, pp. 123–127, Jul.

2017.

[25] A. Javaid, Q. Niyaz, W. Sun, and M. Alam, “A Deep Learning Approach for Network

Intrusion Detection System,” EAI Endorsed Trans. Secur. Saf., vol. 2016, no. 09, 2016.

[26] E. Hodo et al., “Threat analysis of IoT networks using artificial neural network intrusion

detection system,” presented at the 2016 International Symposium on Networks, Computers

and Communications (ISNCC), Yasmine Hammamet, Tunisia, 2016, pp. 1–6.

[27] M. M. A. Abdallh, A. Babiker, and K. H. Bilal, “Machine Learning Algorithms,” Int. J.

Eng. Appl. Manag. Sci. Paradig., vol. 36, no. 01, Jun. 2016.

[28] Yang Haopu, “Method for behavior-prediction of APT attack based on dynamic Bayesian

game,” presented at the IEEE International Conference on Cloud Computing and Big Data

Analysis (ICCCBDA), Chengdu, China, 2016, vol. 2016, pp. 177–182.

[29] M. Mahurin, “Basic NGIPS Operation and Management for Intrusion Analysts,” SANS

Institute InfoSec Reading Room, 2017.

[30] R. Dewanjee and R. Vyas, “A Study on IDS (Intrusion Detection System) and Introduction

of IFS (Intrusion Filtration System),” in Computing and Network Sustainability, vol. 12,

Singapore: Springer Singapore, 2017, pp. 119–126.

[31] J. Andress, “Working with Indicators of Compromise,” ISSA J. May 2015, vol. 13, no. 5,

pp. 14–20, May 2015.

[32] J. S. White, T. Fitzsimmons, and J. N. Matthews, “Quantitative analysis of intrusion

detection systems: Snort and Suricata,” presented at the SPIE Defense, Security, and

Sensing 2013, Baltimore, Maryland, United States, 2013, vol. Proc. SPIE 8757, p. 875704.

[33] S. A. R. Shah and B. Issac, “Performance comparison of intrusion detection systems and

application of machine learning to Snort system,” Future Gener. Comput. Syst., vol. 80, pp.

157–170, Mar. 2018.

[34] D. Hedemalm, “An empirical comparison of the market-leading IDS’s,” Bachelor,

Halmstad University, Halmstad, 2018.

[35] K. Thongkanchorn, S. Ngamsuriyaroj, and V. Visoottiviseth, “Evaluation studies of three

intrusion detection systems under various attacks and rule sets,” presented at the 2013 IEEE

International Conference of IEEE Region 10 (TENCON 2013), Xi’an, China, 2013, vol.

2013, pp. 1–4.

[36] M. Roesch, C. Green, and B. Caswell, “SNORT Users Manual.” Cisco, 31-Aug-2017.

[37] J. S. White, T. Fitzsimmons, and J. N. Matthews, “Quantitative analysis of intrusion

detection systems: Snort and Suricata,” 2013, p. 875704.

64

[38] V. Shah, A. Aggarwal, and N. Chaubey, “Alert Fusion of Intrusion Detection systems using

Fuzzy Dempster shafer Theory,” J. Eng. Sci. Technol. Rev., vol. 10, no. 3, pp. 123–127, Jul.

2017.

[39] N. Pappas, “Network IDS & IPS Deployment Strategies.” Sans Institute, 2008.

[40] A. Panwar, “iGen: Toward Automatic Generation and Analysis of Indicators of

Compromise (IOCs) using Convolutional Neural Network,” Arizona state university,

Arizona, 2017.

[41] T. Sommestad and U. Franke, “A test of intrusion alert filtering based on network

information: Intrusion alert filtering,” Secur. Commun. Netw., vol. 8, no. 13, pp. 2291–

2301, Sep. 2015.

[42] C. Harrington, “Sharing Indicators of Compromise: An Overview of Standards and

Formats,” presented at the RSA Conference, San Francisco, 27-Feb-2013.

[43] D. Rhoades, “Machine actionable indicators of compromise,” in 2014 International

Carnahan Conference on Security Technology (ICCST), Rome, Italy, 2014, pp. 1–5.

[44] J. Wang, Y. Li, and H. R. Rao, “Overconfidence in Phishing Email Detection,” J. Assoc.

Inf. Syst., vol. 17, no. 11, pp. 759–783, Nov. 2016.

[45] L. Zaichkowsky, “Hunting for Indicators of Compromise,” presented at the RSA

Conference, 2013.

[46] M. D. Rowell, “Cyber indicators of compromise: a domain ontology for security

information and event management,” Master, Naval Postgraduate school, Monterey,

California, 2017.

[47] M. Malowidzki, P. Berezinski, and M. Mazur, “Network Intrusion Detection: Half a

Kingdom for a Good Dataset.” Military Communication Institute, 2015.

[48] T. Sommestad and A. Hunstad, “Intrusion detection and the role of the system

administrator,” Inf. Manag. Comput. Secur., vol. 21, no. 1, pp. 30–40, Mar. 2013.

[49] K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A Design Science

Research Methodology for Information Systems Research,” J. Manag. Inf. Syst., vol. 24,

no. 3, pp. 45–77, Dec. 2007.

[50] W. J. Orlikowski and C. S. Iacono, “Research Commentary: Desperately Seeking the ‘IT’

in IT Research—A Call to Theorizing the IT Artifact,” Inf. Syst. Res., vol. 12, no. 2, pp.

121–134, Jun. 2001.

[51] T. Sommestad, “Interview with researcher within IDS technology,” 18-May-2018.

[52] M. Roesch, Snort. Cisco, 2018.

[53] V. Julien, Suricata - Open Source IDS / IPS / NSM engine. Open Information Security

Foundation, 2009.

[54] L. Khalvati, M. Keshtgary, and N. Rikhtegar, “Intrusion Detection based on a Novel Hybrid

Learning Approach,” J. AI Data Min., vol. 6, no. 1, pp. 157–162, 2018.

[55] E. Biglar Beigi, H. Hadian Jazi, N. Stakhanova, and A. A. Ghorbani, “Towards effective

feature selection in machine learning-based botnet detection approaches,” presented at the

2014 IEEE Conference on Communications and Network Security, 2014, pp. 247–255.

65

Appendix

Continuation of 7.2.4 Test results of spikes in network traffic.

The purpose of this test is to examine whether it is a useful indicator of compromise, if and when, there are unusual spikes in the network traffic. If botnet attacks generate a lot of network activity, it might be possible to see a correlation between malware alarms and traffic spikes.

All Wireshark I/O graphs are listed in Scenario order 1-9, 11-13. If there are Malware-CNC alerts in a scenario, a graph showing the frequency per minute of the capture, is also included directly following the I/O graph.

Scenario 1: A bot sending spam, connected to an HTTP CC, using HTTP to do ClickFraud.

Timeline: 1 hour of background traffic, then attack up until 5 minutes before the end of capture.

66

Scenario 2: Neris malware attack

Timeline: Attack from 10:33:04 to 13:55:37



4 4

1 1 1 1 1 1 1 1 1

3

4

1 1 1 1 1 1 1 1 1 1 1 1 1

2

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

:37

:39

:41

:43

:48

:08

:38

:45

:47

:49

:52

:54

:56

:58

:00

:02

:07

:12

:31

:24

:32

:40

:42

:44

:49

:52

:54

10 11 12 13

Nu

mb

er

of

Ale

rt o

ccu

rre

nce

s

Time of day

Summa

67

Scenario 3: The virtual machine is infected with rbot.exe, that can be controlled with IRC

Timeline: Attack from Fri Aug 12 15:00:37 CEST 2011 to Mon Aug 15 09:55:11 CEST 2011

First occurrence of Malware-CNC alert: No occurrence

Last occurrence of Malware-CNC alert: No occurrence

Scenario 4: DOS attack using rbot.exe

Timeline: Started bot 10:59:23. DOS attack 12:21:33 to 13:06:40



68

Scenario 5: VM infected with fast-flux malware

Timeline: 16:52:23 to 17:13:07


Last occurrence of Malware-CNC alert: Same as first occurrence.

Scenario 6: Donbot attack

Timeline: Attack from 10:08:47 to 12:10:31



69

Scenario 7: Sogou botnet sending spam, connected to IRC, using HTTP to do ClickFraud.

Timeline: Bot running from 13:56:07 to 14:12:17



70

Scenario 8: Murlo botnet, C&C to China hosts

Timeline: According to documentation attack and capture range from 14:19:28 to 14:24:01 Tue Aug 16 2011, but the correct time according to dataset capture ranges from



0

1

2

3

4

5

6

7

8

9

10

Nu

mb

er

of

Ale

rt o

ccu

rre

nce

s

Time of day

71

Scenario 9: Mimic of a real Neris infection from client to client

Timeline: from 13:01:03 incoming external attacks not originating from the lab.

Bot infection from 14:24:08 to 17:10:28.



Scenario 10: Corrupt and therefore omitted.

2

1 1 1 1 1 1

2 2 2 2 2

1

2 2

1 1

3

1 1 1 1

2

0

0,5

1

1,5

2

2,5

3

3,5

:24 :34 :43 :44 :48 :49 :54 :55 :56 :58 :01 :02 :03 :04 :05 :06 :11 :14 :15 :22 :32 :42 :52

14 15

Nu

mb

er o

f A

lert

occ

urr

ence

s

Time of day

72

Scenario 11: ICMP DOS to a computer by an IRC botnet controlled from the lab

Timeline: ICMP attack from 15:52:39 to 15:54:44



Scenario 12: NSIS.ay botnet

Timeline: Infection from 10:42:39 to 10:50:25. VMs stopped at 11:44:17



73

Scenario 13: Infection of VMs with a fast-flux malware

Timeline: Bot running from Mon Aug 15 17:18:01 to Tue Aug 16 09:35:05



0

0,5

1

1,5

2

2,5

21 17 17 09

:14 :19 :20 :21

Nu

mb

er o

f Ale

rt o

ccu

rren

ces

Time of day

Improving the precision of an Intrusion Detection...

Documents

Transcript of Improving the precision of an Intrusion Detection...