Speaker ： Hong-Ren Jiang

A Novel Testbed for Detection of Malicious Software Functionality

1

OutlineAbstract

Introduction

Related Work

Detection TheoryHost-Based DetectionNetwork-Based Detection

Laboratory SetupPhysical LevelVirtual Level

Experiments and Results

Discussion and Further Work

Conclusion2

Abstract

3

This paper presents a novel open-source testbed for behavioural software analysis.designed to meet current trends in the malware community by allowing controlled access to the Internet in the analysis phase.A novel way of using honeypot technology is proposed to build a testbed that is able to analyse current threats.

Introduction (1/2)

4

Malicious code may be spread by introducing malicious hidden functionality.From the more than 500.000 files they investigated,approximately 15% contained malicious code.However, attackers are constantly improving their code to evade the traditional detection algorithms.

Introduction (2/2)

5

Sandboxing is a relatively new approach to behavioural malware detection.However, a trend among malware authors, presented by Symantec,is to use staged downloads of malware.A modern behavioural analysis environment designed to detect unwanted functionality in downloaded applications.

Related Work

6

Virtual machine technology is proposed as a flexible solution for building the laboratory environment.such as Norman Sandbox and Panda TruPrevent.

This paper contributes to existing malware analysis documentation in the following areas:

A powerful open-source software analysis testbed is providedThe testbed design is based on a number of updated malware detection theoriesThe testbed allows for safe analysis of software,also with malicious content, utilizing staged download techniques.

Detection Theory

7

Most traditional anti-virus systems use signature-based detection,the major drawback being the inability to detect new, unknown malicious code. When new instances of malicious software are found, new signature files have to be written and distributed to the detection tools.

Host-Based Detection(1/2)

8

This section presents techniques that can be used to observe changes in the host operating system,during or immediately after installation of an application. Wang et al.suggest complementing the traditional approaches with a concept they refer to as Auto-Start Extensibility Points.an overwhelming majority of all spyware programs infect a system in such a way that they are automatically started upon reboot and the launch of most commonly used applications.

Host-Based Detection(2/2)

9

Software using the auto-start extensibility points is distinguished by two categories:

Standalone applications that are automatically run by registering as an OS auto-start extension, such as a Windows NT serviceExtensions to existing applications that are automatically run, or extensions to popular applications commonly run by users, like web browsers

Wang et al.introduce a concept called cross-view diff to detect stealth software.Cross-time diff aims at comparing the state of a running system with a previous snapshot of the same system. such as Tripwire.

Network-Based Detection(1/4)

10

Network-based detection refers to techniques used to discover the presence of malicious entities by studying properties related to network activity.Router Access Lists is to update routers with records of hosts or network segments of which are allowed or denied access to network resources. In a behavioural analysis scenario, access control lists can be utilized to help detect usage of address spoofing.

Network-Based Detection(2/4)

11

Intrusion detection systems (IDS) are utilities used to inspect both network flow and the content of each packet sent on a network.Mukherjee et al.suggest using IDSs to detect the presence of malware or intruders on a network.Backdoors can be detected and payload fields.Polymorphic worms have proved to make IDS-based detection more difficult.recent research has demonstrated that even such obfuscation techniques are possible to handle

Network-Based Detection(3/4)

12

Spitzner is one of the leaders of the honeypot development.A honeypot is a security resource whose value lies in being probed, attacked, or compromised.Honeypots can be used in production environments to help raising alerts of unauthorized activity on the network and to slow down worms.Honeypots are categorized into three different classes: low-interaction, medium-interaction and highinteraction honeypots.

Network-Based Detection(4/4)

13

Malware enabled to communicate with its environment utilizes different techniques to establish the communication.Skaggs et al.suggest using remote vulnerability scanners to detect the presence of malware on a host.

Laboratory Setup(1/3)

14

Sherlock is configured as a firewallMarple is a “gatekeeper”Horatio acts as a remote vulnerability scannerDrew is a network sniffer Bond is the victim where malicious software is installed.

Laboratory Setup(2/3)

15

Laboratory Setup(3/3)

16

Physical Level

17

It is critical that the physical level has been subject to system hardening so that the probability of unwanted infections is reduced to a minimum.The probability of malware being able to infect both Windows and Linux systems is relatively low, though greater than nil.To further reduce the risk of unwanted infections,the Linux system hardening tool Bastille、 Administrative tools、 Samhain.

Virtual Level

18

three roles were identified and included in the behavioural detection environment; a victim, an eavesdropper and an attacker.The victim is the host to be analyzed and where the object to be scrutinized should be installed.The eavesdropper should silently listen in to all traffic to be able to determine if any malicious or unwanted data is sent.The attacker should actively probe the victim for vulnerabilities by simulating attacker behaviour.

Experiments and Results(1/2)

19

Experiments and Results(2/2)

20

Discussion and Further Work

21

look at one example of stealth enabled software. It is therefore difficult to say something about the general detection capabilities of the testbed.None of the existing tools surveyed include aspects from all malware detection approaches.In the future, the quality of the testbed should be validated by running several tests with different categories of malware.

Conclusion

22

This paper has presented a testbed for behavioural analysis of MS Windows software where current malware trends and detection theories are considered.It is demonstrated that analysis environments can be designed without any license cost allowing organizations and students with limited resources to contribute to this important area of research.