Introduction to Software Security Malwaresecuresw.dankook.ac.kr/ISS18-1/ISS_2018_05_malware.pdf ·...

Seong-je Cho

Spring 2018

Computer Security & Operating Systems Lab, DKU

Introduction to Software Security

Malware(Chapter 11)

- 2 -

Sources / References

Textbook

N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Nicholas Weaver, Computer Science 161: Computer Security, Berkeley

Myrto Arapinis, Computer Security: INFRA10067, University of Edinburgh

Please do not duplicate and distribute

Computer Security & OS Lab, DKU

- 3 -

Contents

Malware

Malicious Software = Malicious Code

Types of Malware

Virus

Worm

Malware Detection

Signature/Change/Anomaly based Detection

New Trends of Malware

Do You Trust Software?

Smartphone app, Anti-malware software, DBMS, Compiler, OS/Kernel, Boot loader…


- 4 -

Malicious Software

Malware

any software program designed to cause harm to a computer system or designed to frustrate the user experience.

Fred Cohen’s initial virus work in 1980’s Used viruses to break the MLS (multi-level security) systems of the time

Types of malware (lots of overlap) Virus passive propagation

Worm active propagation

Trojan horse unexpected functionality

Trapdoor/backdoor unauthorized access

Rabbit exhaust system resources can implemented by virus, warm …


- 5 -

Malware Timeline

Preliminary work by Cohen (early 80’s)

Brain virus (1986)

Morris worm (1988)

Code Red worm (2001)

SQL Slammer worm (2004)

Future of malware?


- 6 -

Where do Viruses live?

Boot sector Take control before anything else

Memory resident Stays in memory – Rebooting system can remove the virus out

Applications, macros, data, etc.

Library routines

Compilers, debuggers, virus checker, etc. These are particularly nasty!


- 7 -

Brain Virus

First appeared in 1986 More annoying than harmful

Brain did nothing malicious Not much reaction by users

A prototype for later viruses

What it did1. Placed itself in boot sector (and other places)2. Screened disk calls to avoid detection3. Each disk read, checked boot sector to see if boot sector

infected; if not, goto 1

Screen: = check (적절한지)확인하다[거르다], 가려내다. I use my answerphone to screen my phone calls.


- 8 -

Morris Worm – 1/4

First appeared in 1988 What it tried to do

Determine where it could spread Spread its infection Remain undiscovered

Morris claimed it was a test gone bad “Flaw” in worm code it even re-infected already-infected

systems Led to resource exhaustion Adverse effect was like a so-called rabbit

The worm (consists of two parts)

a 99-line bootstrap program written in the C language, plus

a large relocatable object file that comes in VAX and Sun-3 flavors

Source: Seeley, Donn. "A Tour of the Worm.“ http://securitydigest.org/phage/resource/seely.pdf


http://securitydigest.org/phage/resource/seely.pdf

- 9 -

Morris Worm – 2/4

How to spread its infection?

Tried to obtain access to machine by

User account password guessing

Exploited buffer overflow in fingerd

Exploited trapdoor in sendmail

Flaws in fingerd and sendmail were well-known at the time, but not widely patched

① Once access had been obtained to machine

② “Bootstrap loader” sent to victim

Consisted of 99 lines of C code

③ Victim machine compiled and executed the code

④ Bootstrap loader then fetched the rest of the worm

⑤ Victim even authenticated the sender!


- 10 -

Morris Worm – 3/4

SW flaw --- in sendmail - in debug mode –

Normally sendmail runs in the background.

It receives a ‘send’ instruction along with dest address.

However in debug mode the worm can send a command string, in place of dest address.

Then this command string may be executed.

How to remain undetected?

If transmission of the worm was interrupted, all code was deleted

Code was encrypted when downloaded

Downloaded code was deleted after decrypting and compiling

When running, the worm regularly changed its name and process identifier (PID)


- 11 -

Result of Morris Worm – 4/4

Shocked the Internet community of 1988

Internet designed to withstand nuclear war Yet it was brought down by a graduate student!

At the time, Morris’ father worked at NSA…

Could have been much worse not malicious

CERT (Computer Emergency Response Team) began, The worm increased security awareness

Though limited actions to improve security


- 12 -

Code Red Worm

Appeared in July 2001

Infected more than 250,000 systems in about 10 ~ 15 hours

In total, infected 750,000 out of 6,000,000 susceptible systems

To gain access to a system, exploited buffer overflow in Microsoft IIS server software

Then monitored traffic on port 80 looking for other susceptible servers

What it did

Day 1 to 19 of month: tried to spread infection

Day 20 to 27: distributed denial of service (DDOS) attack on www.whitehouse.gov

Later versions (several variants)

Included trapdoor for remote access to infected system

Rebooted to flush worm, leaving only trapdoor

Has been claimed that Code Red may have been “beta test for information warfare”


- 13 -

SQL Slammer Worm

Infected 250,000 systems in 10 minutes!

Code Red took 15 hours to do what Slammer did in 10 minutes

At its peak, Slammer infections doubled every 8.5 seconds

Slammer spread too fast

“Burned out” available bandwidth


Why was Slammer so successful?

Worm fit in one 376 byte UDP packet

Firewalls often let small packet thru, assuming it could do no harm by itself

Then firewall monitors the connection

Expectation was that much more data would be required for an attack

Slammer defied assumptions of “experts”

Y 축: 트래픽, 초당 1백만패킷

- 14 -

Trojan Horse – 1/2

A Trojan has unexpected function

Prototype of Trojan for the Mac

Example

File icon for freeMusic.mp3:

For a real mp3, double click on icon

iTunes opens

Music in mp3 file plays

But for freeMusic.mp3, unexpected results…

Double click on freeMusic.mp3

iTunes opens (expected)

“Wild Laugh” (probably not expected)

Message box (unexpected)

A wolf in sheep’s clothing


- 15 -

Trojan Horse – 2/2

How does freeMusic.mp3 trojan work?

This “mp3” is an application, not data!

This Trojan is harmless,

but… Trojan could have done anything user can do delete files, download files, launch apps, etc.


- 16 -

Definitions of Terms in 2003

[source] Separating Trojan Horses, Viruses, and Worms - A Proposed Taxonomy of Software Weapons, Martin Karresand

Virus:

A self-replicating program.

Some definitions also add the constraint that it has to attach itself to a host program to be able to replicate

Worm:

Also a self-replication program, which does not need another program to be able to replicate but instead is a stand alone program.

The difference between a worm and a virus is often said to be the way they replicate, worms replicate over network connections, while viruses replicate on the host computer.

Trojan horse:

A program performing for the user unknown and unwanted actions, while at the same time posing as a legitimate program.


- 17 -

Definitions of Terms in 2003

[source] Separating Trojan Horses, Viruses, and Worms - A Proposed Taxonomy of Software Weapons, Martin Karresand


- 18 -

Other Types of Malware and Anti-malware

Rootkits

Ransomware

Keyloggers

What is anti-malware?

How does anti-malware work?

Why should we know types of malware?

Why should anti-virus program distinguish the unique types of malware from each other?

Why is malware family classification important?


Malware Detection

Anti-Virus SW (AV SW):

V3, ALYac, ViRobot, Kaspersky, Norton, …


- 20 -

Malware Detection

Three common methods

Signature detection 흔적탐지 (패턴매칭)

Change detection 변경탐지

Anomaly detection 비정상탐지 (이상탐지)

For signature detection & change detection, cryptographic graphic hash functions can be used

We’ll briefly discuss each of these

And consider advantages and disadvantages of each


- 21 -

Malware Signature

Brain Virus

The virus marks the six disk sectors as faulty, so that OS may not use them.

Signature: in 5th and 6th bytes of the file, it stores 1234 ( HEX ).

Action : with every disk read, it examines the file for its signature.

If it is not there, it infects the file.

Signature of another malware


- 22 -

Malware Signature

Example of simple code obfuscation


- 23 -

Signature Detection

A signature is a string of bits found in software (or could be a hash value)

Suppose that a virus has signature 0x23956a58bd910345

We can search for this signature in all files

If we find the signature, are we sure we’ve found the virus?

No, same signature could appear in other files

But at random, chance is very small: 1/264

Software is not random, so probability is higher

Advantages Effective on “traditional” malware

Minimal burden for users/administrators

Disadvantages Signature file can be large (10,000’s)…

…making scanning slow

Signature files must be kept up to date

Cannot detect unknown viruses

Cannot detect some new types of malware

By far the most popular detection method!


- 24 -

Change Detection

Viruses must live somewhere on system

If we detect that a file has changed, it may be infected

How to detect changes? Hash files and (securely) store hash values

Re-compute hashes and compare

If hash value changes, file might be infected

Advantages

Virtually no false negatives

Can even detect previously unknown malware

Disadvantages

Many files change and often

Many false alarms (false positives)

Heavy burden on users/administrators

If suspicious change detected, then what?

Might still need signature-based system


- 25 -

Anomaly Detection – 1/2

Monitor system for anything “unusual” or “virus-like” or potentially malicious

What is unusual?

Files change in some unusual way

System misbehaves in some way

Unusual network activity

Unusual file access, etc.

But must first define “normal”

And normal can change!


- 26 -

Anomaly Detection – 2/2

Advantages Chance of detecting unknown malware

Disadvantages Unproven in practice

Attacker can make anomaly look normal

Must be combined with another method (such as signature detection)

Also popular in intrusion detection (IDS)

A difficult unsolved (unsolvable?) problem! As difficult as AI?


- 27 -

Quiz

What are the similarities and differences between virus and worm?

What kinds of executable file formats are there?

MS Windows: ?, Linux: ?, Android: ?, iOS: ?

What is malware analysis?

What is static analysis technique for malware detection?

What is dynamic analysis technique for malware detection?

What are the main differences between static analysis and dynamic analysis?

How do you create a unique signature for malware detection?

Is it OK to run a malicious code on bare machine?

What are the common evasive techniques used by malware to avoid anti-malware detection?

What do you do to be a malware analyst?Computer Security & OS Lab, DKU

- 28 -

Malware Detection using Machine Learning

Machine Learning for Malware Detection, www.Kaspersky.com

"Learning and classification of malware behavior." International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2008.

"Analysis of machine learning techniques used in behavior-based malware detection." IEEE Second International Conference on Advances in Computing, Control and Telecommunication Technologies (ACT), 2010.

"Automated static code analysis for classifying android applications using machine learning." IEEE International Conference on Computational Intelligence and Security (CIS), 2010.

Automatic analysis of malware behavior using machine learning, Journal of Computer Security, 2011

"A machine learning approach to android malware detection." European Intelligence and security informatics conference (eisic), 2012.

Machine learning for Android malware detection using permission and API calls, IEEE 25th International Conference on Tools with Artificial Intelligence (ICTAI), 2013.

Evading Machine Learning Malware Detection, BlackHat USA, 2017


- 29 -

Machine Learning for Malware Detection (Kaspersky)


- 30 -

The Best Antivirus Protection of 2018- https://www.pcmag.com/article2/0,2817,2372364,00.asp


https://www.pcmag.com/article2/0,2817,2372364,00.asp

- 31 -

Detection Performance


Source: https://www.pcmag.com/article2/0,2817,2481367,00.asphttps://en.wikipedia.org/wiki/F1_score

https://www.pcmag.com/article2/0,2817,2481367,00.asp

https://en.wikipedia.org/wiki/F1_score

- 32 -

Open-Source Antivirus / Free Antivirus

ClamAV

OpenAV

Armadito AV


The Best Free Antivirus Protection of 2018 by PC magazine

VirusTotal Analyze suspicious files and URL to detect types of malware

https://www.virustotal.com/#/home/upload

YARA The pattern matching swiss knife for malware researchers

https://virustotal.github.io/yara/

YaraRules Project: http://yararules.com/

https://www.virustotal.com/#/home/upload

http://yararules.com/

New Trends of Malware(Malware with advanced technology)

You had better learn this part after encryption.

(Polymorphic malware is related to cryptography)


- 34 -

New Trends of Malware

Polymorphic 다형성 malware and

Metamorphic 변성 malware

The techniques of polymorphism and metamorphism change the form of each instance of software in order to evade “pattern matching” detection during the detection and investigative process

Fast replication/Warhol worms

Warhol worm could infect all vulnerable machines on the entire Internet in 15 minutes or less

The term is based on Andy Warhol's remark that "In the future, everyone will have 15 minutes of fame"

Flash worms, Slow worms, etc.

Future is bright for malware

Good news for the bad guys…

…bad news for the good guys

Future of malware detection? machine learning (?)


https://en.wikipedia.org/wiki/Andy_Warhol

https://en.wikipedia.org/wiki/Fifteen_minutes_of_fame

- 35 -

Polymorphic Malware

The first responses of virus writers to avoid signature detection

Polymorphism loosely means: “change the appearance of”

Polymorphic malware is (usually) encrypted Data appending / Data pre-pending can be used too

New key is used each time malware propagates

The purpose of encryption is for masking

The encryption is weak (repeated XOR)

Malware body has no fixed signature

Malware must include code to decrypt itself

Signature detection searches for decrypt code

Detectable by signature-based method

Though more challenging than non-polymorphic…

That is, there are limitations:

The decrypted code is essentially the same in each case, thus memory based signature detection is possible

Block hashing can be effective in identifying memory based remnants


- 36 -

Memory Block Hashing

Source: Chet Hosmer, Polymorphic & Metamorphic Malware, WetStone Technologies (Black Hat briefings and Training)


- 37 -

Memory Block Hashing


Memory code Snapshot

- 38 -

Metamorphic Malware

“automatically recodes itself each time it propagates or is distributed”

Simple techniques include:

Adding varying lengths of NOP instructions

Permuting use registers

Adding useless instructions and loops within the code segments

Advanced techniques include:

Function reordering

Program flow modification

Static data structure modification

Reordering structures

Inserting unused data types


- 39 -

Metamorphic Malware

Limitations

Identification of Morphing Engine

Code semantics

Behavior

Automated code identification and analysis of memory snapshots or analysis of swap space remnants


Morphing Engine ComponentsMetamorphic Structure

- 40 -

Cyber vs Biological diseases

One similarity

In nature, too few susceptible individuals and disease will die out

In the Internet, too few susceptible systems and worm might fail to take hold

One difference

In nature, diseases attack more-or-less at random

Cyber attackers select most “desirable” targets

Cyber attacks are more focused and damaging


- 41 -

Packing, Metamorphic Malware and Others …

Packing and Obfuscation

Packed and Obfuscated Malware

Malware writers often use packing or obfuscation to make their files

more difficult to detect or analyze

Please see Textbook, or

Malicious Code Analysis (악성코드분석) course will handle these topic in more detail


Miscellaneous Attacks

• Numerous attacks involve software

• Can you ever trust software?


- 43 -

Logic Bomb / Time Bomb

In 1986 Donald Gene Burleson told employer to stop withholding taxes from his paycheck

His company refused

He planned to sue his company He used company computer to prepare legal docs

Company found out and fired him

Burleson had been working on a malware…

After being fired, his software “time bomb” deleted important company data

Company was reluctant to pursue the case

So Burleson sued company for back pay!

Then company finally sued Burleson

In 1988 Burleson fined $11,800

Took years to prosecute

Cost thousands of dollars to prosecute

Resulted in a slap on the wrist

One of the first computer crime cases

Many cases since follow a similar pattern

Companies often reluctant to prosecute


http://www.internetwright.com/drp/RiskAssess.htm

- 44 -

Trusting Software

Can you ever trust software? See Reflections on Trusting Trust

Consider the following thought experiment

Suppose C compiler has a virus When compiling login program, virus creates backdoor (account with known

password)

When recompiling the C compiler, virus incorporates itself into new C compiler

Difficult to get rid of this virus!

Suppose you notice something is wrong

So you start over from scratch

First, you recompile the C compiler

Then you recompile the OS Including login program…

You have not gotten rid of the problem!

In the real world Attackers try to hide viruses in virus scanner

Imagine damage that would be done by attack on virus signature updates


- 45 -

Summary, Q & A

Malware

Malware family classification

Malware Detection

Signature/Anomaly-based Detection

Anti-malware software, Vaccine = Anti virus,

Most Common Evasive Techniques used by Malware

Antivirus evasion technique,

50 Open Source Tools to Replace Popular Security Software https://www.datamation.com/osrc/article.php/3882711/50-Open-Source-Tools-To-Replace-Popular-Security-Software.htm


• What types of threat are related to this lecture?

‒ Consider the STRIDE model !

https://www.datamation.com/osrc/article.php/3882711/50-Open-Source-Tools-To-Replace-Popular-Security-Software.htm

Introduction to Software Security Malwaresecuresw.dankook.ac.kr/ISS18-1/ISS_2018_05_malware.pdf ·...

Documents

Transcript of Introduction to Software Security Malwaresecuresw.dankook.ac.kr/ISS18-1/ISS_2018_05_malware.pdf ·...