Using Engine Signature to Detect Metamorphic Malware

Mohamed R. Chouchane and Arun Lakhotia

Software Research LaboratoryThe University of Louisiana at Lafayette

Fourth Workshop on Rapid Malcode (WORM)November 3rd, 2006

George Mason University, Fairfax, VA, USA

SCAM'06 29/28/2006

Metamorphic Malware

Virus

Form - C

M M

VirusVirus

Form - AForm - B

•Metamorphic malware changes as it propagates

•It creates multiple variants of itself

SCAM'06 39/28/2006

Metamorphic Malware Challenge

Signature

Virus

Form - C

M M

VirusVirus

Form - AForm - B

Too many signatures challenge the AV Scanner

Using different signatures for most variants cannot scale.

Antivirus scanners using extracted byte sequences, or “signatures” to identify known malware.

WORM'06 411/03/2006

Engine Signature: Track Variants to their Engine

One Engine Source of Variation Engine-friendly code is “Code written for the engine” Idea: Engine Signature vs. Virus Signature

Lightens burden of one signature per variant Analogous to determining likelihood of engine authorship

E-friendly malware

release

feedback

Engine variant

WORM'06 511/03/2006

Engine-Friendliness

10% friendly 20% friendly 90% friendly 100% friendly

Low E-friendliness

InputVariants

Output Variants

Metamorphic Engine Instruction Substitution

Garbage Insertion

WORM'06 611/03/2006

Code Substitution: Evol

mov [esi+4], 9 mov esi+4], 6 add esi+4], 3

mov [ebp+8], ecx push eax mov eax, ecx mov [ebp+8],

eax pop eax

push 4 mov eax, 4 push eax

push eax push eax mov eax, 2Bh

Clues

WORM'06 711/03/2006

Scoring Function

SE(V)=cs wc ecs / |V|

SE(V) measures how dense a code segment V is with clues from some code-substituting engine E.

Clues are weighted according to their length.

Can explore other weight assignments

Code Segment

Clue Count per Site

push 7mov 2sub 0 mov 0 pop 0mov 2add 0mov 2add 0 push 8 mov 2add 0mov 2add 0 pop 0

SE = 25/15 =1.667

WORM'06 811/03/2006

Evaluation: Non-Evol Segments

Frequency distributions of the scores of 2nd to 7th generationwith initial E-friendliness 5%(figure at left) and 50% (figure at right)

The E-friendlier the Eve, the higher the score

Later variants tend to score higher

“Convergence” behavior

WORM'06 911/03/2006

Evaluation: Simulated Evol Segments

Frequency distributions of the scores of 2nd to 4th generations (left to right) of simulated Evol

variants

Certain range of values

Gaussian Like

2nd, 3rd, and 4th gen variants scored 1.62, 1.95, and 2.13, respectively

WORM'06 1011/03/2006

Discussion Limitations

Small clues Less transformation options Low friendliness Malware open to traditional

signature scanning More analysis may be needed

Improvement and Further work Investigate other weight assignments Investigate engines which expand and shrink code Functional relationship among parameters Use engine signature to determine toolkit authorship

WORM'06 1111/03/2006

Software Research LabCenter for Advanced Computer StudiesUniversity of Louisiana at Lafayette

Arun LakhotiaDirector

Andrew WalensteinResearch Scientist

Michael VenableSoftware Engineer and Alumnus

Ph.D. StudentsMohamed R. ChouchaneMd Enamul Karim

M.S. StudentsMatthew HayesChristopher Thompson

Alumni Nitin Jyoti,

Avertlabs Aditya Kapoor,

McAfee Erik Uday Kumar,

Authentium Rachit Mathur,

McAfee Moinuddin Mohammed,

Microsoft Prashant Pathak,

Symantec Prabhat Singh,

Symantec

Funded by: Louisiana Governor’s IT Initiative

WORM'06 1211/03/2006

more at www.cacs.louisiana.edu/labs/SRL

“Using Engine Signature to Detect Metamorphic Malware”

Mohamed R. Chouchane and Arun LakhotiaSoftware Research Laboratory

The University of Louisiana at Lafayette

{mohamed,arun}@louisiana.edu