“You're like me, aren't you? I'd like to see how that works.”Sylar (Heroes)

For Master Safe 2012 (UjF) & Securimag

Introduction to Reverse Engineering

Karim Hossen

Summary

• Reverse engineering

• Definitions

• Tools

• PE format

• x86 Assembler

• Protections

• Anti-debugging

• Packer, encryption

• Obfuscation, VM

2

Definitions

• Reverse engineering (RE) is the process of discovering

the technological principles of a device, object, or system

through analysis of its structure, function, and operation.

to

3

Motivations

• Interoperability, lost documentation (Samba, RC4, doc)

• Product analysis, patent infringement

• Academic & learning (You)

• Security auditing

• Military or commercial espionage

• Curiosity

• Competition (CTF, SSTIC) (Leet)

• Competitive technical intelligence

• Don’t want to pay (Everybody )

4

Tools

• Two categories

• Static program analysis : performed without actually

executing programso Source code, object, structure, doc

• Dynamic program analysis : performed by

executing programs built from that software system

on a real or virtual processor.o Need test inputs, code coverage

• Dynamic > Static

but need more effort

5

Static analysis tools

• Disassembler

• IDA, Ndisasm, objdump …o Binary data -> instructions, function, data, variables …

• Decompiler

• Jad, HexRayso Binary data -> source code (not complete)

6

Dynamic analysis tools

• Debugger

• OllyDbg, GDBo Control of the execution

– Set breakpoint

– Modify data, flow, register

– Step by step execution

7

Portable Executable format

• Format for Windows executable

• Based of COFF

• Since Windows 3.1

• Portable means

portable on Windows …

• Some headers

• Then sections

8

MSDOS Header

• e_magic : MZ (Mark Zbikowski) :o

• e_lfanew : where is the PE header ?

9

PE Header

• Signature : “PE”

• Number of sections

• SizeOfOptionalHeader

10

Optional Header

• AddressOfEntryPoint

• BaseOfCode

11

PE Format summary

• Tools

• LordPE

• PeEditor

• PeID

• …

• Doc

• PE

cheatsheet

12

X86 ASM

• Stack

• Data structure that stores data values contiguously in

memory

• LIFO queue

• Grows downward, from high to low memory

addresses

• Main operationso Push, pop

o Sub, add

13

X86 ASM

• Stacko ?

14

X86 ASM

• Stacko ?

15

Calling convention

• CDECL

• Push parameters

from last to first

• Caller cleans

• That’s all for x86 !

• Doco Use the cheatsheet

o Instruction Set

Reference manual

from Intel website

16

Protections

• Antidebug

• Detecting the debuggero Process check

o CheckRemoteDebuggerPresent

o SetUnhandledExceptionFilter

o IsDebuggerPresent / BeingDebugged bit

o Self-debugging

o Timers

o Rogue Int3

• Using the debugger : OutputDebugString

• Using the loader : Corrupted header

• Using you : Obfuscation, Deadcode, Junkcode …

17

Protections

• Packer

• Code added to the program to protect it

• Hide important functions

• Can use encryption

• Specific device (USB dongle, card)

• Can need Internet connection (requests)

• For all : Aspack, Yoda

• For professional (all + $) : Themida, Starforce

18

Protections

• Packer

• Need to unpack it !

19

Protections

• Virtual Machine

• Virtualize the codeo Code can’t be disassembled easily

• Execute it on a virtual processoro Code can’t be debugged easily

• Need to :o Find how to decode each instruction

o Write the corresponding disassembler

o Reverse again

o Have time

• Ex :VMProtect used by SpyEye trojan

20

Conclusion

• It can be really hard to reverse a binary protected by a

professional packer

• These protections are expensive but efficient

• But the perfect protection doesn’t exists …

• In 2011, Reverse Engineers Dream Crew (RED) cracked

the hardware identification of SpyEye and release the

crack along the source code.

21

Webography

• http://www.hex-rays.com/products/ida/index.shtml

• http://www.intel.com

• http://www.woodmann.com/collaborative/tools/index.php

• http://www.openrce.org

http://www.symantec.com/connect/articles/windows-anti-debug-

reference

• http://touronster.blogspot.fr

• http://www.theinquirer.net/inquirer/news/2102503/spyeye-malware-

source-code-leaked

• http://xylibox.blogspot.fr

22