Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti...
-
Upload
belinda-miller -
Category
Documents
-
view
217 -
download
3
Transcript of Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti...
![Page 1: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/1.jpg)
1
Effective and Efficient Malware Detection at the End Host
Clemens Kolbitsch, Paolo Milani Comparetti @ TU ViennaChristopher Kruegel @ UCSB
Engin Kirda @ Institute EurecomXiaoyong Zhou, XiaoFeng Wang @ Indiana Univ. at Bloominton
USENIX Security Symposium ‘09
![Page 2: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/2.jpg)
2
Outline
• Motivation• System Overview• System Details• Evaluation• Limitation• Conclution
![Page 3: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/3.jpg)
3
MOTIVATIONEffectiveness & Efficiency
![Page 4: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/4.jpg)
4
Motivation
• Efficiency– Binary signature based detection– Network-based detection
• Effectiveness– Behavior-based detection• Detection based on malware's behavior• Behavior is hard to obfuscate• Behavior is hard to randomize• Behavior is often stable across various malware version
![Page 5: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/5.jpg)
5
Motivation
• This Paper proposes…– A behavior-based solution with Efficiency– For end hosts
![Page 6: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/6.jpg)
6
SYSTEM OVERVIEWModeling Behaviors and Making detection efficient
![Page 7: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/7.jpg)
7
System Overview
• Malware behaviors– Manifest on system (i.e., survive reboot)
• (Over-) write system executables, dlls, files• Create registry entries• Register as Windows (startup) service
– Conceal from being detected• Restart under some stealthy name (e.g., svchost.exe)• Inject into legitimate processes
– Replicate• Send emails• Copy to Samba shares, USB drives, etc.• Scan and exploit services on LAN or WAN
![Page 8: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/8.jpg)
8
System Overview
• Detection based on execution characteristics– Execute malware in full system emulator (Anubis)– Monitor interaction with the operating system– Perform detailed taint analysis– Generate detection graphs
• Describe sequence of required system calls leading to security relevant system activity
• Include dependencies to related, previous calls (using taint dependencies)
• Detect described behavior on end host– Log system call activity of unknown executable– Match against behavior graph
![Page 9: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/9.jpg)
9
System Overview
• Example: Agent (trojan)• As part of its system manifestation, it– Reads content from binary image– Decrypts binary content
• Proprietary decryption routine• Simple, XOR based algorithm
– Stores binary in system file (C:\Windows\system32\drivers\ip6fw.sys)
– Later, restarts IPv6 firewall• Turns itself into a system service
![Page 10: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/10.jpg)
10
System Overview
![Page 11: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/11.jpg)
11
SYSTEM DETAILS
Generate Behavior Graphs,Match Behavior Graphs
![Page 12: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/12.jpg)
12
System Details
• Behavior graphs– Directed acyclic graph– Node: system calls– Edges: dependencies
• Dependencies– Handle dependencies• Direct value propagation• System provided identifiers• Must be constant
![Page 13: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/13.jpg)
13
System Details
• Data dependencies– Arbitrary data (& control) dependency between
system calls– Might modify values between system calls
![Page 14: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/14.jpg)
14
System Details
• Generate behavior graphs– Analyze executable in Anubis sandbox• Obtain instruction level log• Obtain program flow log• Obtain memory access log• Generate precise taint propagation trees
– Data/control dependencies– Instructions that access/generate tainted data– Link system calls consuming data with all taint generating calls
(sources)
![Page 15: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/15.jpg)
15
System Details
• Generate behavior graphs (cont.)– Scan logs for security relevant behavior• Provided with a list of interesting system calls
• Extract propagation formulas
![Page 16: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/16.jpg)
16
System Details
• Match behavior graphs– Active(inactive) node– Simple(complex) function– Security-relevant system calls or the Buttom– Confirmed(deactivate all)
![Page 17: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/17.jpg)
17
System Details
![Page 18: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/18.jpg)
18
System Details
![Page 19: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/19.jpg)
19
EVALUATION
Effectiveness,Efficiency
![Page 20: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/20.jpg)
20
Evaluation
• Effectiveness
![Page 21: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/21.jpg)
21
Evaluation
![Page 22: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/22.jpg)
22
Evaluation
• False Positive– IE, Firefox, Thunderbird, putty, notepad– 0
![Page 23: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/23.jpg)
23
Evalution
• Efficiency
![Page 24: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/24.jpg)
24
LIMITATION & CONCLUSION
![Page 25: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/25.jpg)
25
Limitation
• Evading signature generation– Detect the virtual environment– Delays, time-triggered behavior
• Modifying the algorithm behavior
![Page 26: Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649d145503460f949e86d9/html5/thumbnails/26.jpg)
26
Conclusion
• Behavior can be detected• Behavior detection is fast enough for end
hosts– Approach intrinsically robust against
polymorphism and metamorphism– To some extent, behavior graphs are usable across
malware variants