· PLEKHANOVA ПЛЕХАНОВА ... Kolesnikov V. KWPN ... ПАККАРД Basova O. Басова О. HANN Ганн. RUS
Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg...
-
date post
20-Dec-2015 -
Category
Documents
-
view
214 -
download
1
Transcript of Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg...
![Page 1: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/1.jpg)
Anomaly Detection Using Call Stack Information
Security Reading GroupJuly 2, 2004
Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong
Presenter: Jonathan McCune
![Page 2: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/2.jpg)
2
Overview
• Introduction• Related work• VtPath• Experiments• Exploits• Comparisons• Conclusion
![Page 3: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/3.jpg)
3
Introduction
• Runtime, training-phase based, anomaly detection system
• Uses call stack and program counter in addition to system call hooks
• Novel features based on virtual paths
• Compared experimentally and analytically with many other approaches
![Page 4: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/4.jpg)
4
Related work
• Static analysis– Wagner, et al. (abstract stack)– Callgraph
• Run-time, training based:– This– Sekar, et al. (FSA) (last week’s srg)– N-gram, Var-gram
![Page 5: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/5.jpg)
5
“Intrusion detection via static analysis.” Wagner, et al.
• Static analysis of program source code• NDFA from control-flow graph
– Cannot predict branches statically
• Impossible path problem• Abstract stack model
– Pushdown automaton
• Large automata can be resource intensive• Claimed zero false positives
![Page 6: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/6.jpg)
6
“Detecting Manipulated Remote Call Streams.” Giffin, et al.
• Static analysis of binary executables
• Tied to platform, not programming language
• Insertion of “null calls” helps impossible path problem
• Unusual performance analysis
![Page 7: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/7.jpg)
7
“A fast automaton-based method for detecting anomalous program
behavior.” Sekar, et al.
• Compact deterministic FSA from system call analysis of running programs
• Suffers from false positives, but not as a result of non-determinism
• Impossible paths not addressed
• DLLs not adequately addressed
![Page 8: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/8.jpg)
8
Virtual Path (VtPath)
• Run-time analysis• Learns correct behavior via training
executions • Utilizes return address information• Generates abstract path and compares
with learned behavior• Similar to abstract stack of Wagner, et
al., but avoid pushdown automaton
![Page 9: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/9.jpg)
9
Virtual Path Example
main()
foo1()
foo()
foo2()
foo3()
main()
fooA()
foo()
fooB()
fooC()
![Page 10: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/10.jpg)
10
VtPath Training Phase
• Two hash tables:– RA (return address) table– VP (virtual path) table
• RAs and VPs gradually added during normal program execution
• Use NULL entries at beginning and end of paths
![Page 11: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/11.jpg)
11
VtPath Online Detection Phase
• Stack anomaly – if virtual stack list unavailable (common during buffer overflow attacks)
• Return address anomaly – if virtual stack list {a0, a1, … an} contains ai not in RA table
• System call anomaly – if an does not have the correct system call
• Virtual path anomaly – if virtual path at current system call is not in VP table
![Page 12: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/12.jpg)
12
VtPath – Impossible Path Problem
• Return addresses are part of virtual path information
• Modifying return address saved on the stack will cause the program to return to a different location
• The next system call is likely to trigger a virtual path anomaly
![Page 13: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/13.jpg)
13
VtPath - Implementation Issues
• Non-standard control flows– Signals (sigreturn system call)
• Treat each one like a separate program invocation– setjmp()/longjmp()and function pointers
• Hard to handle statically; handled at runtime if trained properly
• Dynamically linked libraries– Relative loading positions can change, invalidating PC
values from training runs– Use a “block” model during training to capture file name and
block length, ignoring start address
• Block anomaly – block lookup fails because attacker is trying to load a malicious DLL
![Page 14: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/14.jpg)
14
Experiments – VtPath vs FSA
• Convergence times similar, but – FSA generates more transitions than VtPath (less
efficient, less precise)– In practice, multiple levels of DLL functions are
called quite frequently – VtPath more effective
• False Positives nearly identical• VtPath executes faster, uses more memory• Common exploits detected by both
![Page 15: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/15.jpg)
15
Exploits
• Authors developed two masked mimicry attacks detectable only by VtPath
• Impossible Path Execution (IPE) Attack 1– Exploits parallel structure of privileged /
unprivileged code within same function
• IPE Attack 2:– F() called twice from within same function with
different arguments– Overflow local variable to change option passed in
![Page 16: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/16.jpg)
16
Attack 1
![Page 17: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/17.jpg)
17
Attack 2
![Page 18: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/18.jpg)
18
Comparison of SysCall-based Anomaly Detection Schemes
1. State-based / Information captured• Data/heap values least useful (transient)• Code segment of some value• Syscalls and callstack most useful• More information = runtime overhead
2. False positives• How well is normal behavior captured?• Only relevant for dynamic systems• Proportional to resolution of program analysis
3. Detection capability• More granularity = better detection capability• SysCalls from invalid points vs. statistical regularity of
training data vs. attacks’ deviation from perceived normal
![Page 19: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/19.jpg)
19
Comparison of SysCall-based Anomaly Detection Schemes
4. Space requirement• System call sequences• Number of NDFA transitions (Wagner)• Transitions in automaton
5. Convergence (training) time• Cover most possible states / transitions• VtPath requires more data than FSA• Static techniques have big advantage here
6. Runtime overheads• SysCall interception• Hash lookup for valid state / return address / virtual path
![Page 20: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/20.jpg)
20
Conclusions
• Using call stack (VtPath) has value
• There is no magic bullet– Everything is “complementary”
• Impossible path execution (IPE) is an important class of attack
• Use of FSA to analyze more than two consecutive SysCalls could improve VtPath
![Page 21: Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter:](https://reader036.fdocuments.us/reader036/viewer/2022062714/56649d475503460f94a22b85/html5/thumbnails/21.jpg)
21
Discussion!