Slide used at ACM-SAC 2014 by Suzaki

National Institute of Advanced Industrial Science and Technology

Rollback MechanismRollback Mechanism of Nested Virtual Machines for Protocol Fuzz Testing

Kuniyasu Suzaki*, Toshiki Yagi*, Akira Tanaka*, Y taka Oi a* Ets a Shiba ama*‡Yutaka Oiwa , Etsuya Shibayama ‡

* National Institute of Advanced Industrial Science and Technology(AIST) National Institute of Advanced Industrial Science and Technology(AIST) ‡The University of Tokyo

ACM SAC(Symposium On Applied Computing)2014 at Gyeongju, South Korea March 25


Motivation 1/2Motivation 1/2• Sec re protocols (SSL/TLS SSH etc) are important for• Secure protocols (SSL/TLS, SSH, etc) are important for

Internet commerce.l bili i f i l i f• However, vulnerabilities of implementation of secure

protocols are discovered repeatedly. – Incidents of SSL/TLS vulnerabilities

• Insufficient entropy of random numbers [2012] – It creates same secret-keys for SSL/TLS certificates.It creates same secret keys for SSL/TLS certificates.– EFF offers a site to check SSL certificates.

» https://www.eff.org/observatory

• CRIME[2012]• CRIME[2012]– It attacks to the data compression used by the TLS and SPDY.

• Lucky Thirteen[2013]– It attacks padding oracle using timing side-channel.


Motivation 2/2Motivation 2/2• The implementations of secure protocols should be verified.

• Our group increases TLS security with 2 approaches.F l ifi i– Formal verification

• Verify important parts of protocol hander program.– “Formal Network Packet Processing with Minimal Fuss: Invertible Syntax Descriptions atFormal Network Packet Processing with Minimal Fuss: Invertible Syntax Descriptions at

Work.” [PLPV’12]– “Towards Formal Verification of TLS Network Packet Processing Written in C”

[PLPV’13]

– Exhaustive Protocol Fuzz Testing• Try to check all reaction of protocol handler.

– Protocol Test Generator creates and sends many fuzzed packets to check implementation of protocol handler.

– (Today’s Topic) Computing environment for protocol fuzz testing


O liOutlineM i i• Motivation

• Protocol Test Generator: fuzz testing on a protocol handler• Why we use nested VM for protocol fuzz testing?• Design issuesDesign issues• Implementation

P f l i• Performance evaluation on current prototype.• Conclusion


Protocol Test GeneratorV if t l h d f h t l t t• Verify a protocol hander for each protocol state– Repeat the following sequence for each protocol state

1. Creates a fuzzed packet2. Send it to a protocol hander 3. Check the reply

• Fuzzed packets are tested on each protocol stateFuzzed packets are tested on each protocol state many times.

• Snapshot and rollback mechanism is required.S p q


Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)Normal

TargetProtocol Generator

Client Hello

Take Snapshot

Client Hello

Server Hello

Server Certificate

Snapshot1

Server Hello Done

Take SnapshotTake Snapshot

Client Key Exchange… Snapshot 2


Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)Normal Fuzz for Client Hello

TargetProtocol Generator TargetProtocol Generator

Client Hello

Take Snapshot

[Fuzz1] Client Hello

Rollback Snapshot1

Client Hello

Server Hello

Server Certificate

Snapshot1 [Fuzz1] Client Hello

????

Good or Bad?

Rollback1

Server Hello Done

Take Snapshot Rollback Snapshot1

Good or Bad?

Take Snapshot

Client Key Exchange… Snapshot 2


????Rollback 1

Good or Bad?

[FuzzN] Client Hello


Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)Normal Fuzz for Client Hello Fuzz for Client Key Exchange

TargetProtocol Generator TargetProtocol Generator TargetProtocol Generator

Client Hello

Take Snapshot


Rollback Snapshot1[Fuzz1] Client Key

Rollback Snapshot2

Client Hello

Server Hello

Server Certificate


????

Good or Bad?

Snapshot1 Rollback1

[Fuzz1] Client Key Exchange

???

Rollback2

Server Hello Done

Take Snapshot Rollback Snapshot1

Good or Bad?

Rollback Snapshot2

Good or Bad?

Take Snapshot

Client Key Exchange…


????Snapshot 2 Rollback 1 Rollback 2

[Fuzz2] Client Key Exchange

???

[FuzzN] Client Hello

Good or Bad????

Good or Bad?[FuzzN] Client Key Exchange


Requirement for Protocol Test GeneratorRequirement for Protocol Test Generator1. Packet level granularity control

– Generator wants to take a snapshot just before sending a packet.

2. Hardware-level repeatabilityB l biliti d b it ti i t (– Because some vulnerabilities are caused by its computing environment (e.g., random number)

3. Consistency between generator and target protocol hander3. Consistency between generator and target protocol hander– Most snapshot/rollback mechanism does not care of packets on a wire.

1. We propose a special protocol to control target computing environment.

2. We uses Virtual Machine which offers hardware-level repeatability f h / llb kfor snapshot/rollback.

3. Design issues. (Mention in later slides)


Proposed Protocol (VTP: Virtual Test Protocol)Proposed Protocol (VTP: Virtual Test Protocol)• VTP treats packets of test-target protocol as data.

T t t t k t l t d t “C l ” k t– Test-target packets are en-capsulated to “Capsule” packets– “Capsule” packets must be de-capsulated on target computing environment.

• VTP has special packets to control target computing environment• VTP has special packets to control target computing environment– “Take_Snapshot” and “Rollback”

Protocol Action

CapsulePackets of secure communication are encapsulated with capsule protocol. The flow of capsulated packets is managed

h t k h t llb k k t i i dwhen take_snapshot or rollback packet is issued. Take_ Snapshot

Take VM snapshot image. Packets on the wire must be managed by a certain method. Return snapshot ID.R h i C i fRollback

(with ID)

Resume snapshot image. Connection of secure communication must be operative, when a previous image is resumed.

Target computing environment (VM) has to deal with the VTP.


Design Issues for VM (Computing environment)Design Issues for VM (Computing environment)

• Virtual Machine runs a protocol handler.• En-capsulated packets by VTP must be de-capsulated on a target VM.

– Original packets must be passed to a target protocol handler on the VM.

• The control packets (take_snapshot and rollback) must be dealt by the VM.• The VTP packets on the network must be maintained.

– We have to deal with consistent global state defied by [Chandy-Lamport, 1985].

ProtocolGenerator

Target EnvironmentSnapshots of VM

VTP

Protocol handler

TLS

VTP

VM deals with VTP protocol. Packets on the network must be maintained.

VM


Design choice for handling VTPDesign choice for handling VTP • Customize VM’s network function for VTPCustomize VM s network function for VTP

– We planned to customize SLiRP of QEMU• However VM control (snapshot/rollback) from SLiRP is strange, because it ( p ) g ,

is a part of VM. It is not flexible.

• Nested VM and Proxies (external & internal)– External proxy manages VTP control packets and takes snapshot of

External VM. – A snapshot of External VM keeps connectivity of network between

internal proxy and a protocol handler on Internal VM.• Pros: Easy and flexible implementation.

» It utilizes existing VM control commands. Don’t need to change VM.• Cons: Overhead? Performance is improved by many previous researchesCons: Overhead? Performance is improved by many previous researches.


Design: Nested VM and Proxiesg• External VM (a unit of snapshot)

– It is used to take snapshot of Internal VM and Internal Proxy.• Internal VM (hardware repeatability)( p y)

– It includes protocol hander and keeps connection with Internal Proxy, when External VM is taken a snapshot.• External Proxy

– Connects to Protocol Generator directly, and controls snapshot/rollback of External VM.

Real Machine External VMControl the target VM

• Internal Proxy– Maintain connectivity of VTP Protocol when a snapshot image is resumed.

Real Machine External VM

Internal VM

Control the target VM(SAVEVM, LOADVM)

Decapsulatecontrol commands

InternalＰｒｏｘｙ openSSL

Port forward

ExternalＰｒｏｘｙExternalＰｒｏｘｙ

InternalＰｒｏｘｙTLS

VTP

TLSTLS

VTP

External VM

Internal VM

External VM

Internal VM

ProtocolGenerator “take_snapshot” packet make a

round‐trip between external and internal proxy to confirm no VTP packets on the wire

13

Internal VMSSH

openSSLKeep connectionSnapshots

no VTP packets on the wire.

RollbackTLS


Handling VTP “capsule” packetHandling VTP capsule packet• Protocol Generator en-casuplates test-target packets with “capsule” packets. • External Proxy passes capsule packets to Internal Proxy without change.• Internal Proxy de-capsulates VTP packets and passes original packets to protocol

hander on Internal VMhander on Internal VM.

Real Machine External VMReal Machine External VM

Internal VMDecapsulatecontrol commands


Port forward


InternalＰｒｏｘｙTLS

VTP

TLSTLS

VTP

ProtocolGenerator


Handling VTP “take snapshot” packetHandling VTP take_snapshot packet• When a “take_snapshot” packet is issued, the packet makes a round trip between

External and Internal Proxy to confirm that there is no VTP packet on the wire (consistent global state defied by Chandy Lamport)(consistent global state defied by Chandy-Lamport).

• After that, External Proxy sends SAVEM command to External VM to take a snapshot.

• Connection between Internal Proxy and protocol handler is kept in a snapshot.

Real Machine External VMControl the target VMReal Machine External VM

Internal VM

g(SAVEVM)



Port forward


InternalＰｒｏｘｙ

P t l

TLS

VTP

TLSTLS

VTP

External VM

Internal VM

External VM

Internal VM

ProtocolGenerator “take_snapshot” packet make a

round‐trip between external and internal proxy to confirm no VTP packets on the wire.

15

Internal VMSSH

openSSLKeep connectionSnapshots

no VTP packets on the wire.TLS


Handling VTP “rollback” packetHandling VTP rollback packet• When a “rollback” packet is issued, the External Proxy send

LOADVM command to External VM It resumes an old imageLOADVM command to External VM. It resumes an old image.• After that, External and Internal proxies re-establish the connection

between them, because they loses the network states when rollback. , y– Internal proxy and protocol handler do not need to care about network connection.

Real Machine External VMControl the target VMReal Machine External VM

Internal VM

g(LOADVM)



Port forward


InternalＰｒｏｘｙ

P t l

TLS

VTP

TLSReestablish

External VM

Internal VM

External VM

Internal VM

ProtocolGenerator

16

Internal VMSSH

openSSLKeep connection Rollback

TLS


OptimizationOptimization• Most overhead of VM is caused by video emulation.

– Nested VMs must emulate the GUI of internal VM on external VM again.g

• Fortunately QEMU and KVM have “curses mode” which emulates text user interface.

• We eliminate GUI on both internal and external VMs.N GUI d d k h t i k– No GUI reduces memory usage and make snapshot quick.

GUIs of Nested VMs No GUI of Nested VMs

OpenSSL on Linux onInternal VM (QEMU)

on External VM (KVM)ith d

Internal VM (QEMU)

OpenSSL on Windows

Extenal Proxy

Protocol Tester

with curses mode

Protocol Tester

External Proxy

External VM (KVM)Internal Proxy on

Linux


Current ImplementationCurrent Implementation• We implement Nested VM and proxies on normalWe implement Nested VM and proxies on normal

Debian GNU Linux x86. – External VM: KVMExternal VM: KVM

• The VM is a unit of snapshot and is NOT required hardware repeatability.

• KVM is fast with virtualization technology of X86.

– Internal VM: QEMU h l h d f bili ( d b• The VM emulates hardware for repeatability (e.g., random number

generator).

– Proxies are written by PerlProxies are written by Perl.• External Proxy: 430 LOC • Internal Proxy: 132 LOC• Internal Proxy: 132 LOC


Performance evaluationPerformance evaluation • We evaluated performance on Protocol Test Generator.

– Test for hand shake of TLS (PolarSSL) takes 9 snapshots and 2,311 rollbacks.– Effect of GUI and noGUI.– Effct of Memory size (large 1024MB/512MB small 512MB/256MB)– Effct of Memory size (large 1024MB/512MB, small 512MB/256MB)

GUIMem:

NoGUIMem:

NoGUIMem:

1024/512(sec)

1024/512(sec)

512/256(sec)

Setting up nested VMs 266 107 93g pFuzz packet generator 1,307 1,164 1,080

Nested VMs and Proxies

Snapshot(9 times)

57(Av 6.33)

37(Av 4.11)

24(Av 2.67)

R llb k 3 135 2 197 1 286 VM overhead isRollback(2,311times)

3,135(Av 1.36)

2,197(Av 0.96)

1,286(Av 0.56)

Other 12 13 12Total 5 043 3 622 2 587

VM overhead is almost same to packet generator.

Total 5,043 3,622 2,587

ThinkPad T410 (CPU Intel Core i7-M620 2.67Ghz, Memory 4GB).


R lt f P t l F T tiResults of Protocol Fuzz Testing W ifi d th h d h k f 4 TLS• We verified the handshake of 4 TLS servers.– OpenSSL, GnuTLS, CyaSSL, and PolarSSL.

• The fuzz testing found 2 bugs in CyaSSL– 2 bugs in CyaSSL

– 1 bug in PolarSSL• We reported the results to the mailing list, and the

bugs were fixed.g


R l t d kRelated works• Nested VM• Nested VM

– Migration• Xen-Blanket [EuroSys’12] enables user level migration on CloudXen Blanket [EuroSys 12] enables user level migration on Cloud.

– Security• CloudVisor [SOSP’11] is inserted on existing VM on multi-tenant

Cloud.

• Process level snapshot– DMTCP: Distributed Multi Threaded Check Pointing

• http://dmtcp.sourceforge.net/If d ’t t d t b t h d l l t bilit f• If we don’t not need to care about hardware-level repeatability for rollback, DMTCP will be a candidate for protocol test generator.


Conclusion• We propose a combination of nested VM and proxies

for protocol fuzz testing environment.– It treats hardware-level repeatability (e.g., random number).– It is easy and flexible to implement.

• Our method is not limited to protocol testing. It can be applied on other areas (e.g., I/O fuzz testing, memory pp ( g , g, yfuzz testing).

AcknowledgementThis work is supported by the National Institute of Information and Communications Technology of Japan.

Slide used at ACM-SAC 2014 by Suzaki

Technology

Transcript of Slide used at ACM-SAC 2014 by Suzaki