Eleos: Exit-Less OS Services for SGX Enclaves · 2017-05-18 · 22 May@Systor' 2017 Meni Orenbach,...

Eleos: Exit-Less OS Services for SGX Enclaves

Meni OrenbachMarina MinkinPavel Lifshits

Mark Silberstein

Accelerated Computing Systems Lab

Haifa, Israel

22 May@Systor' 2017 Meni Orenbach, Technion 2

What do we do?Improve performance: I/O intensive & memory demanding SGX enclaves

Why?Cost of SGX execution for these applications is high

How?In-enclave System Calls & User Managed Virtual Memory

ResultsEleos vs vanilla SGX

2x Throughput: memcached & face verification serversEven for 5x available enclave memory

Available for Linux, Windows*

(*) Without Eleos, these applications crash in Windows enclaves


● Background● Motivation● Overhead analysis● Eleos design● Evaluation


SGX enclaves are already here!

● Secured execution environment● Reversed sandbox● Small TCB● Private code & data

– Confidentiality

– Integrity

– Freshness

● Only CPU is trusted

Operating system

ApplicationEnclave Enclave




– Confidentiality

– Integrity

– Freshness


Operating system





– Confidentiality

– Integrity

– Freshness


Operating system


Lets look atHow to secure server applications with enclaves


Background: Lifetime of a secured server

Untrusted (Host & OS) Trusted (Enclave)


Untrusted memoryUnsecured access




Untrusted memoryUnsecured access



Dedicated SGX memLimited to: 128 MB

Secured access


Wait for networkrequests


Hostapp





Hostapp





Decrypt requests

Enter enclave

Hostapp





Decrypt requests

Enter enclave

Process requests

Hostapp





Decrypt requests

Enter enclave

Process requests

Hostapp

Encrypt responses



Send responses



Decrypt requests

Enter enclave

Process requests

Exit enclave

Hostapp

Encrypt responses



SGX enclaves should be fast

● ISA extensions● Implemented in HW & Firmware● Same CPU HW● In-cache execution suffers no overheads


SGX enclaves should be fast

● ISA extensions● Implemented in HW & Firmware● Same CPU HW● In-cache execution suffers no overheads

However...


Executing a Key-Value Store in enclave is slower


64 MB 512 MB0

5

10

15

20

25

30

35

40

Memory footprint


Throughput: Slowdown factor

11X

34X


64 MB 512 MB0

5

10

15

20

25

30

35

40

Memory footprint



11X

34X

Crashesin Windows


Send responses


Overhead analysis

Enter enclave

Exit enclave

Hostapp


Decrypt requests 150 cycles/32B

Process requests *100 cycles/32B

Encrypt responses *150 cycles/32B


Overhead analysis

Enter enclave

Exit enclave

Hostapp


Send responses


Enter enclave

Exit enclave

Hostapp Decrypt requests

150 cycles/32B



~3,300cycles


Overhead analysis

Enter enclave

Exit enclave

Hostapp


Send responses


Enter enclave

Exit enclave


150 cycles/32B

Process requests*100 cycles/32B

Encrypt responses*150 cycles/32B

~3,300cycles

~3,800cycles


Overhead analysis

Enter enclave

Exit enclave

Hostapp


Send responses


Enter enclave

Exit enclave


150 cycles/32B



~3,300cycles

~3,800cycles

Exits causes indirect costs:1.5X – 5X slower execution

FlexSC [OSDI'10] syscall analysis


Eleos does better!

64 MB 512 MB0

5

10

15

20

25

30

35

40SGX Eleos

Memory footprint

3.5x

5x



Eleos does better!

64 MB 512 MB0

5

10

15

20

25

30

35

40SGX Eleos

Memory footprint

3.5x

5x

How does Eleos achieve this?



Eleos: Exit-less services

Exit-less system calls with RPC infrastructure

Exit-less SGX paging


Background: SGX paging

System mem

SGX mem

Dedicated memoryEnclave code & data

Limited to 128 MB



System memsecret_foo():...*p = 1; SGX mem

EnclaveTrusted

Untrusted



System memsecret_foo():...*p = 1; SGX mem

HardwareAddress translation

EnclaveTrusted

Untrusted



System memsecret_foo():...*p = 1;

Encrypted

SGX mem

Page table


EnclaveTrusted

Untrusted



System memsecret_foo():...*p = 1;

Encrypted

SGX mem

Page table


Swapped-out

EnclaveTrusted

Untrusted



System mem

Faulthandler

secret_foo():...*p = 1;

Encrypted

SGX mem

Page table


Swapped-out

EnclaveTrusted

UntrustedSGX-driver



System mem

Faulthandler

secret_foo():...*p = 1;

Encrypted

Integrityvalidation

Decrypted

SGX mem

Page table


Swapped-out

EnclaveTrusted

UntrustedSGX-driver



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted

Fast path



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted

Since SGX memory is smallpaging is not as rare as in native applications

What are the overheads?

Fast path



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted


SGX paging overheads

System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted

Enclaveresume

Enclaveexit



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted

Enclaveresume

Enclaveexit

Indirect costs



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted

Enclaveresume

Enclaveexit

Overaheads: Untrusted softwaremanages enclave memory

Indirect costs


Wanted: In-enclave virtual memory management

No more exits!


Ideal in-enclave VM management

System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

SGX mem

Page table


EnclaveTrusted

SGX driverUntrusted




System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

SGX mem

Page table


EnclaveTrusted




System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

SGX mem

Page table


EnclaveTrusted


No availablehardware



System mem

Faulthandler

secret_foo():...*p = 1;*(++p) = 2;

SGX mem

Page table


EnclaveTrusted

SoftwareAddress translation


SUVM: Secured user-space VM

System mem

Faulthandler

secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1; SGX mem

Page table

EnclaveTrusted




System mem

Faulthandler

secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1; SGX mem

Page table

EnclaveTrusted


Template class: SecuredPointer.



System mem

Faulthandler

secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;

Encrypted

SGX mem

Page table

EnclaveTrusted





System mem

Faulthandler


Encrypted

SGX mem

Page table

EnclaveTrusted



Swapped-out



System mem

Faulthandler


Encrypted

Decrypted

SGX mem

Page table

EnclaveTrusted



Swapped-out

Integrityvalidation



System mem

Faulthandler


Encrypted

Decrypted

SGX mem

Page table

EnclaveTrusted



Swapped-out

Integrityvalidation

Control pathin-enclave



System mem

Faulthandler

secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;*(++p) = 2;

Encrypted

Decrypted

SGX mem

Page table

EnclaveTrusted




System mem

Faulthandler


Encrypted

Decrypted

SGX mem

Page table

EnclaveTrusted




System mem

Faulthandler


Encrypted

Decrypted

SGX mem

Page table

EnclaveTrusted


Fast pathNo page table

Lookup!


Wait...Software based VM management?

Based on software address translation on GPUs, ActivePointers [ISCA'2016]


SUVM key contributions

● Multi-threaded

Compared to SGX:

Fast path: up to 20% overheads

Slow path: Eliminates costs of exits

1 Thread 4 ThreadsREAD 5.5x 7xWRITE 3.5x 5.9x

Throughput speedup


Software address translation offers new optimizations

● Customized page size● Customized eviction policy● Multi-enclave memory coordination● Write-back only dirty pages● Sub-page direct access to backing store




Virtual Machineballooning


Biometric Identity checking server

Face verificationserver

Workloadgenerator

?=

450MB DB(5X SGX mem)

+ ID

10Gb NIC


1 2 40

0.5

1

1.5

2

2.5

3

3.5

Eleos Native

Server threads

Biometric Identity validating serverSpeedup compared to vanilla SGX


1 2 40

0.5

1

1.5

2

2.5

3

3.5

Eleos Native

Server threads



1 2 40

0.5

1

1.5

2

2.5

3

3.5

Eleos Native

Server threads


Eleos scales better than vanilla-SGX:Saves inter-processor-interrupts


1 2 40

0.5

1

1.5

2

2.5

3

3.5

Eleos Native

Server threads


Eleos scales better than vanilla-SGX:Saves inter-processor-interrupts

Saturate 10Gb network


Memcached

WorkloadGenerator(memaslap)

GET( )

~75 LOC modificationfor SUVM

MemcachedGraphene LibOS [Eurosys'2014]

500MB DB(5.5X SGX mem)

10Gb NIC


1 Thread 4 Threads0

0.5

1

1.5

2

2.5

3

Eleos (500MB DB) vanilla SGX (20MB DB)

Server threads

MemcachedSpeedup compared to vanilla SGX (500 MB)

No SGX Faults

No SGX Faults


1 Thread 4 Threads0

0.5

1

1.5

2

2.5

3

Eleos (500MB DB) vanilla SGX (20MB DB)

Server threads

MemcachedSpeedup compared to vanilla SGX (500 MB)

Disclaimer: Eleos+Graphene is 3x slower than native

No SGX Faults

No SGX Faults


Take aways

● Eleos eliminates enclave exits costs● Eleos available for Windows and Linux

– Makes memory demanding applications available on Windows today

● Eleos takes a modularize approach– Memory demanding app? Link to SUVM

– I/O intensive app? Link to RPC

– Maintaining small TCB


Traditional SGX:Host-centric OS services

Enclave

Operating System



Enclave

Operating System

Getdata



Enclave

Operating System

Getdata

DataUnavailable



Enclave

Operating System

Fetch data

Getdata

DataUnavailable


Eleos Insight:Enclave-centric OS services

Enclave

Getdata

Fetch data

In-enclaveServices


Take aways (2)

● Eleos adapts 'accelerator-centric management'– System calls: GPUfs [ASPLOS'13], GPUnet [OSDI'14]

– Virtual memory: ActivePointers [ISCA'16]

● We can do more!– Asynchronous DMA host copies

– Non-blocking enclave launches

More information at:

“SGX Enclaves as Accelerators" [Systex'16]


Thank you

Code is available at:https://github.com/acsl-technion/eleos

[email protected]

https://github.com/acsl-technion/eleos


Backup slides

Eleos: Exit-Less OS Services for SGX Enclaves · 2017-05-18 · 22 May@Systor' 2017 Meni Orenbach,...

Documents

Transcript of Eleos: Exit-Less OS Services for SGX Enclaves · 2017-05-18 · 22 May@Systor' 2017 Meni Orenbach,...