Post on 26-Aug-2020
Achieving Keyless CDNswith Conclaves
Stephen Herwig
Dave LevinChristina Garman
Measurement and Analysis of Private Key Sharing in
the HTTPS Ecosystem
Frank Cangialosi, Taejoong Chung, David Choffnes, Dave Levin, Bruce M. Maggs, Alan Mislove, Christo Wilson
User Bank
customer’s origin server
Content Delivery Networks host their customers’ websites
CDNs
customer’s origin server
CDN’sedge server
Content Delivery Networks host their customers’ websites
CDNs CDNs reduce page load times
CDN’sedge server
customer’s origin server
CDNs CDNs reduce page load times
CDN’sedge server
customer’s origin server
CDNsCDNs mitigate and block attacks
CDN’sedge server
customer’s origin server
CDNsCDNs mitigate and block attacks
CDN’sedge server
customer’s origin server
Customers share their keys with CDNs
CDN’sedge server
bank’s private key
Customers share their keys with CDNs
CDN’sedge server
Key sharing is widespread
Cangialosi et al., CCS 2016
Key sharing is widespread
Who shares?
0
0.2
0.4
0.6
0.8
1
0 200k 400k 600k 800k 1M
Fra
cti
on
of
Do
main
s H
oste
do
n T
hir
d-p
art
y P
rovid
ers
Alexa Site Rank (bins of 10,000)
At least one key sharedAll keys shared
43% of the top 10k most popular websites
Cangialosi et al., CCS 2016
Key sharing is widespread
Who shares?
0
0.2
0.4
0.6
0.8
1
0 200k 400k 600k 800k 1M
Fra
cti
on
of
Do
main
s H
oste
do
n T
hir
d-p
art
y P
rovid
ers
Alexa Site Rank (bins of 10,000)
At least one key sharedAll keys shared
43% of the top 10k most popular websites
Cangialosi et al., CCS 2016
The web has consolidated keys in the hands of a few CDNs
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
Key server performs actions requiring private key
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
The CDN learns all session keys
Key server performs actions requiring private key
Keyless SSL
Introduced by Cloudflare to mitigate key sharing
Private keys stay at the key server (origin)
The CDN learns all session keys
In practice: CDN
Third-party resources on
using Legacy applications
Can we Maintain privacy
?
Third-party resources
Legacy applications
Maintain privacy The CDN is no more trusted than a standard on-path attacker
Third-party resources
Legacy applications
Maintain privacy The CDN is no more trusted than a standard on-path attacker
No changes to existing code-bases; facilitates deployment and adoption
Third-party resources
Legacy applications
Maintain privacy The CDN is no more trusted than a standard on-path attacker
Leverage the existing infrastructure.One additional assumption: TEEs
No changes to existing code-bases; facilitates deployment and adoption
Third-party resources
Legacy applications
Maintain privacy The CDN is no more trusted than a standard on-path attacker
Leverage the existing infrastructure.One additional assumption: TEEs
No changes to existing code-bases; facilitates deployment and adoption
Third-party resources
Legacy applications
Maintain privacy The CDN is no more trusted than a standard on-path attacker
Leverage the existing infrastructure.One additional assumption: TEEs
No changes to existing code-bases; facilitates deployment and adoption
Phoenix
Trusted execution environments
Hardware
OperatingSystem
Application
By default, assume all system components are untrusted
Code
Service
Trusted execution environments
Hardware
OperatingSystem
Application
By default, assume all system components are untrusted
Small trusted CPU Resistant to physical attacks
Code
Service
Trusted execution environments
Hardware
OperatingSystem
Application
By default, assume all system components are untrusted
Small trusted CPU Resistant to physical attacks
Code
Service
Enclave: Isolated application memory
Enclave
Trusted execution environments
Hardware
OperatingSystem
Application
By default, assume all system components are untrusted
Small trusted CPU Resistant to physical attacks
Code
Service
Model: Code and data can safely reside inside an enclave
Enclave: Isolated application memory
Enclave
Hardware
OperatingSystem
Application Code
Service
Practical limitations of TEEs
Syscalls
Untrusted
Applications inside enclaves cannot make syscalls
Enclave
Hardware
OperatingSystem
Service
libOSesIdea: Implement a small “OS” inside the enclave
Enclave
Hardware
OperatingSystem
Service
Application Code
libOS Service
libOSesIdea: Implement a small “OS” inside the enclave
Enclave
Hardware
OperatingSystem
Service
Application Code
libOS Service
libOSes
"Syscalls"
Idea: Implement a small “OS” inside the enclave
Enclave
Hardware
OperatingSystem
Service
Application Code
libOS Service
libOSes
"Syscalls"
Idea: Implement a small “OS” inside the enclave
Enclave
Service locallywhen possible
Hardware
OperatingSystem
Service
Application Code
libOS Service
libOSes
"Syscalls"
Idea: Implement a small “OS” inside the enclave
Enclave
Service locallywhen possible
Syscalls
Graphene-SGX
Tsai et al., ATC 2017
A libOS for Intel SGX that supports some services
Graphene-SGXA libOS for Intel SGX that supports some services
fork
exec
Graphene’s supported services:
pipes, signals, semaphores
Tsai et al., ATC 2017
Graphene-SGXA libOS for Intel SGX that supports some services
fork
exec
Graphene’s supported services:
pipes, signals, semaphores
What constitutes a CDN?
Cache
Web Application Firewall
Key Server
Web serverMultipletenants
Needs plaintext
Needs disk
Needs safe
storage
Graphene-SGXA libOS for Intel SGX that supports some services
fork
exec
Graphene’s supported services:
pipes, signals, semaphores
What constitutes a CDN?
Cache
Web Application Firewall
Key Server
Reading & writing files
Shared memory
Also critical to a CDN:
Access to private keys
Web serverMultipletenants
Needs plaintext
Needs disk
Needs safe
storage
The first truly keyless CDN
fork
exec
Graphene’s supported services:
pipes, signals, semaphores
What constitutes a CDN?
Cache
Web Application Firewall
Key Server
Web serverMultipletenants
Needs plaintext
Needs disk
Needs safe
storage
PhoenixContainers of enclavesConclaves
The first truly keyless CDN
fork
exec
Graphene’s supported services:
pipes, signals, semaphores
What constitutes a CDN?
Cache
Web Application Firewall
Key Server
Reading & writing files
Shared memory
Also critical to a CDN:
Access to private keys
Web serverMultipletenants
Needs plaintext
Needs disk
Needs safe
storage
PhoenixContainers of enclavesConclaves
Enclave
Enclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Enclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
TLS
Enclaves mutually authenticate via
attested TLS
Knauth et al., 2018
Enclave
Enclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
TLS
Private key operation
Enclaves mutually authenticate via
attested TLS
Knauth et al., 2018
Enclave
Enclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
TLS
Private key operation
ResultEnclaves mutually authenticate via
attested TLS
Knauth et al., 2018
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Shared memory read
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Shared memory read
Shared memory write
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Application Code
libOS Service
Enclave
Enclave
Memory Server
Hardware
OperatingSystem
Service
Shared memoryConclaves
Application
libOS Service
Enclave
Enclave
Memory Server
Hardware
OperatingSystem
Service
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory ServerRPC
Hardware
OperatingSystem
Service
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory ServerRPC
Hardware
OperatingSystem
Service
Coordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
Coordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
SyscallCoordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
SyscallCoordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
Syscall
Memory file Encrypted on untrusted disk
Coordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
"Syscall"Enclave
Memory Server
Hardware
OperatingSystem
Service
Syscall
Memory file Encrypted on untrusted disk
Coordinates locks Maintains memory locations
fcntl()
Shared memoryConclaves
Application
libOS Service
Enclave
Enclave
Memory Server
Hardware
OperatingSystem
Service
Memory file Encrypted on untrusted disk
Coordinates locks Maintains memory locations
Shared Memoryfcntl()
Shared memoryConclaves
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Enclave
File Server
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Enclave
File Server
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Enclave
File Server
File access
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Insight: Treat enclaves like a distributed system Implement services using kernel servers
Enclave
Memory Server
Enclave
File Server
File access
Verified data
Application Code
libOS Service
Enclave
Enclave
File Server
Hardware
OperatingSystem
Service
File system accessConclaves
Application Code
libOS Service
Enclave
Enclave
File Server
File system accessConclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
Application
libOS Service
Enclave
Enclave
File Server
Hardware
OperatingSystem
Service
read()
Conclaves
Merkle TreeEncrypted on untrusted disk
File system access
Application
libOS Service
Enclave
"Syscall"Enclave
File Server
Hardware
OperatingSystem
Service
read()
Conclaves
Merkle TreeEncrypted on untrusted disk
File system access
Application
libOS Service
Enclave
"Syscall"Enclave
File ServerRPC
Hardware
OperatingSystem
Service
read()
Conclaves
Merkle TreeEncrypted on untrusted disk
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
Syscall
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
Syscall
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
Syscall
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
Syscall
Verifies branches
Decrypts blocks
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
"Syscall" Enclave
read()
libOS
Conclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
RPC
Syscall
Verifies branches
Decrypts blocks
ext2fs server
Block layerMerkle root
File system access
Application
libOS Service
Enclave
read()
File system accessConclaves
Hardware
OperatingSystem
Service
Merkle TreeEncrypted on untrusted disk
Enclave
ext2fs server
libOS
Block layerMerkle root
Data Verifies branches
Decrypts blocks
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
Key ServerEnclave
Web server
Cache
Web Application Firewall
Execution environment is a distributed system of enclaves
Enclave
Memory Server
Enclave
File Server
The first truly keyless CDNPhoenixContainers of enclavesConclaves
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
Key ServerEnclave
Web server
Cache
Web Application Firewall
Execution environment is a distributed system of enclaves
Enclave
Memory Server
Enclave
File Server
Conclave
The first truly keyless CDNPhoenixContainers of enclavesConclaves
EnclaveEnclave
Web server
Cache
Web Application Firewall
Enclave
Web server
Cache
Web Application Firewall
The first truly keyless CDN
Key Server
PhoenixContainers of enclavesConclaves
Enclave
Web server
Cache
Web Application Firewall
Execution environment is a distributed system of enclaves
Enclave
Memory Server
Enclave
File Server
Conclave
fork
exec
Conclaves supported services:
pipes, signals, semaphores
Reading & writing files
Shared memory
Access to private keys
Trusted time server
The first truly keyless CDNPhoenix
The first truly keyless CDNPhoenix
Websites delegate provisioning to CDNs
Phoenix supports many deployment configurations
Other details in the paper
Supports multi-tenancyBoth CDN and website can store private data
The first truly keyless CDNPhoenix
Websites delegate provisioning to CDNs
Phoenix supports many deployment configurations
Other details in the paper
Conclave Conclave Conclave
Supports multi-tenancyBoth CDN and website can store private data
The first truly keyless CDNPhoenix
Conclave Conclave Conclave
ARTIFACTEVALUATED
PASSED
Implemented on top of Graphene-SGX
Evaluated to understand throughput and scalability
0 500
1000 1500 2000 2500 3000 3500
1 KiB 10 KiB 100 KiB
# Workers1 2 4 8
Thro
ughp
ut (r
eque
sts/
sec)
Downloaded file size
Fetch a file 10,000 times over non-persistent HTTPS connections from among 128 concurrent clients
What is Phoenix’s request throughput?
0 500
1000 1500 2000 2500 3000 3500
1 KiB 10 KiB 100 KiB
# Workers1 2 4 8
Thro
ughp
ut (r
eque
sts/
sec)
Downloaded file size
Linux
Fetch a file 10,000 times over non-persistent HTTPS connections from among 128 concurrent clients
NGINX running on normal Linux
What is Phoenix’s request throughput?
0 500
1000 1500 2000 2500 3000 3500
1 KiB 10 KiB 100 KiB
# Workers1 2 4 8
Thro
ughp
ut (r
eque
sts/
sec)
Downloaded file size
LinuxPhoenix-cryptPhoenix-vericrypt
Fetch a file 10,000 times over non-persistent HTTPS connections from among 128 concurrent clients confidentiality
confidentiality & integrity
What is Phoenix’s request throughput?
How does Phoenix scale to multiple tenants?
0 200 400 600 800
1000 1200 1400 1600
1 2 4 6
Tim
e pe
r req
uest
(ms)
Number of tenants
Linux (shared NGINX)
40 ms
264 ms
0 200 400 600 800
1000 1200 1400 1600
1 2 4 6
Tim
e pe
r req
uest
(ms)
Number of tenants
Linux (shared NGINX)
Phoenix-crypt (shared nothing)
816
32
48
How does Phoenix scale to multiple tenants?
264 ms
40 ms127 ms
1437 ms
Number of enclaves
0 200 400 600 800
1000 1200 1400 1600
1 2 4 6
Tim
e pe
r req
uest
(ms)
Number of tenants
Linux (shared NGINX)Phoenix-crypt (shared NGINX)
810
14
18
Phoenix-crypt (shared nothing)
816
32
48
Number of enclaves
How does Phoenix scale to multiple tenants?
264 ms
40 ms
128 ms127 ms
1437 ms
806 ms
0 200 400 600 800
1000 1200 1400 1600
1 2 4 61
1K
10K
100K
1M
10M
Tim
e pe
r req
uest
(ms)
SGX
pagi
ng e
vent
s
Number of tenants
Linux (shared NGINX)Phoenix-crypt (shared NGINX)
810
14
18
Phoenix-crypt (shared nothing)
816
32
48
How does Phoenix scale to multiple tenants?
Number of enclaves
264 ms
40 ms
128 ms
806 ms
127 ms
1437 ms
Other results
Benchmark overhead of running WAFs (ModSecurity) in SGX (overhead about the same as in Linux)
Perform detailed micro-benchmarks of each kernel server
Compare standard ocalls to exitless ocalls(not always better)
ARTIFACTEVALUATED
PASSED
Moderate performance overheads
https://phoenix.cs.umd.edu/
0 500
1000 1500 2000 2500 3000 3500
1 KiB 10 KiB 100 KiB
# Workers1 2 4 8
Thro
ughp
ut (r
eque
sts/
sec)
Downloaded file size
LinuxPhoenix-cryptPhoenix-vericrypt
The first truly keyless CDNPhoenixContainers of enclavesConclaves
Run legacy apps in enclaves