WebDAV synchronization protocol optimizations....HTTP1 vs HTTP2 tests Files in parallel limitation...

HTTP/WebDAV synchronization protocol optimizations.

Piotr Mrowczynski

HTTP/WebDAV synchronization protocol optimizations.

- HTTP2 (https://github.com/owncloud/client/compare/http2)

- Bundling (https://github.com/owncloud/client/pull/5319)

- Request Scheduling (https://github.com/owncloud/client/pull/5440)

- Dynamic Chunking for new chunking algorithm (https://github.com/owncloud/client/pull/5368)

- Request Scheduling (https://github.com/owncloud/client/pull/5440)

- Prioritize by modification time (https://github.com/owncloud/client/pull/5349)

Scope of this talk

Current ownCloud WebDAV / HTTP1.1 implementation

- HTTP/1.1 without pipelining – head of line blocking

- HTTP/1.1 without pipelining – head of line blocking

- max 3-6 parallel connections (as in web)

- each file is single PUT / GET / DELETE / MKDIR / MOVE request within a single persistent (Keep-Alive Header) connection

Tim

e

MKCOL

...

Con. 1 Con. 2 Con. 6PUT PUT

Server can handle 100 requests in parallel at specific moment → client will use anyways max 6

- server can handle 100 requests in parallel at specific moment → client will bind to max 6

- server with concurrent syncs is overloaded with x6 opened connections (usually SSL)

- server can handle 100 requests in parallel at specific moment → client will bind to max 6

- server with many concurrent syncs is overloaded with opened connections

- Latency: Each file in separate connection has to waste time on latency (usually 15-300 ms)

Tim

e

Latency and transf

er

Latency

Serv

er-s

ide

Ope

ratio

ns

1000 files / 6 parallel = 167 lines of blocking → x 53 ms = 9s

x 320 ms = 53s

HTTP1, HTTP2 and BUNDLING

Tim

e

MKCOL

...


HTTP/1.1

Tim

e

MKCOL

...


HTTP/1.1

...

Con. 2BUN

DLE

Tim

e

MKCOL

Con. 1 Con. 6BUN

DLE

Con. 1

BUNDLING

Reduced latency gain

Tim

e

MKCOL

...


HTTP/1.1

Tim

e

PUTMax Parallel {

Con. 1

Bandwidth/Time Max Parallel Gap

Optimization target – request scheduling (will talk later)

HTTP 2 with ownCloud's

requests limitation

...

Con. 2BUN

DLE

Tim

e

MKCOL

Con. 1 Con. 6BUN

DLE

Con. 1

BUNDLING

Reduced latency gain

HTTP2 with ownCloud request limitation

If optimized by pumping more requests or binary data transfers: - might hide request-response latency as in bundling

- might utilize bandwidth

- only limited by server/database capability of accepting parallel files

HTTP2 - possible benefits for ownCloud

- The parallel multiplexed requests and response do not block each other.



- Optimized and faster encryption




- Binary framing – less errors, overhead and more

- Header compression




- Binary framing – less errors, overhead and more

- Header compression

- Flow control (separate from TCP flow control)

BUNDLING - possible benefits for ownCloud

- Files packed in group of requests, send over the network and single response is returned

- Above results in latency reduction and possible better network utilization

BUNDLING - possible benefits for ownCloud

- Files packed in group of requests, send over the network and single response is returned

- Above results in latency reduction and possible better network utilization

- Reduces PHP overhead (script is fired up once for whole the group instead of per file)

- PHP overhead – can be also reduced by optimizing server side for single requests

HTTP1 vs HTTP2 tests

HTTP1 vs HTTP2 testsFiles in parallel limitation

1000 files 1kB – total 1MB of data

CERNBOXGeneva, Switzerland Berlin, Germany

Melbourne, Australia

Measurement repeated 10 times using Smashbox Benchmarking Tool

SSD, 8GB RAM, 4x2,4GHz,

Openstack, 12GB RAM, 4x2.5Ghz

WiFi, 53 ms latency, Upl

76Mbit/s, Dow 79 Mbit/s

Ethernet, 320 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s


1000 files 1kB – total 1MB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool

Synchronization to CERNBox (EOS), Geneva, Switzerland

Protocol Parallel Limit Location Upload Time [s] Download Time [s]

HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9

HTTP2 98.8 +/- 34.8 53.8 +/- 19.2

HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1

HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)

HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7

HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)

Latency and WAN impact

-17s

-43s





HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9

HTTP2 98.8 +/- 34.8 53.8 +/- 19.2

HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1

HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)

HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7

HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)

1. Head of line blocking2. Server bookkeeping blocks upload

HTTP2 PipeliningClient still allows only max 6 connections

-170s





HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9

HTTP2 98.8 +/- 34.8 53.8 +/- 19.2

HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1

HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)

HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7

HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)

On latency 50ms from Berlin to Geneva, we got even 20s →50 Hz (files per second)

HTTP1 vs HTTP2 testsSSL Overhead

12 files 1kB – total 12kB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool


Protocol Files Synced Location Upload Time [s] Download Time [s]

HTTP112 320 ms, AU

6.7 +/- 1.4 5.0 +/- 1.3

HTTP2 5.9 +/- 2.2 4.4 +/- 0.4

3-way-handshake and SSL optimization gain, connection reuse limit (only ?)

-1.8s

HTTP1 vs BUNDLING tests

HTTP1 vs BUNDLING tests1000 files 1kB – total 1MB of data

DAMKEN CLOUDNuremberg, Germany Berlin, Germany

Melbourne, Australia


SSD, 8GB RAM, 4x2,4GHz,

Openstack, 12GB RAM, 4x2.5Ghz

WiFi, 37 ms latency, Upl

76Mbit/s, Dow 79 Mbit/s

Ethernet, 279 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s

HTTP1 vs BUNDLING tests Files in parallel limitation


Synchronization to Damken Cloud, Nuremberg, Germany

Protocol Bundled Files Location Upload Time [s] Download Time [s]

HTTP1 -37 ms, DE

155.1 +/- 0.4 41.0 +/- 0.8

Bundling 100 149.1 +/- 0.8 38.0 +/- 3.5

HTTP1 -

279 ms, AU

187.8 +/- 5.6 67.6 +/- 0.2

Bundling 100 158.4 +/- 3.2 66.0 +/- 1.3

Bundling 10 152.6 +/- 1.2 67.9 +/- 0.1

Latency Influence on HTTP1Bundling in this prototype works only for upload

-6s

-28s

HTTP1 vs BUNDLING tests Files in parallel limitation

100 files 1kB – total 100kB of data

1000 files to bundle →10 requests needed

100 files to bundle→6 requests needed (as number of connections)

It occurred that for 100 files sync time reduced: 20s → 16s in upload for 37ms latency

Requests Scheduling

“Wide and Narrow Pipe” problem with max 3-6 connections

2 MB/savailable

10 kB 10 kB 10 kB 10 kB10 kB

10 kB 10 kB

“Wide and Narrow Pipe” problem with max 3-6 connections

2 MB/savailable

5 MB 5 MB 5 MB

Better solution utilizing 6 connections

2 MB/savailable

5 MB 5 MB

10 kB 10 kB

10 kB 10 kB

Even better solution using HTTP/2 for fast and idle servers

Boosted using http2 prioritization? (E. Bocchi, Politecnico di Torino)

2 MB/savailable

5 MB 5 MB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

10 kB 10 kB

TU Berlin → TU Berlin Test

20 x 5MB50 x 100kB

Folder A Folder B


TU Berlin → TU Berlin TestOld implementation

Folder-wise, first folder with small, then folder with big

TU Berlin → TU Berlin TestNew implementation

Cross-folder, big files and small files at the same time

TU Berlin → TU Berlin Test

20 x 5MB 50 x 100kB

Upload: ~23s → ~20s Download: ~16s → ~13s


- Small files don't block bigger ones

- Big files don't block smaller ones

Future using HTTP2, Dynamic Chunking and Scheduling?

Sending you 3 files

Ok, but I can handle 50 files at this moment

My dynamic chunk is now 20 MB.

Sending you 30 fileswhich fit into it

I have more users now

Reduce to 15 files

- Using HTTP2 PUSH in Discovery phase? (E. Bocchi, Politecnico di Torino)

- Max Parallel Negotiation

Take Away Message- Changing the way the requests are being send may significantly reduce the sync time (HTTP2, Bundling, Scheduling)

- If you want to test HTTP2 or any of the features, please contact me [email protected]

HTTP2 Bundling

Number of Connections 1 Max 6

Optimized transfer Optimized encryption, Binary stream, header compression, 1 connectionPHP overhead reduced, files

“buffered” in bundles on server

Number of files processed in parallel

Limited only by server capability and client OS/hardware – probably max

100Max 6

Observed improvement

CERNBox + EOS (1000 files)115s → 20s on 50ms lat.210s → 40s on 320ms lat.

6.7s → 5.9s for 10 files on 320ms lat

DamkenCloud (1000 files)155s → 149s on 37ms lat188s → 158s on 279ms lat

20s → 16s for 100 files on 37ms lat

Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40

WebDAV synchronization protocol optimizations....HTTP1 vs HTTP2 tests Files in parallel limitation...

Documents

Transcript of WebDAV synchronization protocol optimizations....HTTP1 vs HTTP2 tests Files in parallel limitation...