Netflix customer service 1 888 811 4532 Netflix Customer Support
Netflix CDN and Open Source
-
Upload
gleb-smirnoff -
Category
Software
-
view
1.249 -
download
3
Transcript of Netflix CDN and Open Source
![Page 1: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/1.jpg)
Introduction
Netflix CDN and Open Source
Gleb [email protected]
Nginx.conf 2014San Francisco
October 21, 2014
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 1 / 1
![Page 2: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/2.jpg)
Introduction to Netflix CDN
Netflix in numbers
I 50+ millions subscribers in 40+ countries
I Over 1 Petabyte of data
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 2 / 1
![Page 3: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/3.jpg)
Introduction to Netflix CDN Netflix traffic
2014 1H Sandvine North Amertica Traffic Report
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 3 / 1
![Page 4: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/4.jpg)
Netflix CDN Netflix Streaming network
Components of Netflix Streaming
I Amazon Web ServicesI Website, Business Logic, Customer AuthenticationI Data Sciences, Encoding/EncryptionI Command and Control
I Content StreamingI Big 3 CDN’s (Akamai, Limelight, Level3)I Own CDN
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 4 / 1
![Page 5: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/5.jpg)
Netflix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
I Control the server side of HTTP connection
I Build a CDN specialized in Netflix content delivery
I Put the content closer to a client
Solution: Open Connect1
1https://www.netflix.com/openconnectGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 5 / 1
![Page 6: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/6.jpg)
Netflix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
I Control the server side of HTTP connection
I Build a CDN specialized in Netflix content delivery
I Put the content closer to a client
Solution: Open Connect1
1https://www.netflix.com/openconnectGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 5 / 1
![Page 7: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/7.jpg)
Netflix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
I Control the server side of HTTP connection
I Build a CDN specialized in Netflix content delivery
I Put the content closer to a client
Solution: Open Connect1
1https://www.netflix.com/openconnectGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 5 / 1
![Page 8: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/8.jpg)
OpenConnect Introduction to OpenConnect
The OpenConnect Initiative
Amazon Cloud The Internet ISPs
video
encoding
account
authorization
streaming
control
Origin
Server
ISP
Cache
IX
Cache
Client Client
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 6 / 1
![Page 9: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/9.jpg)
OpenConnect Introduction to OpenConnect
OpenConnect Appliance
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 7 / 1
![Page 10: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/10.jpg)
OpenConnect OpenConnect hardware
OpenConnect Appliance
The big cacheI 4U chassisI Storage
I 36 HDDs, 4 Terabyte eachI 6 SSDs, 0.5 Terabyte each
I Two 10 Gbit/s Chelsio NICs
I 8-core CPU
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 8 / 1
![Page 11: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/11.jpg)
OpenConnect OpenConnect hardware
OpenConnect Appliance
The fast cacheI 1U chassisI Storage
I 14 SSDs, 1 Terabyte each
I 40 Gbit/s Chelsio NICs
I 8-core CPU
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 8 / 1
![Page 12: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/12.jpg)
OpenConnect OpenConnect software
OpenConnect Appliance software
I FreeBSD operating systemI nanobsd(8) based firmwareI some enhancements
I NGINX web serverI custom modules
I BIRD routing daemon
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 9 / 1
![Page 13: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/13.jpg)
OpenConnect OpenConnect software
Software choice
I Open source
I BSD license
I FreeBSDI Known to be fast and stableI Strong developer community
I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules
I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 10 / 1
![Page 14: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/14.jpg)
OpenConnect OpenConnect software
Software choice
I Open source
I BSD licenseI FreeBSD
I Known to be fast and stableI Strong developer community
I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules
I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 10 / 1
![Page 15: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/15.jpg)
OpenConnect OpenConnect software
Software choice
I Open source
I BSD licenseI FreeBSD
I Known to be fast and stableI Strong developer community
I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules
I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 10 / 1
![Page 16: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/16.jpg)
OpenConnect OpenConnect software
Software choice
I Open source
I BSD licenseI FreeBSD
I Known to be fast and stableI Strong developer community
I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules
I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 10 / 1
![Page 17: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/17.jpg)
OpenConnect OpenConnect operation
OpenConnect operation: serve & fill
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 11 / 1
![Page 18: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/18.jpg)
OpenConnect OpenConnect operation
OpenConnect: streaming numbers
I 400,000 stream files per appliance
I 5000-30,000 client streams per appliance
I 300-1000 clients per disk
I Clients request 80Kb-300Kb ranges
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 12 / 1
![Page 19: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/19.jpg)
Netflix & open source Dealing with open source
Dealing with open source
The traditional way:
I Take a stable, well tested version,fork it
I Develop your product on top of it
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 13 / 1
![Page 20: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/20.jpg)
Netflix & open source Dealing with open source
Dealing with open source
The traditional way:
I Take a stable, well tested version,fork it
I Develop your product on top of it
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 13 / 1
![Page 21: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/21.jpg)
Netflix & open source Dealing with open source
Dealing with open source
The Netflix rules:
I Pull the bleeding-edge version ofsoftware
I Push your changes upstreamaggressively
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 14 / 1
![Page 22: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/22.jpg)
Netflix & open source Different myths on different versions
Myths about development version
Myth #1:Development version is full of bugs
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 15 / 1
![Page 23: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/23.jpg)
Netflix & open source Different myths on different versions
Myths about development version
Myth #1:Development version is full of bugs
Truth: most bugs are discovered instable versions
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 15 / 1
![Page 24: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/24.jpg)
Netflix & open source Different myths on different versions
Myths about stable version
Myth #2:We will wait for a stable version, and
someone else will fix bugs 2
2http://en.wikipedia.org/wiki/Free rider problemGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 16 / 1
![Page 25: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/25.jpg)
Netflix & open source Different myths on different versions
Myths about stable version
Myth #2:We will wait for a stable version, and
someone else will fix bugs 2
Truth: No one will discover your bugs
2http://en.wikipedia.org/wiki/Free rider problemGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 16 / 1
![Page 26: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/26.jpg)
Netflix & open source Different myths on different versions
Discovering bugs in different versions
development stable
code is “hot” code is unmaintained
no API/ABI constraintsAPI/ABI must be
preserved
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 17 / 1
![Page 27: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/27.jpg)
Netflix & open source Different myths on different versions
More myths
Myth #3:Not following development version saves
us a lot of time
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 18 / 1
![Page 28: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/28.jpg)
Netflix & open source Different myths on different versions
More myths
Myth #3:Not following development version saves
us a lot of time
Truth: some day you will need to goforward
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 18 / 1
![Page 29: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/29.jpg)
Netflix & open source Opening sources
Myths on open source
Myth #4:Sharing code discloses know-how’s
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 19 / 1
![Page 30: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/30.jpg)
Netflix & open source Opening sources
Myths on open source
Myth #4:Sharing code discloses know-how’s
Truth: know-how’s reside in a tinypercent of code, or even outside
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 19 / 1
![Page 31: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/31.jpg)
Netflix & open source Opening sources
Noble or selfish?
I We want to influence the direction of open sourcedevelopment
I We want to outsource maintainance of our code tocommunity
I We want more eyes to examine our code
I We want more people to discover bugs in it
I And, of course, we want to be considered noblegivers of code to community ,
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 20 / 1
![Page 32: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/32.jpg)
Netflix & open source Opening sources
Noble or selfish?
I We want to influence the direction of open sourcedevelopment
I We want to outsource maintainance of our code tocommunity
I We want more eyes to examine our code
I We want more people to discover bugs in it
I And, of course, we want to be considered noblegivers of code to community ,
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 20 / 1
![Page 33: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/33.jpg)
Netflix & open source Netflix’s commitment to open source
OpenConnect performance
I OpenConnect started in 2011< 10 Gbps per appliance
I Now, in 2014, we achieve> 30 Gbps per appliance
I Next goal is > 80 Gbps per appliance
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 21 / 1
![Page 34: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/34.jpg)
Netflix & open source Netflix’s commitment to open source
OpenConnect performance
I OpenConnect started in 2011< 10 Gbps per appliance
I Now, in 2014, we achieve> 30 Gbps per appliance
I Next goal is > 80 Gbps per appliance
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 21 / 1
![Page 35: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/35.jpg)
Netflix & open source Netflix’s commitment to open source
OpenConnect performance
I OpenConnect started in 2011< 10 Gbps per appliance
I Now, in 2014, we achieve> 30 Gbps per appliance
I Next goal is > 80 Gbps per appliance
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 21 / 1
![Page 36: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/36.jpg)
Netflix & open source Netflix’s commitment to open source
Areas of focus
I network stack: sockets, TCP, drivers
I storage: drivers, UFS
I VM subsystem: caching
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 22 / 1
![Page 37: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/37.jpg)
Netflix & open source Netflix’s commitment to open source
Performance improvements
I Reducing lock/cache line contention inI link aggregation driverI kernel socket buffersI sendfile(2) kernel memory buffers
I Reducing complexity, structuring code inI kernel flowtableI NGINX sendfile(2) code
I Introducing multithreading inI UFS softupdates
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 23 / 1
![Page 38: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/38.jpg)
Netflix & open source Netflix’s commitment to open source
Performance improvements
I Reducing lock/cache line contention inI link aggregation driverI kernel socket buffersI sendfile(2) kernel memory buffers
I Reducing complexity, structuring code inI kernel flowtableI NGINX sendfile(2) code
I Introducing multithreading inI UFS softupdates
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 23 / 1
![Page 39: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/39.jpg)
Netflix & open source Netflix’s commitment to open source
Performance improvements
I Reducing lock/cache line contention inI link aggregation driverI kernel socket buffersI sendfile(2) kernel memory buffers
I Reducing complexity, structuring code inI kernel flowtableI NGINX sendfile(2) code
I Introducing multithreading inI UFS softupdates
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 23 / 1
![Page 40: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/40.jpg)
Netflix & open source Netflix’s commitment to open source
Various improvements & bugfixes
I NGINX core and modules
I IPv6 network stack
I UFS journaling
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 24 / 1
![Page 41: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/41.jpg)
Netflix & open source Netflix’s commitment to open source
Completely new features
I Per-CPU statistical counters:I Precise: do not lose updatesI Fast: do not use any locking, neither critical sections
I Completely new sendfile(2) implementation:I Doesn’t block on disk I/O!I Allows to specify readaheadI Allows to deny VM caching
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 25 / 1
![Page 42: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/42.jpg)
Netflix & open source Netflix’s commitment to open source
Completely new features
I Per-CPU statistical counters:I Precise: do not lose updatesI Fast: do not use any locking, neither critical sections
I Completely new sendfile(2) implementation:I Doesn’t block on disk I/O!I Allows to specify readaheadI Allows to deny VM caching
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 25 / 1
![Page 43: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/43.jpg)
Netflix & open source Netflix’s commitment to open source
Working with community
I unmapped I/O
I VM radix
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 26 / 1
![Page 44: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/44.jpg)
Netflix & open source Netflix’s commitment to open source
Future and work in progress
I cc netflix: new TCP congestion control algorithm
I hardware assisted TCP pacing
I kernel-side TLS offload
I SSD I/O performance improvements
I multithreading pagedaemon
I NUMA support
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 27 / 1
![Page 45: Netflix CDN and Open Source](https://reader033.fdocuments.us/reader033/viewer/2022052223/559af37f1a28abc3708b4728/html5/thumbnails/45.jpg)
Netflix & open source Netflix’s commitment to open source
Questions?
Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 28 / 1