Building our own CDN
-
Upload
zoltan-nemeth -
Category
Internet
-
view
360 -
download
2
description
Transcript of Building our own CDN
Building our own CDN
How to serve millions of viewers efficiently
Zoltán Németh
at a glance Live streaming
Free broadcastingSoftware as a Service
Big events• Sony Playstation• Nintendo• Lady Gaga• Festivals
Citizen journalism• Rescue of Chilean miners• Earthquake in Japan• Revolutions in Egypt, Syria• Protests in Ukraine
• Obama campaign• Climate Reality• Football• Justin Bieber
Concurrent viewership peaks at almost 2 million
Streaming Definitions
Broadcaster, viewer
Media
Buffering
History
FMS, RTMP
Edge
UHS, HTTP
How we use the network
Live streaming properties
Continuous connection vs
chunk download
Big bandwidth
Stability critical
Large audience
Global delivery
Providers performance
Mobile networks
First approach: CDN
Pros
Scaling, flexibility
Bigger capacity
Cons
Costs
Origin requests
Limitations
Regional and provider
differences
Capacity race conditions
Multiple CDNs Global coverage solved
Costs high
Unified configuration problem
How to select which one to use
Own capacity Why
Cost-effectiveness
Keep CDNs for peaks
Transit lines
Peering
Billing model
95% / Flat rate
How to utilize own network
Basic decision logic
Peering first
Transit second
CDNs last
Cache layer
Varnish
Monitoring
POPs, Edge clusters
Viewer side switching - QoS
List all providers
In case of problem – switch
Challenges:
How to detect a problem
Sync
Broadcaster side problems
Server side prediction
Initial guess for QoS
Minimize useless switches (UX)
Cost optimization
How?
Viewers report metrics
Server side processing and
aggregation
UMS Ustream Media Server
Continuous connection to all
clients
Real-time push-based updates
First use: viewer number
display
Channel status poll
Viewer authentication and
authorization
Stream information
Java, kernel tweaks
UMS Frontend layer
Up to 150.000 clients /
machine
3 IP addresses per machine
1.5 billion outgoing messages
per machine per day (10
billion for the live cluster)
Aggregation / logic layer
Current: 400.000 clients per
media limit
Future: distributed
The data source: Streamstat
Viewer reports
UMS process
Log
Historic data processing
From log
Hadoop + Tableau
Realtime processing in UMS
Chandra
{"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPDATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc-uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_UPDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","networkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575,"streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs-level3.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UPDATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams-uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}
Streamstat reports
Types of reports
Playing
Buffering
Bandwidth
Errors, QoS events
Verticals
Geo
AS number
Media
Chandra Real-time data aggregation layer
Replaceable data store
Redis
Simple logic: based on increments
Easily scalable
Chandra demo
Decisions Ways of routing viewers
Initial setup
Updates
Stream priority
Stream module
Ways of decision
Rule-based
Dynamic
Stream module generation flow
Filter the Stream Priority list
Dedications / whitelists
Stream module generation flow
Filter the Stream Priority list
Dedications / whitelists
Stream module generation flowRule blocking UCDN
Routing on own network
Provider resolution service
Reject to CDN if full
Capacity monitoring
Lines
Clusters
What we have now
3 big CDN providers
~5 smaller providers
UMS clusters
Core in SJC
POPs: NRT, AMS, BUD
Exchanges
QoS and static rules
Future plans Growth
Dynamic rules
QoS and MBR
UMS Elastic Logic
Open sourcing
Framework
Chandra
Questions