© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Michael Raposa, Principal Consultant, AWS
Brinton Miller, SVP, Discovery Communications
Dave Duvall, SVP, Discovery Communications
November 2016
MAE303
Discovery Channel’s Broadcast Workflows and Channel
Origination on AWS
What to expect from the session
M&E on AWS
AWS Professional Services overview
Broadcast on AWS
Discovery’s journey to AWS
AWS adoption
Content ingest & processing
Broadcast playout
Playout & Distribution
Visual Effects & Editing Analytics
DAM & Archive
Digital Supply Chain
Publishing
OTT
Acquisition
AWS Media Workloads
What is broadcast playout?
• Transmission of content
• Typically over satellite (broadcast)
• From broadcaster to affiliate (and ultimately
to the audience, e.g., set-top box)
• Largest generator of company
revenue
AWS Professional Services:
Accelerating the enterprise
IT transformation
AWS Professional ServicesPartnering in your journey
Technical
specialists
Specialty practices for
AWS skills transfer,
security, infrastructure
architecture,
application
optimization, analytics,
big data, and
operational integration
Advisory
services
Portfolio strategy and
planning, cost/benefit
modeling, governance,
change management
and risk management
as it relates to
implementing the AWS
platform
Collaboration
Working together with
you and APN Premier
Partners you already
trust to provide you
with access to all
resources needed to
realize breakthrough
results
Proven process
Best practices and
patterns to help your
teams get the
foundation right, deploy
and migrate workloads,
and create a modern IT
operating model to
support your business
Focus on every aspect of cloud adoption
Prescriptive guidance and implementation
assistance across the stack:
• IT transformation leads focus on the business
aspects of cloud adoption
• Application architects help developers build
AWS-aware applications
• Big data/analytics specialists help you design
and deploy your analytics platform
• Infrastructure architects take your unique
landscape of application, network, and security
needs to the cloud
• Security leads provide well-proven designs to
meet your compliance needs
• Operational integration specialists help you
successfully adopt DevOps practices
Application
Optimization
Big Data & Analytics
Infrastructure
ArchitectureS
ecu
rity
, R
isk &
Co
mp
lian
ce
Op
era
tio
nal
Inte
gra
tio
n
IT Transformation
Broadcast on AWS
Broadcast on AWS overview
Pre-rendered content
• Video
• Graphics
• Overlays
• Ads
• Etc.
Storage
Scheduler
GPU Playout
Live assembly of content
• Download all the pre-rendered pieces based on schedule
• Live assembles the pieces
• Outputs the stream
Injects schedule
into playout
Downloads
content
On-premises DC
50 mbps transmission
Affiliate
Users Users Users
Affiliate Affiliate
Broadcast challenges on AWS
• Live transcoding
• Designed for 99.999% availability
• Migration trifecta
• Hardware to software
• Bare metal to hypervisor
• On-premises to cloud
• Massive scale
• TCO
• Fixed bitrate output – NOT OTT ABR
NFS storage on AWS
• On-premises video assets stored on NFS
• Minimize changes to AWS
• EFS (Preview) performance testing
• 6.1 GBps read & 3.1 GBps write
• 375 clients
• “Three AZ replication is not cost effective for our
high-availability architecture.”
–DiscoveryAmazon EFS
NFS storage on AWS
• Try Intel’s Lustre File System
• Lustre performance testing
• 6.1 GBps read & 3.1 GBps write
• 375 clients
• 0.0 dropped frames per 1M
• “We prefer a managed storage solution.”
–Discovery
NFS S3 storage on AWS
• Switch to S3
• New file system interface
• Refactoring
• S3 performance testing
• 19 GBps read & 20 GBps write
• 500 clients
Amazon
S3
NFS S3 storage on AWS
• S3 is TOO FAST!
• Afraid that it will conflict with 50 mbps out
• Wrote S3 rate limiter
• Control download/upload speed to S3
• Token bucket algorithm
• Multi-threaded, byte-range request for high
throughput
• GitHub Gist: http://bit.ly/2e1Nxf1
Amazon
S3
NFS S3 storage on AWS
• We need to replicate content across regions
• S3 cross-region replication – automated, fast, &
reliable
• Is it fast enough? Let’s test it:
• Single file xFer US-East to/from Dublin: 31 MBps
• Aggregate throughput (multiple xFers): 103
MBps
• 2,993 xFers with 0 failures
Amazon
S3
Clustering MySQL for Auto Scaling
• Using MySQL with Galera for multi-master
replication
• How do use Auto Scaling where
“instances” have different roles?
• Master
• First node in Auto Scaling group
• Starts a cluster
• Slave
• Second+ node in Auto Scaling group
• Joins an existing cluster
• Common pattern with other software, e.g.
RabbitMQ
Clustering MySQL for Auto Scaling
First
instance
in Auto
Scaling
group
Get instance ID
from instance
metadata
Using instance ID
determine Auto
Scaling group
membership
AWS API
Promote to master
Second+
instance
in Auto
Scaling
group
Get instance ID
from instance
metadata
Using instance ID
determine Auto
Scaling group
membership
Get IP of other
cluster nodes in
Auto Scaling group
Contact master &
join the cluster
curl http://instance-data/latest/meta-data/instance-id
Extracting technical metadata from S3 object
• Video assets being uploaded to an S3
bucket
• How do you quickly get the technical
metadata about the asset, e.g., bit rate,
audio codec, aspect ratio, etc.
• Use Lambda and Mediainfo
• Blog: Extracting Video Metadata Using
Lambda and Mediainfo
• Extracting Video Metadata using
Lambda and Mediainfo
S3 bucket
AWS
Lambda
Cross-account auditing
• Third party needs access to your AWS
account
• Ex: managed service provider
• You want to limit and audit access
• Solution: cross-account IAM roles
• But the IAM role looks “shared” – How
do I audit access?
3rd Party Account
Cross-account auditing
• Use CloudWatch Logs to audit
• Blog: How to Audit Cross-Account
Roles Using AWS CloudTrail and
Amazon CloudWatch Events
• NEW! - Shared EventIDs
• AWS CloudTrail Now Tracks
Cross-Account Activity to Its Origin
3rd Party Account
22
Video removed for uploading; re-insert in
Speaker Ready Room
23
The big opportunity…
Transition Discovery’s supply chain
and business systems to a scalable
and flexible infrastructure to improve
speed to market, increase quality of
services, and reduce costs
Storage and infrastructure resources providing an architectural platform
Content distribution
Data mgmt
Media mgmt
Business systems
24
Global scale
• 6 data centers
around the globe• Sterling, VA
• Silver Spring, MD
• Miami, FL
• London UK (2)
• Paris
• Singapore
25
Why cloud? Why now?
Challenging information
Security threats
Growing business and commercial
opportunities, new platforms
Changing technology environment
Global company with
global supply chain
26
Evaluation process
Educate & evaluate Dogfood/foundational
Cloud Tiger team
AWS architecture workshop
Financial case & scale
Foundational design
Cloud native app build
Hybrid app build/lift and shift
Financial analysis
Migration Factory
Media vendor research Monitoring & compliance
27
Our view of the world: What can we move?
Linear delivery
StorageDelivery encoding
Post/ custom Playout
Transcode
Incoming content
Tech eval
Yellow light:• Industry newly defined• Discovery in POC
Green light: • Industry-proven• Discovery tested
Red light:• Industry to be defined• Discovery untested
HR systems
DR/data center
Finance systems
Collabora-tion tools
Ad sales systems
Affiliate systems
Office and portal tools
Content systems
MEDIA TECH CORP IT
Key
28
What workflows to move first?
• Corporate IT migration will be a combination of lift and shift and refactoring apps. Focus is 80%+ over the next year
• Content pipeline has already moved.
• Cloud delivery has allowed us to normalize content delivery globally.
• Additional media workflows required content library in AWS.
• Linear playout starts moving now.
• 150 channels over the next 12 months.
• Required significant development by DCI and our vendors.
• Post Production – Soon.
• Content volume is a challenge.
• Software licensing is a challenge.
• B2B distribution (reduction of satellite footprint).
• Vendors are not motivated to change.
• POCs start in March.
29
Network design for media
Bandwidth & interconnection
Security & inspection
Timing/latency
Resiliency/redundancy
Operational needs
Risks
30
Network design for media
Bandwidth and interconnection
Redundant colocation Direct Connect sites
Playout traversing back to on-premises uplink
facilities
Timing and latency
PTP is a must for real-time video traffic
Account for latency to resync primary and
backup streams for playout
31
Network design for media
Resiliency/redundancy
Redundant DirectConnect sites (Equinix,
CoreSite)
Juniper carrier grade routing design
(MPLS, Advanced QoS)
Security and inspection
VRFs used to isolate video traffic, SGs
used for service segmentation
Service-based routing - only subset of
traffic inspected by layer 7 firewall
Project On Ramp:
Moving our media supply chain
33
Traditional delivery pipeline
Tape-based, labor-intensive
Delivery to air in weeks/months
34
Cloud content factory
• Data-driven content
workflows
• Builds redundant
content repository in the
cloud
• Scale out infrastructure
eliminates needs to build
to ceiling
• Allows us to make
workflow decisions
based on the economics
of manufacturing in real
time
35
Media Supply Chain
Inventory systems
and asset managementInfrastructure and resource
management
Video
processing:
file check, AV,
QC, Transcode
Antivirus
Content and status messaging to
regional MAMs
File delivery to S3
Media storage/cloud archive
Management
layer
File delivery
Storage
Linear playout
37
Legacy hardware
NVISIONROUTER
OMNEON SD/HD PORTS
HY/BHB MAIN
S&W QU ASARUPCONV
EVERTZDWN
MIXER
EVERTZDWN
MIXER
MIRANDAIMAGESTORE
IS750
INTUITION XG
EVERTZGPI
EMBEDDER7721DE4H
EVERTZ 9084HD
XD SVCHIP
INSERTER
EVERTZ 9084HD
XD SVCHIP
INSERTER
OMNEONDELAY
SERVERADVANCED
FEED
OMNEONDELAY
SERVERADVANCED
FEED
EVERTZ9084HDCC ENC
EVERTZ9084HDCC ENC
MIRANDAIMAGESTORE
IS750
INTUITION XG
MIRANDAIMAGESTORE
IS750
INTUITION XG
MIRANDAIMAGESTORE
IS750
INTUITION XG
EVERTZGPI
EMBEDDER7721DE4H
EVERTZGPI
EMBEDDER7721DE4H
EVERTZGPI
EMBEDDER7721DE4H
S&WQU ASARUPCONV
OMNEON SD/HDPORTS
HY/BHY BACKUP
S&WQU ASARUPCONV
EVERTZDWN
MIXER
EVERTZDWN
MIXER
S&WQU ASARUPCONV
HD PGM PVW
PGMPVW
LIN EAR ACOU STICAERO.2000UPMIXER
(FUTURE)ARBITRON
ARBITRON
EVERTZFRAMES YNC
SCTE104VANC EMB
HD ADV CC HD ADV
PGM REC
HD ADV PGM
HD RLTPGM
EVERTZGPI
DE-EMB7721DD4H
ARBITRON WHAK-IT
ROSSNIELSEN
ENCODERNEW-3GA
EVERTZGPI
DE-EMB7721DD4H
RLT PGM
ADV PGMREC
ADVPGM
HD PGM OUT
ADV CC
7800AD8-3G+DMX
EVER
TZ 1
2x2
R
OU
TER
IN 01
IN 02
IN 03
IN 04
IN 05
OUT 1
OUT 2
IN 02
IN 04
IN 05
IN 01
IN 03
EVER
TZ 1
2x2
R
OU
TER
OUT 1
OUT 2
4 x 1 BYPASS
IN 01
IN 02
IN 03FROM ADV
T O W E ST
IN PUT PATCH
20
22
21
1 91816
PGM OUT
Marsha lMonitor
TO ADV MASSLOGGER
TO ADV MASSLOGGER
TO QC1 PATCH
TO QC1 PATCH
EVERTZAUDIO
DE-EMBEDDER
DTMF DECODER
TO WEST INPUT PATCH
ROUTER
ROUTER
1
7
8
10
11
9
14
54
19
18
15 16
4 5
10
7
8
9
11
1 4
1
17
TO MOTOROLA
TO MOTOROLA
TO DEM
20
22
21
TO MOTOROLA
TO MOTOROLA
EAST COASTSIMULCASTAERO2000
SIGNAL FLOW3/10/14
SD eq uipment
RLT
ADV
MVP view
TOC MVP only
HD equipment
AUDIO
• Drawing to the right
represents a single
channel/air chain
• Each of the 64 icons
represents a device
• Represents 100K in cap
investment
• Takes months to deploy
• Support complexity
• Exiting equipment is end of
life and end of support
38
Why playout in public cloud?
• Motivation to go fast was burning technology and real estate issues in a few facilities around the globe
• How do we support the fast changing world of Discovery?
• Stand up 6 feeds for 2.5 weeks every other year in support of the Olympics
• Support for new digital products
39
Challenges getting there
• Traditional media vendors
• Most have not embraced software only solutions.
• Those that have still focused on “private cloud”
• Industry challenges
• Measurement tech slow to port to software.
• Economic challenges
• Shift from capex to opex
• Must go all-in for a great financial story
• Public cloud architecture challenges
• Networking – egress, multicast support, storage, SLAs or lack of
• Freaks the hell out of the staff
40
Move key functions to software
Video
Server
Video
Mixer
Graphics
Insert
V-chip(Parental
ratings)
Delay
Server
Video
Switcher
GPI Box(MVPD
commercial
insertion)
Audio
Mixer
Nielsenwatermark
CC/Subtitle
EncodeDistribution
EncoderStat
Mux
Sat
Uplink
Hardware Layer
Software Layer
Distribution Infrastructure Remains Hardware
Housed in Discovery Data Centers
Video
Server
Video
Mixer
Graphics
Insert
V-chip(Parental
ratings)
Delay
Server
Video
Switcher
GPI Box(MVPD
commercial
insertion)
Audio
Mixer
Nielsenwatermark
CC/Subtitle
Encode
41
Move to cloud
Vide
o
Serv
erVideo
Mixer
Grap
hics
InsertGPI
Box(MVPD
commercial
insertion)
V-
chip
Dela
y
Serv
erVideo
Switc
her Audi
o
Mixe
rNiels
enwater
markCC
Enc
US – East
CoastEU - Ireland
Vide
o
Serv
erVideo
Mixer
Grap
hics
InsertGPI
Box(MVPD
commercial
insertion)
V-
chip
Dela
y
Serv
erVideo
Switc
her Audi
o
Mixe
rNiels
enwater
markCC
Enc
Distribution
EncoderStat
Mux
Sat
Uplink
42
COLO/TELCO/VRF
“Air chain”
US EAST REGION
AZ #1 AZ #2 AZ #3
EU WEST REGION
AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
43
COLO/TELCO/VRF
Broadcast reliability
AZ #1 AZ #2 AZ #3 AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
US EAST REGION EU WEST REGION
44
COLO/TELCO/VRF
Broadcast reliability
US EAST REGION
AZ #1 AZ #2 AZ #3
EU WEST REGION
AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
AZ #4
45
COLO/TELCO/VRF
Broadcast reliability
AZ #1 AZ #2 AZ #3 AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
US EAST REGION EU WEST REGION
46
COLO/TELCO/VRF
Broadcast reliability
AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
EU WEST REGION
AZ #1 AZ #2 AZ #3
US EAST REGION
47
COLO/TELCO/VRF
Broadcast reliability
AZ #1 AZ #2 AZ #3
Unicast to Multicast
Conversion
Unicast to Multicast
Conversion
IP To Video
Conversion
IP To Video
Conversion
2X1Distribution
EncodingStat Mux
US Region
EU Region
EU WEST REGIONUS WEST REGION
AZ #1 AZ #2 AZ #3
48
Monitoring
What’s next
5050
This is just European distribution…….
51
• B2B distribution
• Statistical Multiplexing as a Service?
• MVPD support for come meet me approach? Cloud or colo?
• Similar but different egress issues
• Encryption: Do we need industry-specific encryption tech?
Thank you!
Remember to complete
your evaluations!
Industry Pre-Day: Monday 11/28
• MAE304 - High Performance Cinematic Production in the Cloud
• MAE302 - Turner's cloud native media supply chain for TNT, TBS, Adult Swim, Cartoon Network, CNN
• MAE301 - Accelerating the Transition to Broadcast and OTT Infrastructure in the Cloud
Regular Conference: Tuesday – Thursday 11/29 – 12/1
• ARC406 - Encoding Artifacts to Emmy Awards: Taking on Terabyte-Scale, 1-Gbps, 4K Video Processing in the Cloud
• CMP312 - Powering the Next Generation of Virtual Reality with Verizon
• CTD305 - Media Delivery from the Cloud: Integrated AWS Solutions for Premium Over the Top (OTT) Content
• STG302 - Deep Dive on Amazon Glacier
Related Sessions
Top Related