IP Technology for Media Production

The Last Workflow to Join

Graphics - File

NLE, NLW - File

DVB, VOD - Packets

Ingest - Packets

IPTV - Packets

OTT - Packets

Where we have been

Color television workflows embodied a brittle,

camera-to-the-home stovepipe

Production was Directly attached to the home TV

Every business model ROI was based on 1 to many

Kanabus, et al findings

• Over 75% correct

responses when

• Visual A to Visual B

stimuli are separated

by 80ms

• Auditory to Visual

stimuli (not shown on

graph) separated by

5ms

(Inter-Stimulus-Interval)

75%

Correct

Responses

80 ms

Mike Cronk/Chuck Meyer, Grass Valley – VSF October, 2016

Color Television Time Domain

SC/H Phase

Leader, Teleproduction Test Volume 1 Number 10

Broadcast Spectrum

Strict Spectral Control

Tight Frequency Tolerance

Managed Signal Amplitude

Production Switcher

Circa 1970

nsec SC/H Phase resolution

Memory does not exist

Production Video Multiplier

Pixel Alignment Required: 64 MB RAM is 100 USD

Circa 1995

Where we are Now

Digital television ( HD ) broke the strict connection

between the camera and the home

The requirement for nano-seconds is gone

Video production became line based

IP – Faster than real time –

Many new ROI models are proposed, leveraging IP

technology

Workflow Timing Model

SDI Still Used Latency Stratum Network

Latency Human Factor

Absolute Real-

Time ~10 us Imperceptible

Pseudo Real-

Time < 1 ms

Edge of

Perceptible

Fast Non-Real-

Time 100ms Noticeable

Non-Real-Time 1s PVR

Workflow Timing Model

Latency Stratum Network

Latency Workflow

Absolute Real-

Time ~10 us Computation

Pseudo Real-

Time < 1 ms Live Production

Fast Non-Real-

Time 100ms

Near On-

Line/Streaming

Non-Real-Time 1s File Based

SDI Still Used

Time Relationships

Sampling time

Processing Samples and Pixels

Homogeneous

Media Time

Relative relationship between media

Audio->Image, 3D

Lip Sync

Time of Day

Live Production Technology Inflection Point

Wire Speed Routing

$ per signal per physical network

segment

Transparent, low latency CODEC

Wire Speed Latency Update

Network Speed Delay Lines 2014 2017 2020

2K/60 4K/60 4k/120 8K/120

Line Time usec 14.8 7.4 3.7 1.9

1 GbE 37 2.5 5 10 20

10 GbE 3.7 0.25 0.5 1 2

25 GbE 1.5 0.1 0.2 0.4 0.8

40 GbE 0.9 0.06 0.12 0.24 0.5

Latency Validation Data

Courtesy

Brian Keane Aperi Corporation

Thomas Edwards Fox Networks E&O

Jitter Validation Data

Courtesy

Brian Keane Aperi Corporation

Thomas Edwards Fox Networks E&O

Technology is simply a tool

"I suppose it is tempting, if the only

tool you have is a hammer, to treat

everything as if it were a nail."

The Law of the Instrument. Abraham

Maslow, 1966

Fasteners as an analogy

Fasteners as an analogy

Fasteners as an analogy

Fastener Explosion

Each Physical Interface needs a tool

One Tool for all

Decreased labor cost

Why IP?

IP is Internet Protocol, it is just a tool

Are we trying to replace SDI with IP?

Yes, but not quite……

We are trying to leverage Internet Technology

for new business models which effectively

compete in the modern connected world

The Tool

The Tool and the principle

Encapsulation

OSI Protocol Stack

L7 = SMPTE 274

L6 = 2022-6 Header

L5 = RTP ( for example )

L4 = UDP ( for example )

L3 = Internet Protocol

L2 = Ethernet

L1 = Copper, Fiber, etc.

The argument for OSI

Audio, Video, metadata and time are all data

Strict compliance to the OSI stack ensures that

ALL media can transit over ANY physical layer

WiFi

802.XYZ ( copper, fiber )

SONET, OTN

DVB – XYZ 2

ATSC 3.0

Application of the

OSI Layer Model to

Media Streaming

Long-Haul Gateway

Layers

- Ethernet with

MPLS

Long-Haul

Gateway Layers

- Ethernet with

Sonet/SDH or

OTN

Virtualization and Abstraction

Layers 3 and 4 isolate layer 1 from layer 7

The application does not depend on the physical layer

The infrastructure is abstracted

A property of the network is bandwidth

The application may run multiple instances provided

there is sufficient bandwidth

The infrastructure enables virtualization of applications

Content is Production’s end goal

Neither the fastener nor its tool are as

important as what is being built

New Content Factories must exploit Internet

Technology to simultaneously provide

immediacy and choice

News

Episodic

Documentary

Long Segment

Sports

Reference Architecture Model

JT-NM RA 1.0 Published at IBC

2015

http://www.jt-nm.org/

Cloud Architecture

JT-NM RA 1.0 Published at IBC

2015

http://www.jt-nm.org/

Client

SaaS

PaaS

IaaS

Production Video over IP

Primary Requirements for the Value Proposition COTS router optimized

Low latency, no dropped packets

Compressed or uncompressed

Time aware

Manage bundles easily with session protocols

– Everything is in-band

Technical metadata

It is all just data

Return

RTP

Real-time Transport Protocol

Developed by the IETF for real time transport of audio and video

data.

Developed in the full context of the OSI protocol stack

First published as RFC 1889 in 1996

Same age, but ATM, while deployed, frequently with SONET, is

far outstripped by the combination of IP and Ethernet today even

in carrier networks.

IP and Ethernet now map directly, based on G.709 and OTN,

(Optical Transport Network) to replace ATM

RTP

As Internet Network Protocol and Ethernet data link layer fabrics

became dominant, and bandwidth increased, it became more

efficient to map video directly into the RTP payload without the

additional overhead of TS.

RTP/UDP is 50% lighter than TS

Mappings for RTP into virtually any other Network Protocol

already exist.

Provides the ability to abstract the network for modern cloud

architectures

Transition Comments

IP is seen as a business enabler for live production:

SaaS and Client applications have the most leverage

IaaS should be hidden, or transparent, yet easily scaled

Based on contemporary OSI stack

Ethernet, IP, RTP

BW gains and cost declines based on Moore’s Law, or Gartner Curve

– More affordable distributed systems

– Ever decreasing latency

Standards Update

October 19, 2016

44

Open Standards for Interoperability

AIMS

Active in both marketing and technical efforts

JT-NM, SMPTE 2110, VSF and AMWA

Actively contributing to the standards, specifications and engineering

guidelines which blaze the trail of the AIMS roadmap

Active with the JT-NM which established the vision

Actively orchestrating industry aligned interoperability events for

the suite of technologies

45

SMPTE - Transport

SMPTE 2022-1 to -7 all originally intended for transport and

contribution

2022-6

RTP based transport

Includes VANC, HANC and video

All current IP full bandwidth video deployments use 2022-6

Pro: It is a field provide, interoperable standard

Con: It has the same complexity as SDI. Embedding audio and other data

makes signal processing awkward.

46

SMPTE – Transport

2022-5

FEC ( Forward Error Correction ) for SMPTE 2022-6

Protect a single network segment

Robust recovery for long, continuous data drop outs

Downside is delays exceeding 20 msec, based on protection time span

Excessive delay is detrimental to live production

Delay cannot be undone

Delay all signals to the time of the last arriving signal

Push late arriving signals to the next arrival interval

47

SMPTE – 2022-7

RTP based datagram hitless switching

Applies to all RTP transports

– 2022-6, RFC-4175, AES-67, etc.

Differential path delay must be managed

Class 1 defines <= 10 msec

This delay is additive

– Caution with multi-pass

Grass Valley will propose a 100 usec Class to SMPTE for

consideration in the upcoming upgrade to 2022-7

48

SMPTE 2059 – 1/2

Interoperability testing is increasing in frequency and expanding

in product scope.

Recently, AES and SMPTE Interop was tested

AES-R16-2016

Tables 5.3.1 and 5.3.2 define the necessary parameters

5.3.2 is likely more suited to our industry needs

– Excludes the IEEE 1588 Default

49

AES-67

This is the linear PCM audio replacement

AES-3, AES-10 ( MADI )

The rate of adoption is increasing

Current interoperability is based on 1 msec and 8 (10) sample packet

construction

AIMS ( Grass Valley ) proposed adopting the 125 usec operating point with

up to 80 channels as a second operating point

– Lower latency

– Higher Channel count, 16 up to 80

– Under consideration within SMPTE 2110

50

VSF TR-04

SMPTE 2022-6

Video, Audio and VANC/HANC meta data

AES-67 for Audio

8 channels, likely more to come

AES-R16 for Interoperability

SMPTE 2059 -1/2 for Timing

Table 5.2.3 of AES-R16 defines the mutual operating point

Increased number of sync messages

51

VSF TR-03

Essence based transport

RFC-4175 transport construction for video

– Referred to the correct SMPTE standards for video formats

AES-67 for Audio

SMPTE 2059 -1/2 for Timing

– AES-R16 for interop

IETF – ANC_291 meta data

– Encapsulate SMPTE 291 meta data into RTP

– On Track for publication this September

52

VSF TR-03 Essence based transport

All essence packetization is RTP

Each essence type can be optimized

To use 1990 terminology

There is a layer for each signal type

To use 2016 terminology

All layers can traverse the facility in one common fabric

Much easier to parse and process then embedded SDI

53

SMPTE 2110

A standard designed for multiple parts

-10 is the System

-20 is baseband video

-30 is PCM audio

-40 is ancillary data

-50 is 2022-6 as an essence

More to come

54

SMPTE 2110 - 10

The System

The Network basics

– RTP, IP, UDP, etc

System timing

– PTP, SMPTE 2059, RTP

– Reference to AES-R16 place holder

SDP

– Use of session description protocol

55

SMPTE 2110 - 20

Professional video version of RFC 4175

Clarifications of video formats

Clarifications of video sampling structures

Latency considerations

Payload construction considerations

Sources of Latency (2 of 6)

• Redundancy (2022-7) Buffering: R x nR • Class A/Low Skew Path Differential <= 10ms

– per SMPTE ST 2022-7:2013

Mike Cronk/Chuck Meyer, Grass Valley – VSF October, 2016

57

SMPTE 2110 - 30

PCM Audio

There is also a proposal under consideration for wrapping

AES-3 into RTP for non-PCM audio

4 Levels of compliance

– Latency and channel count

– Lower latency level can have higher channel count

– Covers the AES-67 Table 3 options

Preferred audio channel order mapping

Use of SDP

– Alignment with AES-R16

58

SMPTE 2110 - 40

Ancillary data essence

Just started

Based on

https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-

ancillary/ IETF ANC 291

59

SMPTE 2110 - 50

SMPTE 2022-6 as essence

Bridge for 2022-6 into 2110

SMPTE 2022-5

Look for frame aligned FEC

There is also focus on 2022-7 to ensure 2110

essence definitions do not break it

Short delay required

Audio systems, Ravenna and Dante, rely on very short

delay RTP datagram protection

60

AMWA NMOS

Networked Media Open Specifications

Establish the necessary criteria for PaaS

Essential to virtualizing the network infrastructure ( IaaS )

Specification IS-04

Discovery and Registration

Utilizes classic Internet Identity methods

Demonstrated ability to operate as peer-to-peer, yet

scale up to LLDP size

Test on modest size systems, now increasing the scale

61

AMWA NMOS

Finished review of RA 1.0 with focus on control

Develop control specifications

63

AMWA NMOS

Finished review of RA 1.0 with focus on control

Continue Development of Proposed methods and

stack build up

64

Flows and Grains

65

Why?

Create an abstraction for media construction

Encoded

Object

Computer generated

Create an extensible abstraction for how media

is carried

ATM, SONET, Ethernet

66

AMWA NMOS

Continue Efforts in Connection Management

Once the Connection is made, Capability and Capacity for

interoperability must be shared

Clean On-Air Switch

A/B Master Control

Upstream pre-select Program

Pre-set

100+ Inputs

Clean On-Air Switch in IP

Clean Switch

Edge Switch

End-Point Switch

IGMP Leave and Join

Program

Pre-set

100+ Inputs

69

AMWA NMOS

New Developments for Network API

Northbound interface for the COTS Router

Broadcaster OF

3 Possible Strategies to Switch

Source Changing source ports, or IP addresses is new control paradigm

In the fabric

Mimics broadcast. Solutions will be coming to market.

End-Point

The easiest to implement

Replicates current up-stream pre-select model

72

Library of References

Reference Architecture

http://jt-nm.org/RA-1.0/index.shtml

Videos Services Forum TR-03 and TR-04 http://www.videoservicesforum.org/download/technical_recommendations/VSF_TR-

03_DRAFT_2015-10-19.pdf

http://www.videoservicesforum.org/download/technical_recommendations/VSF_TR-

04_2015-11-12.pdf

AMWA NMOS ( Networked Media Open Specifications ) IS-04

https://github.com/AMWA-TV/nmos-discovery-registration

73

Library of References

SMPTE

http://www.smpte.org

AES

http://www.aes.org