Using IT Equipment In Live Broadcast

@openbroadcastsy - #ITinBroadcast

Kieran Kunhya – kierank@obe.tv

What I’ll talk about today

• IT in Live Broadcast Television (orders of magnitude

differences for latency/bitrate compared to OTT).

• History of IT in Live Broadcast

• Being a “broadcast manufacturer” around modern

software development.

• Using IT equipment in live broadcast

• Can’t explain everything (Wikipedia your friend! -

WestminsterGuest Wi-Fi)

• Dense topics – so some attempt of lightheartedness

Who are we? What do we do?

• Founded in 2012

• Globally Distributed Team of 4, average age 27

• No traditional broadcast background (3x

Physics, 1 CompSci)

• Main business, encoders and decoders for

news/sport. Entirely around IT equipment.

• Open Source vast majority of code (Redhat-like)

• Developers of FFmpeg, VLC etc.

• Goal: All code written in house (99.99% there)

What we do (2)

• Operate a satellite downlink

centre, with nine racks and

fibre connectivity.

• Lets us develop software in

a real-world environment

(DevOps?).

What we do (3)

• Drastically lower the

barrier to entry to

broadcast television.

• Make the broadcast apps

room look exactly like an IT

server room.

• Vertically integrated

broadcast manufacturer?

Traditional hardware architecture

• Single or limited function

cards with onboard FPGAs

or DSPs doing majority of

work.

• Smaller CPU to perform

control aspects.

A true software architecture

Standard CPU performs all non-

physical data processing

ASI

SDI

MADI

AES/EBU

IP

etc.

ASI

SDI

MADI

AES/EBU

IP

etc.

A true software architecture

Standard CPU performs all non-

physical data processing

ASI

SDI

MADI

AES/EBU

IP

etc.

ASI

SDI

MADI

AES/EBU

IP

etc.

Technological changes

• Intel Nehalem (2008/09)

• Realtime 1080i encoding

• Only possible with overclocking before

• Low cost SDI/ASI boards

• Full 10-bit, VANC

• Commodity hardware

Everything available next-day!

Early trailblazers

• BBC News Raven • Software-based EVS for bureaus/sat-trucks

• Played-out News bulletins at 2012 Olympics

• Free (French ISP) • National IPTV service entirely software

driven

• Encoding/Transcoding/Remux

• 5m+ subscribers

• A lot of ISP tech built in-house

• Not just TV, STB, DSLAM etc

Myth 1: IT hardware is too large and too

power hungry

Units reside in street furniture

outside Downing Street,

Buckingham Palace etc. Low

depth (200mm) chassis, low

power CPU.

Encode/decode, ASI

processing, Intercom.

Myth 2: IT hardware isn’t powerful enough

And another 17 more

cropped out!

• Huge processing power available

• 432 CPUs (small supercomputer)

• Costs falling dramatically

• Powerful enough for example to do high-

bitrate VC-2 encoding/decoding, very high

density H.264 decoding, ASI processing.

• ~2-3x less rackspace for equivalent

functionality compared to hardware

• Processing power for new functions

Building broadcast tech the IT-way

• Can’t have expensive Tektronix/Phabrix etc. in 3 countries

• Built in-house SDI analyser (based of Dektec Dtu-351)

• Unified with IP-codebase

• Testing, testing, testing

• Fuzz testing, intelligent mutation of data to cause crashes

• Continuous process, happens 24/7 (billions of

iterations)

• Process used to find Heartbleed

• Heavy unit testing

Building broadcast tech the IT-way

Unit test physical hardware

Large number of combinations of • OS Version (kernel etc)

• Network card driver

• SDI card model

• SDI card driver

IT and IP in the Studio

Standards in software context

SMPTE 2022-6 – SDI over IP

• Pixels spanning packets

• Costly CRC

TR-03 – separate RTP

streams for video/audio

• Pixels no longer span

packets (RFC4175)

• Soon SMPTE 2110

Pixel formats

Only YUV 4:2:2 domain (as example)!

• Planar 10b – main working format

• Planar 8b - preview quality

• UYVY 10b (16-bit aligned) – SDI datastream

• Apple v210 – hardware

• Contiguous 10-bit – 2022-6/TR-03 packing

Tricky to work

with in software.

Handwritten (no intrinsics!) SIMD for every mapping (and others).

• 5-15x speed improvements compared to C

• Do it once, make it fast once and for all (until new

CPU…)

• Generic conversion library a difficult problem • Intermediate pixel format(s) always a compromise

• Add special cases until you’ve done them all!

Pixel formats

One does not simply…

Bypass the operating system

DPDK, Netmap, Registered I/O

A presentation in itself. See BBC

R&D at UKNOF:

https://www.youtube.com/watch?v=yL

L8wl8YUwA

Kernel Bypass

No network stack – You are the network stack

• Craft Ethernet, IP, UDP headers yourself

• No ARP, hardcoded MAC

• Handle most of this in userspace

• Lets you do pixel processing directly to/from

Network Card memory.

SMPTE 2022-6 in the real-world

MISSING THE POINT • 2x SDI ports per 10Gbps port (Wastes 8.5Gbps duplex – 17Gbps)

• Definitely not COTS, long lead-times, single supplier

SMPTE 2022-6 in the real-world

Issues with 2022-6 receivers not accepting software streams

• Unrealistic tolerances to jitter

• Network induced and source induced

• Interoperability tests in closed, controlled environment

• Limited number of hardware manufacturers

• Tradeshow demos don’t represent reality

More detailed work needed on real-world kit and (loaded) networks.

No metrics exist.

IT-based live production today

• Dual-capable, SDI/ASI/AES and IP

• Make it look the same to end-users, buttons,

SNMP etc.

• Incremental improvements like a web browser

• Allow users to understand multifunctionality

• Integrate seamlessly (e.g MPEG-TS)

IT-based live production tomorrow

• A BIG COMPUTE RESOURCE

• Live and offline on the same CPU

• Remote kit can be processing low-priority batch jobs

• Flash override on live broadcasts

• Possible today but no orchestrator (microservices?)

• FIMS too old (SOAP lol)

• Control network and CPU resources

Decode

MPEG-TS

multicast

Orchestrator goes and finds CPU

resources somewhere, onsite or

otherwise.

Spin me up a

multiviewer

Transcode

this file

Not possible with

SDI because SDI

associated with a

connector on a

machine.

The V-word, Virtualisation

• OS provides priorities to run multiple applications

already

• Many vendors on Windows (often without good

reason)

• Getting > 1Gbps of data out of VM nontrivial.

• Live migration of complex, stateful broadcast

applications not easy. (Not a web server…)

• Complexity may outweigh benefits.

Using IT networks for contribution

• Trend towards using generic, unmanaged, IP networks for

broadcast contribution.

• Regular in News (including cellular)

• Growing use for Sport and Channel contribution

• Packet loss and Jitter (relative to traditional metrics)

• Do it properly, using UDP. TCP not suited.

• Variations on three main techniques.

Using IT networks for contribution (1)

• SMPTE 2022-1/2 FEC

• XOR based matrix (adds 2 * matrix latency)

• Basic but wide support (albeit many broken implementations)

Row FEC Column FEC

Using IT networks for contribution (2)

• Retransmits (aka ARQ)

• Receiver requests sender

to transmit a copy of lost

packet.

• Affected by round-trip

latency

• Negative

acknowledgment

Sender Receiver

Using IT networks for contribution (3)

• Dual-pathing (SMPTE 2022-7)

• Hitless Switching

Path1

Path2

Out

Example of IT and Unmanaged IP

• Three nights of the One

Show on BBC1 delivered

using public internet

• Reverse vision video on

same unit.

• Next step (in UK), long-form

programming delivered over

cellular?

Summary

• IT equipment is on-air in the live broadcast chain today

• The advent of IP-based production provides an opportunity to use

multifunction IT equipment and gain massive flexibility.

• Standards need to reflect this

• The use of generic connectivity is increasing, and not just for

financial reasons.

• Going to be some very high-profile announcements in coming

months