Kranky Geek 2015 - Decisions & Considerations in building your WebRTC App

RUNNING RTCOperational lessons learned at Twilio.

TWILIO?

Communication Channels

T W I L I O C O M M U N I C AT I O N S P L AT F O R M

VO I C E S M S M M S

W E B M O B I L E

Your Business Applications

I P M ESS A G I N G V I D E O

750,000 Developers & Businesses

50 BILLION

API Interactions a year

O P E R AT I N G AT S C A L E

TWILIO AND WEBRTC?

2011 2012 2013 2014 Present

TWILIO CLIENT 1.1: WEBRTC

TWILIO CLIENT: WEBRTC CALL VOLUME

WEBRTC RELEASED IN CHROME 23

TWILIO CLIENT 1.0: FLASH

2011 2012 2013 2014 Present

TWILIO CLIENT 1.2: GLOBAL LOW LATENCY

WEBRTC RELEASED IN FIREFOX 22

2011 2012 2013 2014 Present

TWILIO CLIENT 1.2: GLOBAL LOW LATENCY

WEBRTC RELEASED IN FIREFOX 22

COMMON USE CASES

CRM and SaaS application integration

Call center / customer service

Sharing economy

Education / expertise on-‐demand

Recruiting

Anonymous calling

A HANDFUL OF OUR WEBRTC CUSTOMERS

WEBRTC COMPLEXITY DOES NOT INCREASE LINEARLY.

Signaling Server

TURN Server

Signaling Server Media Server

Conference State

TURN Server

Signaling Server Media Server SIP Gateway

Conference State

TURN Server

Signaling Server SIP Gateway

Conference State

Media Server

TURN Server

Signaling Server SIP Gateway

Conference State

PSTN GatewayMedia Server

Carrier

TS SC PM CDC1

TS SC PM C

DC6DC1

SOMETHING WILL BREAK.

Stuff you can’t control

Random host failure

Network degradation

Spurious network traffic

Stuff you can control

Resource management

Operational mishaps

Bugs / crashes

Tens of failures / week.

100% up-‐time two quarters running.

99.95% up-‐time for the year.

TEST CONSTANTLY.

END-TO-END TESTS. CONTINUALLY TEST THE PRODUCT THE

SAME WAY THE CUSTOMER WILL.

END-TO-END TESTING

node-‐webrtcsignaling

END-TO-END TESTING

Original approach:

-‐ Test app built using node-‐webrtc.

-‐ Acted like a full-‐fledged browser client.

-‐ Ran a suite of tests at high frequency (some tests multiple times / min, other tests every few minutes).

END-TO-END TESTING

Hard to maintain:

-‐ Difficult to keep up-‐to-‐date with WebRTC changes.

-‐ Build challenges (CentOS 6 on AWS).

-‐ (node-‐webrtc has since removed MediaStream API)

END-TO-END TESTING

node signaling

mediarobocallerssignaling

END-TO-END TESTING

node signaling

Revised approach:

-‐ Frequent testing for service monitoring through simple node websocket client.

-‐ Automated browser regression testing and “thick” scenario testing with robocallers.

robocallerssignaling

Rollbar

END-TO-END TESTING

Manual testing tools:

-‐ Automated tests won’t catch everything.

-‐ Simple, reliably manual testing tools for rapidly running standard scenarios are invaluable.

End-‐to-‐end tests

Anomaly detection

Host monitoring (Nagios)

Manual tools

End-‐to-‐end tests

Anomaly detection

Host monitoring (Nagios)

Manual tools

Layered monitoring

-‐ Event aggregated in Rollbar (and assorted other tools).

-‐ Alerts delivered through PagerDuty.

PLAN FOR FAILURE, PLAN FOR RECOVERY.

Load Balancer A Load Balancer B

Gateway A Gateway B Gateway C

DNS distributionwss://www.mywebrtcservice.com

Registrar

Client

Registrar

Client

Registrar

Client

Registrar

x Many

Client

Registrar

x 10,000,000

Client

Registrar

x 10,000,000

Client

Registrar

Rate limit here to protect downstream services

x 10,000,000

Client

Registrar

Be careful about locking here

EXPECT LOUSY NETWORKS.

Network Link Conditioner

https://github.com/vadymy/networkThrottler

.GETSTATS() IS YOUR FRIEND.

pc.getStats()

AMAZON KINESIS

MONITORING

REDSHIFT

SOME OF THE STATS WE CAPTURE…

audioInputLevel

audioOutputLevel

browser

jitter

received_packets

sent_packets

received_packets_lost

SOME OF THE STATS WE CAPTURE…

SOME OF THE PATTERNS WE LOOK FOR…

High percentage of packets lost

Excessive jitter or rtt

Calls > 5 seconds with no packets sent but > 0 audio-‐input level (or the inverse; evidence of one-‐way audio)

…look for patterns by Twilio account, endpoint identity,

browser version, etc.

SOME OF THE PATTERNS WE LOOK FOR…

OPERATIONAL EXCELLENCE IS A CULTURE ISSUE.

SOME CULTURAL TOOLS

Write the SLA first

Build it, operate it

Five-‐Whys Analysis

Fire Drills

Runbooks

Post-‐Mortems

Time to Recovery Reports

COMMUNICATIONS TOOLS ARE ONLY AS GOOD AS THEIR AVAILABILITY.

ROB BRAZIER RBRAZIER@TWILIO.COM

Kranky Geek 2015 - Decisions & Considerations in building your WebRTC App

Technology

Transcript of Kranky Geek 2015 - Decisions & Considerations in building your WebRTC App

Kranky geekwebrtc demo

WebRTC & ORTC

WebRTC GatewaysIntroduction

Web Call Server - Specification - Streaming Video WebRTC ... · WebRTC Audio Calls, SIP, Messaging WebRTC Video Calls Flash Video Calls WebRTC Broadcasting Function Description Purpose

Webrtc puzzle

Kranky Geek WebRTC Show: WebRTC in the Real World

WebRTC standards overview -- WebRTC Barcelona Meetup MWC16

Upperside WebRTC conference - WebRTC intro

WebRTC Presentation

Geek to Geek: Universal Design

Kranky geeklondon build an app

Kranky Geek WebRTC 2015 - A closer look at the WebRTC UX/UI API

WebRTC overview

Kranky Geek WebRTC Show: Krank It Up!

Kranky Geek WebRTC 2015 - Best practices from billions of calls

TADSummit Comverse WebRTC, WebRTC is for telcos too. Today!

WebRTC for Telco: Informa's WebRTC Global Summit Preconference

Kranky Geek WebRTC 2015 - Optimizing the customer experience

WebRTC Workshop - What is (and isn't WebRTC)

Geek , (Name of founder) Geek , (Name of founder) Geek ...hackfwd.s3.amazonaws.com/.../97aeae296382000002/geek_agreement_v1.2.pdf · Geek Agreement Version 1.2 Geek , Geek , Geek