Real-Time Communication Testing Evolution with WebRTC

Real-Time CommunicationTesting Evolution

with WebRTC

Dr Alex Gouaillard

CoSMo Software ConsultingIMTC, W3C, IETF

RTC Interoperability: Web Apps

Chromecast

• W3C Standardization process (features supported or not)

• Stand-alone tests

• Single browser (one at a time)

(JS API) Compliance and Usual Testing

Test the Web Forward

What about WebRTC 1.0?

Why so low test coverage?

• Some parts of specifications were just written• Tests not needed early on in standardization process• Many tests need P2P communication / interop testing

Media devices APIs WebRTC 1.0 APIs

Test coverage as of November 2016 (W3C TPAC)

Ex. of RTC feature not Testable before:Network and Protocol Testing: ICE

Client A Client B

STUN server

What is my public IP address?

Your public IP address is x.x.x.x:y

Compliance and Usual Testing vs.Interoperability Testing

Compliance & Usual Testing

InteroperabilityTesting

• Interaction automation

• Stand-alone tests

• Single browser

• Interaction automation

• Two synchronized instances of the same test

• 2+ browsers run in parallel

New testing tools needed

Browsers Interaction Automation:WebDriver protocol

ChromeDriver

geckodriver

Microsoft WebDriver

SafariDriver

Browsers as a Service:hosted selenium nodes

In-Progress: WebDriver needs updatefor WebRTC Security & others

• Permission prompt

– Geolocation

– Camera

– Microphone

– Bluetooth

– …

• Fake Media / Mock Capturer

• ICE Candidate Filtering

• Unsecure origin

Wrap-Up: WebDriver for RTC Tests

• Interaction

• Multiple browsers in parallel

• Synchronization of instances of same test between browsers

Lots of tools out thereaddressing a subset of the problem

WebRTC-test

Jitsi-Hammer

Kurento Testing Framework

Jattack

Ok, existing tools are limited but …How bad can it be, really?

Rough Goal: Test all possible pairs of

[Browser] x [Browser Revision]x

[OS] x [OS Revision]

Wh

at w

e W

AN

Tto

tes

t

496

© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

Wh

at w

e co

uld

test

to

day 115

23%


38177%

1 OS

2 Oses(Not tested)


65%

Wh

at w

e A

CTU

ALL

Yte

st t

od

ay

2118%

8877%

Tested

WIP

Not tested

Ok, it’s REALLY bad …Get visibility: list and evaluate

Evaluation of Interop Testing ToolsDimension 1: Tests Triggers and frequency

6 months

Weekly (2d)

Daily / Nightly (4h)

Commit-based and fast (20mn)

Experimental (manually triggered)

Evaluation of Interop Testing ToolsDimension 2: Type of SE nodes I want to use

Self-Hosted, Pre-configured VM

Hosted, as-a-service (e.g. BrowserStack)

Self-Hosted, Ephemeral VM

Hosted, Ephemeral VM (e.g. Travis)

Local / physical machines

[Apple]

[Only stable browsers][Extensions and plug-ins][Modified browsers]

[no manual settings][Extensions and plug-ins][Apple]

[limited OS set]

Evaluation of Interop Testing ToolsDimension 3: OS I want to support

Evaluation of Interop Testing ToolsDimension 4: Which Browsers?

Easy, kind of tested already, great WebDriver support

No Apple VM on non-apple HardwareWebdriver is more recent and less mature than others

WebRTC? What’s your flavor~~ ?Permission prompt? Fake Media?

36+% of Enterprise market (Windows 7)which plugin?

Evaluation of Interop Testing ToolsDimension 5: How early do I want to test?

=> How fast do you want your fix inEx: chrome 58 chrome 61

Stable

Windows Insider

Windows Insider

Evaluation of Interop Testing Tools

Own cloud

On-premises

Dimension 6: Freedom, Convenience and Price

Hosted

Trigger

Type of SE node

# OSes

Browsers

Browser channel

Hosted/on-prem

testRTC

Hosted

Trigger

Type of SE node

# OSes

Browsers

Browser channel

Hosted/on-prem

Travis CI

Hosted

Trigger

Type of SE node

# OSes

Browsers

Browser channel

Hosted/on-prem

BrowserStack

Hosted

Trigger

Type of SE node

# OSes

Browsers

Browser channel

Hosted/on-prem

Sauce Labs

Hosted

Ok, I see what’s missing for each.What’s the right design now?

KITE: Karoshi InteroperabilityTesting Engine

Because for most of us, manually testing webrtcinterop felt like “death by overwork”

KITE: Karoshi Interoperability Testing EngineAutomating RTC P2P tests

• Generic (any app, any back-end)

• Reusable (any process, trigger, …)

• Easy to maintain (no death )

Stable / Beta / Dev / Canary

+

Own cloudOn-premises Hosted

KITE

Trigger

Type of SE node

# OSes

Browsers

Browser channel

Hosted/on-prem

KITE

Hosted

+

On-premises

KITE Design

Easy maintenance by divide and conquer

• Selenium Grid

• Test engine

• Test

• Dashboard

KITE Design

Conductor

Dispatcher

Test Runner Test RunnerTest Runner

Config => BrowserObjCompute list< N-Tuple< BrowserObj > >

• { Test, Callback }• list< N-Tuple< BO > >

handle concurrencyif resource: 1 N-tuple => 1 test runner =>1 thread

1 2 … N

• { Test, Callback }• 1 N-Tuple< BO >

Instruments N BORun TestCallback( results )

BrowserObj available• Cr/FF - Mac - Rem-BB• BrX - OSX - Rem-BaaS (SauceLabs)• BrY - OSY - Rem-VM (AWS)

ResultDashboard

• { N, Test, Callback }• list< configs >

Configuration File{

"name": "local selenium example",

"callback": "http://localhost:8080/kiteweb/datacenter",

"remotes": [

{

"type": "local",

"remoteAddress": "http://localhost:4444/wd/hub"

}

],

"tests": [

{

"name": "IceConnectionTest",

"tupleSize": 2,

"testImpl": "org.webrtc.kite.IceConnectionTest"

}

],

"browsers": [

{

"browserName": "firefox"

},

{

"browserName": "chrome"

}

]

}

Configuration File{



"remotes": [

{

"type": "local",


}

],

"tests": [

{


"tupleSize": 2,


}

],

"browsers": [

{


},

{


}

]

}

Location of dashboard to send tests results


Configuration File{



"remotes": [

{

"type": "local",


}

],

"tests": [

{


"tupleSize": 2,


}

],

"browsers": [

{


},

{


}

]

}

Registration of Selenium server

"remotes": [

{

"type": "local",


}

],

Configuration File{



"remotes": [

{

"type": "local",


}

],

"tests": [

{


"tupleSize": 2,


}

],

"browsers": [

{


},

{


}

]

}

List of tests

"tests": [

{


"tupleSize": 2,


}

],

Configuration File{



"remotes": [

{

"type": "local",


}

],

"tests": [

{


"tupleSize": 2,


}

],

"browsers": [

{


},

{


}

]

}

List of browsers to run the tests

"browsers": [

{


},

{


}

]

KITE Dashboard

That’s very nice but it feels like something is missing …

Global RTC Interoperability

Chromecast

Global RTC Interoperability: MOBILE (Browsers)?

Chromecast

Global RTC Interoperability Beyond Desktop Browsers

Appium

• Mobile testing framework

• For native, hybrid and mobile web apps

Global RTC Interoperability:NATIVE, IoT and embedded

ChromecastIoT

Electron

Global RTC Interoperability Beyond Desktop Browsers

Electron

• Open source framework

• To build cross platform desktop applications (HTML, CSS, JS)

• Combines Chromium and node.js into a single runtime

Global RTC Interoperability:Beyond 1-1

Jitsi

• Open source multiplatform

• Videoconferencing

• Instant messaging

Global RTC Interoperability:Beyond 1-1

Jitsi

Global RTC Interoperability Make it easy: Grid Manager

• Set up a grid through a web app

• Set up Selenium hubs and nodes

• Download and set up– Browsers– Browsers WebDrivers

• Backup everything for follow-up uses

• Tear-down and clean-up after tests

Thank you!

Real-Time Communication Testing Evolution with WebRTC

Engineering

Transcript of Real-Time Communication Testing Evolution with WebRTC