Creating rich WebRTC Applications with Kurento

Luis Lopezlulop@kurento.org

http://www.kurento.org - lulop@kurento.org2

Who I amAssociate Professor

Escuela Técnica Superior de Ingenieros de TelecomunicaciónUniversidad Rey Juan Carlos (south Madrid)

Researcher in the area of RTC

• http://www.nubomedia.eu• http://www.fiware.org

FOSS enthusiast

• Kurento project lead: http://www.kurento.org • NUBOMEDIA Community lead: http://www.nubomedia.eu

http://www.kurento.org - lulop@kurento.org3

Real-Time media Communications

Capture Encode Cipher Transport Transport Decipher Decode RenderNetwork

The media plane

The signaling plane

I wan to call you, do you accept?

Yes, do it in this way

http://www.kurento.org - lulop@kurento.org4

WebRTC to the rescue.

Before WebRTCFirst wave of

WebRTC technologies

Begin End• APIs• Standards• FOSS

Development experience

when working with real-time media

Common WebRTC application (p2p communications)

WebRTC video stream

http://www.kurento.org - lulop@kurento.org5

WebRTC infrastructures

Peer-to-Peer WebRTC Application (without media infrastructure)

WebRTC video stream

WebRTC Application based on media infrastructuremedia infrastructure

http://www.kurento.org - lulop@kurento.org6

WWW VS (Web)RTC applications

Control

Multimedia Application logic

(developers’ code)

Multimedia Capabilities

RecordingTranscoding

RoutingMixing

AnalyzingEtc.

MediaTraffic

RTC Media APIs

Multimedia Clients

Events

Application Signaling

Control

WWW Application logic

(developers’ code)

Database Capabilities

Data storageData recoveryData querying

Data processingEtc.

DD.BB. API

WWW Clients

Events

Application Signaling

http://www.kurento.org - lulop@kurento.org7

http://www.kurento.org - lulop@kurento.org8

Control

Multimedia Application logic

(developers’ code)

Multimedia Capabilities

RecordingTranscoding

RoutingMixing

AnalyzingEtc.

RTC Media APIs

Events

http://www.kurento.org - lulop@kurento.org9

Kurento: a WebRTC infrastructure an its APIs

Control

Multimedia Application logic

(developers’ code)

RecordingTranscoding

RoutingMixing

AnalyzingAdapting

MediaTraffic

RTC Media APIs

Multimedia Clients

Events

Application Signaling

Computervision

Augmentedreality

BlendingEtc.

Multimedia Capabilities

Kurento Media Server(KMS)

Kurento Client APIKurento Room APIKurento Tree API

http://www.kurento.org - lulop@kurento.org

10

Cooking Kurento

10

http://www.kurento.org - lulop@kurento.org11

The Kurento FOSS Community

Community support WebRTC worldwide reference

International awards More than 300 companies

Very active mailing list

First result in Google

http://www.kurento.org - lulop@kurento.org12

The Kurento FOSS project• http://www.kurento.org

– Main web site

• https://www.twitter.com/kurentoms – Main social channel

• https://groups.google.com/forum/#!forum/ kurento– Main mailing list

• https://github.com/kurento – Main repository

• https://www.youtube.com/channel/UCFtGhWYqahVlzMgGNtEmKug– Kurento Youtube channel

http://www.kurento.org - lulop@kurento.org13

Developing with KMS:The Kurento Client API

Media Element• Provides a specific media

functionality

› Send/receive media

› Process media

› Transform media• Exchange media through› Sources

› Sinks

Media pipeline• Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

http://www.kurento.org - lulop@kurento.org14

A modular API

• Modular in the sense of modularity*– Isolation

• Internal states of a module don’t affect the rest– Abstracion

• Internal states are hidden behind an interface– Composability

• Interface must enable module recombination and assembly– Reusability

• If a module has it, you can use it– Extensibility

• If a module does not have it, you can add it.

* Baldwin, Carliss Young, and Kim B. Clark. Design rules: The power of modularity. Vol. 1. MIT press, 2000

http://www.kurento.org - lulop@kurento.org15

The paradigm of modularity

http://www.kurento.org - lulop@kurento.org16

The Kurento Client API in one word

sourceElement.connect(sinkElement)

connect

http://www.kurento.org - lulop@kurento.org17

Kurento hello world

http://www.kurento.org - lulop@kurento.org18

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);

Control

Multimedia Application logic

(developers’ code)

RecordingTranscoding

RoutingMixing

AnalyzingAdapting

Kurento Media APIs

Multimedia Clients

Events

Application Signaling

Computervision

Augmentedreality

BlendingEtc.

Multimedia Capabilities

//Where to render the local and remote streamsvar options = {

localVideo : videoInput, remoteVideo : videoOutput, onicecandidate : onIceCandidate }

//Start media capture and communicationswebRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(

options, function(error) { if (error) return console.error(error); webRtcPeer.generateOffer(onOffer); });

Application Server Code (Java)

Client Server Code

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

http://www.kurento.org - lulop@kurento.org19

Recording

Sink

SRC

Sink

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); RecorderEndpoint recorderEndpoint = new RecorderEndpoint.Builder(pipeline).build(“file:///myfolder/myfile.mp4”);

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

http://www.kurento.org - lulop@kurento.org20

Playing

Sink

SRC SR

C

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); PlayerEndpoint playerEndpoint = new PlayerEndpoint.Buildr(pipeline).build(“file:///myfolder/myfile.mp4”); //RSTP and HTTP URIs also supported

//Connect your media elementsplayerEndpoint.connect(webRtcEndpoint);

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

http://www.kurento.org - lulop@kurento.org21

Interoperating

Media Pipeline

WebRTC audiostreaming

Sink

SRC

SRC

Sink

Media Pipeline

Sink

SRC

SRC

Sink

WebRTC videostreaming

RTP audiostreaming

RTP videostreaming

Transparent transcoding!!

http://www.kurento.org - lulop@kurento.org22

Some Media processing modules• Augmented reality

– MarkerArModule– MarkerlessArModule

• Computer vision– FaceDetector– NoseDetector– EyeDetector– CrowdDetector– PointerDetector– MotionDetector– VirtualFence– Etc.

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC

http://www.kurento.org - lulop@kurento.org23

Using media processing modules

Sink

SRC

Sink

SRC

CrowdEventsSR

C

Sink

SRC

CrowdDetector

RtspPlayer

Sink

SRC

See demo

See demo

http://www.kurento.org - lulop@kurento.org24

Implementing group communications: just connect

Media Pipeline

WebRTCstreaming

Sink

SRC

WebRTCstreaming

SRC

Sink

SRC

Sink

WebRTCstreaming

WebRtcEndpoint 1

WebRtcEndpoint 3

WebRtcEndpoint 2

SRC

Sink

WebRTCstreaming

WebRtcEndpoint 4

User 1 User 2

User 3

User 4

http://www.kurento.org - lulop@kurento.org25

Group communications: RTP topologies

Sink

SRC

SRC Sink

SinkSRC

SRC

Sink

• Media Mixing Mixer– Composite

• Media Switching Mixer– Connect primitive– Natural topology

• SFU– WebRtcSfu

• Simulcast

http://www.kurento.org - lulop@kurento.org26

Creating your own modules

KurentoIDL Compiler

JSON

Module IDL Description

Java Java-Script

cpphpp

Media API code Toolchain

Auto-generated code

ModuleImplementation

cmake

Interface

Developer

(1)

(2)

(3)

http://www.kurento.org - lulop@kurento.org27

Going up the API stack: the room API

createRoomdeleteRoom

joinRoomleaveRoom

publisMediaunpublishMedia

onParticipantJoinedonParticipantLeft

onMediaPublishedonMediaUnpublished

onMessage

See demo

http://www.kurento.org - lulop@kurento.org28

Going up the API stack: the tree API

createTreedeleteTree

joinAsPresenterjoinAsViewer

leave

http://www.kurento.org - lulop@kurento.org29

What’s next

http://www.kurento.org - lulop@kurento.org30

Where to start

• http://www.kurento.org/documentation – Installation and administration guide– Java tutorials for Spring and Java EE developers– JavaScript tutorials for Node.js developers– JSON-RPC protocol documentation

• https://github.com/kurento – Main repository

http://www.kurento.org - lulop@kurento.org31

Thanks

Luis Lopezlulop@kurento.org