Developing rich multimedia applications with FI-WARE.
-
Upload
luis-lopez -
Category
Technology
-
view
817 -
download
0
description
Transcript of Developing rich multimedia applications with FI-WARE.
![Page 1: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/1.jpg)
Open APIs for Open Minds
The Stream Oriented Generic Enabler – SO GE
Developing rich multimedia applications with FI-WARE.
![Page 2: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/2.jpg)
Speakers
2
Luis López
• Leader of the Kurento project
• As. Prof. at URJC
• Coordinator of the Stream Oriented GE at FI-WARE
Jose Antonio Santos
• Chief Architect of Kurento project
• Senior Developer at Naevatec
• Researcher at FI-WARE project
![Page 3: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/3.jpg)
The Stream Oriented Generic Enabler – SO GE
3
• Send and receive media (Multidevice/multiprotocol)• Process media (Computer vision, augmented reality, media indexing, etc.)• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)
Provides multimedia capabilities to the FI-WARE infrastructure
• No need of protocol/codec low level understanding
Exposes those capabilities through a simple to use API
• LGPL 2.1
Is distributed through a flexible FOSS licence
• Sorry for the instabilities…
Is still work in progress
![Page 4: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/4.jpg)
How media APIs (usually) work
4
Client side- WWW browser- Native API
Server side- Media repositories- Media capabilities
Hey, I want to send/receive media in this way
Here you have the media you requested
API API
![Page 5: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/5.jpg)
The SO-GE API is a WWW inspired API
5
Dynamic WWW page development
• Lots of capabilities (ex. Java EE)
Multimedia development with the SO-GE
• Don’t want to loose any capability
Create HTML- DDBB access- Transactions- Security tools- Etc.
HTTP request: I want this resource
HTTP response:The resource
Create Media- Media API- DDBB access- Transactions- Security tools- Etc.
HTTP request: I want this media
HTTP response:The media is
here
![Page 6: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/6.jpg)
The SO-GE Media API: media elements and pipelines
6
Media Element
• Provides a specific media functionality
› Building block
› Send/receive media
› Process media
› Transform media
• The Media API provides a toolbox of media elements ready to be used.
• New media elements can be added
Media pipeline
• Chain of media elements implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
![Page 7: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/7.jpg)
The global vision of a SO-GE application
7
SO-GE Media Server Infrastructure
DecodeVideo
AugmentedReality
Encode Video
Computer Vision
Video Playingand Recording
![Page 8: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/8.jpg)
Examples of media elements: HttpEndPoint
8
HttpEndPoint
Function
• Sends/receives media basing on HTTP transports (HTML5 video tag)
• Associates media to a temporary-one-time-use URL
Formats
• WebM
• H.264 support under development
Input parameters
• None
HttpEndPoint
Media Source
Media Sink
HTTP POST requests
HTTP answers
![Page 9: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/9.jpg)
Example of media elements: RtpEndPoint
9
RtpEndPoint
Function
• Sends/receives media basing on RTP transports (Real Time Communications).
• Negotiates media parameterization through SDPs
Formats
• H.263, MPEG4, H.264, VP8
Input parameters
• Input SDP
RtpEndPoint
Media Source
Media Sink
RTP inputmedia
RTP outputmedia
![Page 10: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/10.jpg)
Example of media elements: PlayerEndPoint
10
PlayerEndPoint
Function
• Plays media from file or URI.
Formats
• WebM
• MP4 (H.264)
• 3GPP
• AVI
• Etc.
Input parameters
• File URI
PlayerEndPoint
Media Source
Media fromfile or URI
![Page 11: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/11.jpg)
Example of media elements: RecorderEndPoint
11
RecorderEndPoint
Function
• Records media into a file.
Formats
• WebM
• H.264 formats under development
Input parameters
• File path
RecorderEndPoint
Media Source
Media Sink
Media tofile
![Page 12: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/12.jpg)
Example of media elements: JackVaderFilter
12
JackVaderFilter
Function
• Example of filter
• Recognizes faces
• Adds a Jack Sparrow or Darth Vader hat onto faces
Formats
• Raw
Input parameters
• None
JackVaderFilter
Sink
SRCMedia
StreamAugmented
Media stream
![Page 13: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/13.jpg)
Example of media elements: ZBarFilter
13
ZBarFilter
Function
• Example of filter
• Recognizes Bar and QR codes
• Generates events
Formats
• Raw
Input parameters
• None
ZBarFilter
Sink
SRCMedia
StreamMediastream
Event
![Page 14: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/14.jpg)
Developing media application with the SO-GE
Runtime environment
• Install Kurento Media Server (soon available at FI-LAB)
• Install JBoss Java EE container (soon available at FI-LAB)
Development environment
• Maven (repository: http://repository.kurento.com/archiva/repository/snapshots)
• Soon available at maven central
› kmf-content-api
› Provides handers
› kmf-media-api
› Provides media elements and pipelines
• Eclipse
• Javascript libraries (available at the same repository)
› kws-content-api
14
![Page 15: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/15.jpg)
My first application with the SO-GE: play a video
15
HttpEndPoint
Media Source
Media Sink
PlayerEndPoint
Media Source
Media fromfile or URI
HTTP mediastreaming
Client side Server side
![Page 16: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/16.jpg)
My first application with the SO-GE: play a video
Source code …
16
![Page 17: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/17.jpg)
My second application: including a simple filter
17
HttpEndPoint
Media Source
Media Sink
PlayerEndPoint
Media Source
HTTP mediastreaming
JackVaderFilter
Sink
SRC
Client side Server side
![Page 18: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/18.jpg)
My second application: including a simple filter
Source code …
18
![Page 19: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/19.jpg)
Making it more fun: introducing real time media
19
Media Source
Media Sink
Real timemedia
HTTP mediastreaming
JackVaderFilter
Sink
SRC
RtpEndPoint
Media Source
Media Sink
Client side Client sideServer Side
![Page 20: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/20.jpg)
Making it more fun: introducing real time media
Source code …
20
![Page 21: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/21.jpg)
Programming with media events: the ZBarFilter
21
Real timemedia
ZBarFilter
Sink
SRC
RtpEndPoint
Media Source
Media Sink
HttpEndPoint
Media Source
Media Sink
Media events to browser
Client side Client sideServer Side
![Page 22: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/22.jpg)
Programming with media events: the ZBarFilter
Source code …
22
![Page 23: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/23.jpg)
Digging into the media API: implementing your own filters: requirements Requirements
• GStreamer
› Based on GObject and GLib
› C programing skills needed
• OpenCV or your preferred image processing software
• C++ basic knowledge
• Thrift basic knowledge
23
![Page 24: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/24.jpg)
Digging into the media API: implementing your own filters: GStreamer media element implementation Download media server source code
Look for sub-module gst-kurento-plugins
Enter into src/filters folder
Create your filter as couple of filtername.c filtername.h files
• Your filter must inherit from GstVideoFilter
• You have to implement virtual method transform_frame_ip
› This method gives you each frame to be processed
• Be aware of init and finalize functions if you use attributes that need to be freed
24
![Page 25: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/25.jpg)
Digging into the media API: implementing your own filters: Integration in media server Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType
Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel case)
• This class inherits from kurento::Filter class
• Its constructor creates a “filterelement” gstreamer element and add your filter name as a parameter (it will be created automatically)
• Destructor should free this element
Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch statement
25
![Page 26: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/26.jpg)
Facing the FI-WARE challenge using the SO-GE
Getting started• Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!)
› http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/• Ask the SO-GE team around the FI-WARE stand
Augmenting videos in novel ways• Sensor information on live stream videos
› Pollution/noise/traffic Red-filtering level• Combine audio-visual & sensor information through augmented reality
› Thermometers, charts, gauges on top of a media flow.
Make a camera to become an advanced sensor using computer vision• Traffic status
› Car counter• Weather
› Sky/ground color
Conceive novel WWW and Smartphone interaction GUIs• Up to your imagination
26
![Page 27: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/27.jpg)
http://fi-ppp.eu
http://fi-ware.eu
Follow @Fiware on Twitter !
Thanks !
27
![Page 28: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/28.jpg)
Technical details: Javascript KwsContent API
Creating a media connection
• conn = new KwsContent(uri, options);
• Options
› remoteVideoTag: id of the video tag rendering the remote stream
› Others for WebRtc (currently not available)
• Semantics
› Connects to URI and requests media
› Upon media reception, renders it on the specified video tag
Adding event handlers
• conn.onstart( function …
• conn.onremotestream (function …
• conn.onterminate( function …
• conn.onmediaevent( function …
28
![Page 29: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/29.jpg)
The SO-GE Content API: handlers
Handler
• Is implemented by the application developer (equivalent to a Servlet)
• Defines the sequence of instructions executed when a request for media is received
Four types
• PlayerHandler
› Handles requests for playing media with HTTP transport
› Compatible with the HTML5 video tag
• RecorderHandler
› Handlers request for recording media with HTML transport
› Compatible with HTTP file upload mechanism
• RtpMediaHandler
› Handlers request for establishing RTP media sessions
• WebRtcMediaHandler (under development)
29
![Page 30: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/30.jpg)
Technical details: the SO-GE Architecture
30
![Page 31: Developing rich multimedia applications with FI-WARE.](https://reader033.fdocuments.us/reader033/viewer/2022061220/54846291b4af9f6e0d8b4ba2/html5/thumbnails/31.jpg)
Pla
yerH
anld
er i
nte
rnal
det
ails
: se
qu
ence
dia
gra
m
31