MediaStream processing pipelines on the Augmented Web Platform
29
https://buildAR.com - image credit It's alive! MediaStream processing pipelines on the Augmented Web Platform
-
Upload
rob-manson -
Category
Sports
-
view
1.694 -
download
0
Transcript of MediaStream processing pipelines on the Augmented Web Platform
https://buildAR.com - image credit
It's alive!
MediaStream processing pipelineson the Augmented Web Platform
Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Augmented Web Community Group
Invited Expert with the ISO, W3C & the Khronos Group
Author of “Getting started with WebRTC” - (Packt Publishing)
https://buildAR.com
The web is evolving...
1992-2000WWW: Page based websiteskey innovation: HREF
2000-2013Web 2.0: AJAX driven social serviceskey innovation: XHR
2013-...AWE: Sensor driven real-time web of thingskey innovation: ArrayBuffer
http://mob-labs.com : image credit
https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer
3. Get ArrayBuffer
5. Populate a scene Array with Array Buffer Views
6. Move through Views to process data
7. Output events and metadata
8. Update UI and/or send requests
https://buildAR.com
MediaStream processing pipelines
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process datafor each (var view in scene) { …process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { …process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { …process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com
This is where the fun is!
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { …process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com
Browser's have always had a “binary barrier”
The Video element
stream -> <video>
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
Plus other non-MediaStream pipelines:
????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
Lets take a closer look at theVideo/Canvas MediaStream pipeline
https://buildAR.com
https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) sourcea. getUserMedia()
Setup <video> element in the DOMa. declaratively then via getElementById or similarb. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>a. video.src = stream
Setup <canvas> element in the DOMa. declaratively then via getElementById or similarb. createElement(“canvas”) then appendChild()
Get 2D drawing contexta. canvas.getContext('2d');
Draw <video> frame onto <canvas>a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cyclesa. fore each(var obj in view) { … }NOTE: Integrate other streams & sensor data here
Render resultsa. using HTML/JS/CSSb. using another <canvas> and drawImage()c. using WebGLd. a combination of all
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
5. Populate a scene with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process datafor each (var view in scene) { …process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
Follow along on github: github.com/buildar/getting_started_with_webrtc
https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_webrtc
Access the camera (or some other stream) sourcea. getUserMedia()
Setup <video> element in the DOMa. declaratively then via getElementById or similarb. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>a. video.src = stream
Setup <canvas> element in the DOMa. declaratively then via getElementById or similarb. createElement(“canvas”) then appendChild()
Get 2D drawing contexta. canvas.getContext('2d');
Draw <video> frame onto <canvas>a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cyclesa. fore each(var obj in view) { … }NOTE: Integrate other streams & sensor data here
Render resultsa. using HTML/JS/CSSb. using another <canvas> and drawImage()c. using WebGLd. a combination of all
setInterval()RequestAnimationFrame()
setTimeout()
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
Join the W3C'sAugmented Web Community Group
http://w3.org/community/ar
