Web Standards for AR workshop at ISMAR13

https://buildAR.com - image credit

Web Standards for AR

An intro to the latest developmentson the Augmented Web Platform

http://www.flickr.com/photos/37380823@N03/3442372197/sizes/o/in/photostream/

Who's making these outlandish claims?

Rob Manson @nambor

CEO of MOB-labs the creators of buildAR.com

Chair of the W3C's Augmented Web Community Group

Invited Expert with the ISO, W3C & the Khronos Group

Author of Getting started with WebRTC - (Packt Publishing)

https://buildAR.com

What is the Augmented Web?

https://buildAR.com

It's what comes after HTML5

https://buildAR.com


http://www.flickr.com/photos/hbart/3417599994/sizes/o/in/photostream/

Key Concept:

1992-2000WWW: Page based websiteskey innovation: HREF

2000-2013Web 2.0: AJAX driven social serviceskey innovation: XHR

2013-...AWE: Sensor driven real-time web of thingskey innovation: ArrayBuffer

https://buildAR.com


http://www.flickr.com/photos/congvo/301678287/sizes/o/in/photostream/

https://buildAR.comKey Concept:

https://buildAR.comJSARToolkit demo

https://buildAR.comFAST algorithm feature detection demo

Test your own device at: theAWEsomeWEB.com

https://buildAR.comhttp://www.khronos.org/streaminput/


http://www.flickr.com/photos/eltman/353241574/sizes/o/in/photolist-xdspY-pTuti-5z6PrP-5WtYra-7TraYS-bPXMfB-5hVt66-frMcHT-8sTZzS-7Bjs8T-6MKBmQ-8PU77j-33fJed-Ht9xU-99aKN-cUrcA-6yEAWp-7Ezwo8-aet2qK-4d5TfF-rqwWj-9rRjqk-5Wyg3b-5WtYV4-wwJ1Z-6EdA7y-5f79J2-6G8fja-3yZu6S-4Y6cn2-rjKsj-dwh5ee-4D1juK-7uidYY-nkcwB-4VG9yz-3jUx3T-cu4KkC-cu4PmE-7498qy-6wrgXr-7Rubrv-348E4E-5yBq3E-8e96oG-bWZwfy-6QSudj-6QNEER-6QSug5-6QNEDp-6QNEFX/

First lets look at the broadStream processing pipeline concept

https://buildAR.com

Key Concept:

https://buildAR.com

So what the hell is aStream processing pipeline?

https://buildAR.comThe vision we proposed in 2010 is now here!

ARStandards Workshop in Seoul 2010 Rob Manson

https://buildAR.comStream processing pipelines

1. Get Stream

2. Connect to a Stream pipeline
A way to connect a Stream to an ArrayBuffer

3. Get ArrayBuffer

5. Populate a scene Array with Array Buffer Views

6. Move through Views to process data

7. Output events and metadata

8. Update UI and/or send requests

Now lets look at the more specificMediaStream processing pipeline

https://buildAR.com

https://buildAR.comMediaStream processing pipelines

1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection

2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines

3. Get ArrayBuffernew ArrayBuffer()

5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]

6. Move through Views to process datafor each (var view in scene) { process(view)... }

7. Output events and metadatareturn results

8. Update UI and/or send requestsWave hands inspriationally here!

https://buildAR.comThis is where the APIs are still evolving








https://buildAR.comThis is where the Augmented Web comes to life








https://buildAR.comThis is where the fun is!








webcamtoy.com

https://buildAR.com


http://www.flickr.com/photos/xerones/101094011/sizes/o/in/photostream/

Key Concept:

#$!%@#%@#$@$@!!!!!

https://buildAR.com

https://buildAR.comBrowser's have always had a binary barrier

The Video element

stream ->

https://buildAR.comNow we've broken through the binary barrier

The Video/Canvas pipeline

stream -> -> -> image data -> array buffer -> process -> output




The Mediastream Image Capture API pipeline

stream -> track -> image capture -> image data -> array buffer -> process -> output






The MediaStream Recording API pipeline

stream -> recorder -> blob -> file reader -> array buffer -> process -> output






The MediaStream Recording API pipeline

stream -> recorder -> blob -> file reader -> array buffer -> process -> output

Plus other non-MediaStream pipelines:

????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output

Now lets take a closer look at the
Video/Canvas MediaStream pipeline

https://buildAR.com

https://buildAR.comThe Video/Canvas MediaStream pipeline

Access the camera (or some other stream) sourcea. getUserMedia()

Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(video) (no need to appendChild())

Pipe camera stream into a. video.src = stream

Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()

Get 2D drawing contexta. canvas.getContext('2d');

Draw frame onto a. canvas.drawImage(video, top, left, width, height);

Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;

Burn CPU (not GPU) cyclesa. fore each(var obj in view) { }NOTE: Integrate other streams & sensor data here

Render resultsa. using HTML/JS/CSSb. using another and drawImage()c. using WebGLd. a combination of all


2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines


5. Populate a scene with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]




Follow along on github: github.com/buildar/getting_started_with_webrtc

https://buildAR.comThe Video/Canvas MediaStream pipeline

Follow along on github: github.com/buildar/getting_started_with_webrtc




Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()

Get 2D drawing contexta. canvas.getContext('2d');

Draw frame onto a. canvas.drawImage(video, top, left, width, height);

Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;

Burn CPU (not GPU) cyclesa. fore each(var obj in view) { }NOTE: Integrate other streams & sensor data here


setInterval()RequestAnimationFrame()setTimeout()

Now lets compare that to theWebGLSL based Video/Shader pipeline

https://buildAR.com

https://buildAR.comThe Video/Shader MediaStream pipeline




Compile shaders from text in elements in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()

Get 3D drawing contexta. canvas.getContext('webgl|experimental-webgl');

Load frame into fragment shadera. gl.texImage2D(..., video)

Burn GPU cyclesa. for each(var obj in view) { }NOTE: Integrate other streams & sensor data here


setInterval()RequestAnimationFrame()setTimeout()


http://www.calwatchdog.com/wp-content/uploads/2011/04/JohnCleeseSillyWalk1.jpg

Key Concept:

ArrayBuffers and minimising copiesare the key to an efficient walk

https://buildAR.com

https://buildAR.comViews are moveable windowsinto the raw data streams

1. Get streamMediaStream from getUserMedia or RTCPeerConnection

2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines







http://www.flickr.com/photos/ekcragg/5649296082/sizes/o/in/photostream/

Key Summary:

Where is the Augmented Webup to right now?

https://buildAR.com

https://buildAR.comMobile AR vs. the Augmented Web

Mobile AR at end of 2009

Augmented Web at end of 2013


Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori





Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths





QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.








QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D.






There was limited if any support for rich 3D on all the mainstream AR browsers.











Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled









Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled (except for advertisers!!!?).



https://buildAR.comWhat's in the near future?

Integrating WebRTC and Visual Search

Using WebGL/GLSL to utilise GPU parallelism

Khronos Group's OpenVX

Khronos Group's Camera Working Group

Lots more demos to share! 8)



















https://buildAR.comJoin the W3C'sAugmented Web Community Grouphttp://w3.org/community/ar


Web Standards for AR

An intro to the latest developmentson the Augmented Web Platform

http://www.flickr.com/photos/37380823@N03/3442372197/sizes/o/in/photostream/

Web Standards for AR workshop at ISMAR13

Technology

Transcript of Web Standards for AR workshop at ISMAR13