Augmented Reality with Project Tango - Droidcon 2016 Berlin

with Project Tango

Dominik Helleberg

https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4thttp://bitstars.com/glossary/augm ented-reality/http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG

Augmented Reality

https://www.google.de/trends/explor e#q=Augmented%20Reality&cm pt=q&tz=Etc%2FGMT-1

Augmented Reality

https://www.google.de/trends/explor e#q=Virtual%20reality

Augmented Reality

http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.png

Augmented Reality

https://news.microsoft.com/microsoft-hololens-press-materials/

Augmented Reality

http://www.magicleap.com/#/home

Augmented Reality

https://www.youtube.com/watch?v=kPMHcanq0xM#t= 41

Augmented Reality

https://www.youtube.com/watch?v=CHV9xTGD4Ws

Augmented Reality

http://www.digi-capital.com/news/wp-content/upl oads/2015/04/ARVR-Forecas t.jpg

Augmented Reality

Augmented Reality - Basics

https://en.wikipedia.org/wiki/Augmented_reality

Augmented reality (AR) is a live direct or indirect view of a

physical, real-world environment whose elements are

augmented (or supplemented) by computer-generated

sensory input such as sound, video, graphics or GPS data.

-> It’s important to understand the “context”.

Augmented Reality - Examples

By Rama - http://fr.wikipedia.org/wiki/Image:Hud_on_the_cat.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=129590By Sebastian Klein/Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=3608534

HUD Displays

Context:- Vehicle (Position, Speed, Engine)


http://www.technologybloggers.org/gadgets/google-android-powered-glasses/Von Tim.Reckmann - Eigenes Werk, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=32750308

Weather Forecast(Google Glass)

Context:- Location (coarse)- Time- Weather forecast


https://commons.wikimedia.org/wiki/File:Wikitude_World_Browser_@Salzburg_Old_Town_3.jpg

Wikitude LocationsContext:

- Location (coarse)

- Direction (compass / angle)

- Location of POIs


Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409

Virtual objects

Context:- exact, realtime location and

direction of the mobile device

- camera characteristics

- 3D model of the virtual objects

- 3D model of the real objects

- Light model of the real world

Augmented Reality – Visual FX

https://vimeo.com/155092798

Augmented Reality – Virtual Objects

Context:- exact, realtime location and direction of the mobile device- camera characteristics




Augmented Reality – Virtual ObjectsRealtime location / camera chracteristics

https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan

Augmented Reality – VFXRealtime location / camera chracteristics



Context:- exact, realtime location and direction of the mobile device


- 3D model of the virtual objects- 3D model of the real objects


Augmented Reality – Virtual Objects3D Model

https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan

Augmented Reality – Visual FX3D Model


Augmented Reality – Virtual Objects3D Model of the real world



https://github.com/stetro/project-tango-poc

Augmented Reality – Virtual ObjectsLight model


http://gurneyjourney .blogspot.de/2012/01/reference-balls.htmlhttp://imgur.com/a/P2Nbp

Project Tango

https://www.google.com/atap/proj ect-tango/

https://www.google.com/atap/proj ect-tango/http://www.androidheadlines.com/2014/02/googl es-project-tango-phone-renders-full-3d-maps-heres-look-like.html

Project Tango

2014Public announcement of

“Project Tango” and release of the “peanut”

prototype

2014 / 2015“Yellowstone” Tablet

released an sold 3000 times

Sept 2016Phab2 Pro


Project Tango

• NVIDIA Tegra K1 with 192 CUDA cores

• 7.02” 1920x1200 HD IPS display (323 ppi)

• Dual-band Wi-Fi (2.4GHz/5GHz)

• Bluetooth 4.0

• 4 GB RAM

• Accelerometer

• Ambient light

• Barometer

• Compass

• aGPS

• Gyroscope


Project Tango

Infrared Laser3D depth sensing

Motion tracking camera

4 MP 2µm RGB-IR pixel sensor

Project Tango – Motion Tracking

Project Tango – Depth Sensing

https://www.google.com/atap/project-tango/about-project-tango/index.html

• Tango’s Depth sensor detects the

range from the device to surfaces within

0.5 – 4 meters

• It generates a point cloud which can be

used to generate a mesh or enhance

AR-Apps

DEMO

Project Tango – Depth Sensing

• BACKUP

Project Tango – Area Learning / indoor navigation






Project Tango – Area Learning / indoor navigation

https://www.google.com/atap/project-tango/about-project-tango/index.html

• Tango recognizes or “learns” significant

“markers” of an areas and exports them into an

ADF (“Area Description File”)

• A Tango app may load an ADF file and try to

match visible markers against the ADF file

• After “matching” on an ADF file, Tango can

determine it’s position in local ADF coordinates

https://developers.google.com/proj ect-tango/apis/overview

Project Tango – Getting started

Unreal 4 engine

Project Tango – Getting started JAVA API

Rajawali Framework

Camera Objects / Textures Lights

Rajawali Utils

Tango Java APIs Java

Sup

port

Lib

Tango Service

Sensor Data

Application Logic

Android APIs

https://github.com/Rajaw ali/Rajawali


https://github.com/Rajaw ali/Rajawali

DirectionalLight light = new DirectionalLight(1, 0.2, -1);light.setColor(1, 1, 1);light.setPower(0.8f);light.setPosition(3, 2, 4);getCurrentScene().addLight(light);

// Set-up a material: green with application of the light and// instructions.Material material = new Material();material.setColor(0xff009900);try {

Texture t = new Texture("instructions", R.drawable.instructions);material.addTexture(t);

} catch (ATexture.TextureException e) {e.printStackTrace();

}material.setColorInfluence(0.1f);material.enableLighting(true);material.setDiffuseMethod(new DiffuseMethod.Lambert());

// Build a Cube and place it initially in the origin.mObject = new Cube(CUBE_SIDE_LENGTH);mObject.setMaterial(material);mObject.setPosition(0, 0, -3);mObject.setRotation(Vector3.Axis.Z, 180);getCurrentScene().addChild(mObject);

Project Tango – Getting started AR Scene

public class ARActivity extends Activity {

@Bind(R.id.gl_main_surface_view)RajawaliSurfaceView mainSurfaceView;

protected void onCreate(Bundle savedInstanceState) {

pointCloudManager = new TangoPointCloudManager();renderer = new SceneRenderer(this);

mainSurfaceView.setSurfaceRenderer(renderer);

...

}



@Overrideprotected void onResume() {

tango = new Tango(ARActivity.this, new Runnable() {

@Overridepublic void run() {

try {connectTango();connectRenderer();isConnected = true;

} catch (TangoOutOfDateException e) { }}

});


private void connectTango() {

TangoConfig config = tango.getConfig(TangoConfig.CONFIG_TYPE_DEFAULT);

config.putBoolean(TangoConfig.KEY_BOOLEAN_LOWLATENCYIMUINTEGRATION, true);config.putBoolean(TangoConfig.KEY_BOOLEAN_DEPTH, true);config.putBoolean(TangoConfig.KEY_BOOLEAN_COLORCAMERA, true);

tango.connect(config);

...


tango.connectListener(framePairs, new OnTangoUpdateListener() {@Overridepublic void onPoseAvailable(TangoPoseData pose) { }

@Overridepublic void onFrameAvailable(int cameraId) {

if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {isFrameAvailableTangoThread.set(true);mainSurfaceView.requestRender();

}}

@Overridepublic void onXyzIjAvailable(TangoXyzIjData xyzIj) {

pointCloudManager.updateXyzIj(xyzIj);}


private void connectRenderer() {

renderer.getCurrentScene().registerFrameCallback(new ASceneFrameCallback() {

@Overridepublic void onPreFrame(long sceneTime, double deltaTime) {

...tango.connectTextureId(

TangoCameraIntrinsics.TANGO_CAMERA_COLOR,renderer.getTextureId());


if (isFrameAvailableTangoThread.compareAndSet(true, false)) {

mRgbTimestampGlThread =tango.updateTexture(TangoCameraIntrinsics.TANGO_CAMERA_COLOR);

}

TangoPoseData lastFramePose = tango.getPoseAtTime(mRgbTimestampGlThread, FRAME_PAIR);

if (lastFramePose.statusCode == TangoPoseData.POSE_VALID) {// Update the camera pose from the rendererrenderer.updateRenderCameraPose(lastFramePose, mExtrinsics);mCameraPoseTimestamp = lastFramePose.timestamp;

}


public class SceneRenderer extends RajawaliRenderer {@Overrideprotected void initScene() {

ScreenQuad backgroundQuad = new ScreenQuad();

Material tangoCameraMaterial = new Material();tangoCameraMaterial.setColorInfluence(0);tangoCameraTexture =

new StreamingTexture("camera",(StreamingTexture.ISurfaceListener) null);

tangoCameraMaterial.addTexture(mTangoCameraTexture);backgroundQuad.setMaterial(tangoCameraMaterial);

getCurrentScene().addChildAt(backgroundQuad, 0);


public void updateRenderCameraPose(TangoPoseData devicePose, DeviceExtrinsics extrinsics) {

Pose cameraPose = ScenePoseCalculator.toOpenGlCameraPose(devicePose, extrinsics);

getCurrentCamera().setRotation(cameraPose.getOrientation());getCurrentCamera().setPosition(cameraPose.getPosition());

}

Augmented Reality with Tango

DEMOs

Augmented Reality - Conclusion

http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.pnghttps://news.microsoft.com/microsoft-hololens-press-materials/

Augmented Reality - Conclusion

https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4thttp://bitstars.com/glossary/augm ented-reality/http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG

Context:- exact, realtime location and direction of the mobile device 👍

- camera characteristics 👍

- 3D model of the virtual objects 👍


- Light model of the real world 👎

Project Tango

Project Tango – Lessons Learned

Lessons learned- Point Cloud calculations are hard

- 3D is tricky sometimes

- which API to choose?

- Motion Tracking works really well

- ”closed” APIs make life hard

- Environments change more then we recognize

- Automated Testing is nearly impossible / debugging is tricky

It is a LOT of fun….

Augmented Reality - Demos

http://mashable.com/2016/02/22/project-tango-indoor-navigation/#xOEcMWsfpZqK

Augmented Reality - Demos

Project Tango

Questions?

Augmented Reality with Project Tango - Droidcon 2016 Berlin

Technology

Transcript of Augmented Reality with Project Tango - Droidcon 2016 Berlin