Introduction to Augmented

RealityCreating Augmented reality costume: concepts, tools, decisions, failures

Aurelijus Banelis

Aurelijus Banelis

aurelijus@banelis.ltaurelijus.banelis.ltSoftware developer

Augmented Reality

What is

To the end-user

Augmented Reality

● Real world

● Real world● Camera

● Real world● Camera● Artificial

● Real world● Camera● Artificial● Follows

AugmentedReality

Hello worldexample

Complete product

≠

Augmented Reality

Computer Vision

3D Graphics

Infrastructure

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Found { marker=”m12” x = 10, y = 20, accuracy=0.8}

Found { marker=”m98” x = 15, y = 44, accuracy=0.6}

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Noises by nature

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

So filters in use

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Including redundancy

Folding

if m

arke

r1 a

nd m

arke

r2Low light

Low resolution

Similarity

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

cameraData from ARToolkit

3D models to be drawn

⇒ ++

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

3D models to be drawn

⇒ ++

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

3D models to be drawn

⇒ ++

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

3D models to be drawn

OpenGL ES 1.1

⇒ ++

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

● Frame

● Transform● Draw 3D

3D models to be drawn

OpenGL ES 1.1

⇒ ++

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

⇒

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

⇒ 3D Modeling soft Android

App

Idea → 3D models

Idea / 2D pictures

⇒ .objWavefront

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

⇒

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

⇒

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

Normals Debug

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Invest in 3D rendering

Debug separately

Consider native implementation

Copying ARToolKit native libraries takes time

Implement custom widgets for debugging(faster than reloading whole code)

Garbage collection between Java native and Buffer structures takes time (slow loading)

3D G

raph

ics

Infr

astr

uctu

reC

ompu

ter V

isio

n End-to-end

Android (SDK)

Blender

ARToolKit

QR + Server

Draw on fabric

Wraps computer vision, starting point

Custom logic (e.g. helmet position)

Generating 3D models

Telling, how to find application

Some shops requires vector images

Questions?

References and useful links● http://aurelijus.banelis.lt/prezentations/augmetned-reality-2015/ag-costume.pdf ● https://youtu.be/KIafn8hFi8c?t=4m41s

● http://artoolkit.org/ ● https://github.com/Itseez/opencv ● O’REILLY Learning OpenCV● https://en.wikipedia.org/wiki/Scale-invariant_feature_transform● https://en.wikipedia.org/wiki/Normal_%28geometry%29 ● https://unity3d.com/● https://www.blender.org/ ● https://inkscape.org/en/