Mauro Dalla Mura, Michele Zanin, Claudio Andreatta, Paul Chippendale,

Fondazione Bruno Kessler, Trento, Italy

Augmented Reality: Fusing the Real and

Synthetic Worlds

2 M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale

Morphological and Attribute Profiles Spectral/spatial analysis

IEEE IGARSS’12

What is Augmented Reality?

3 M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale

Morphological and Attribute Profiles Spectral/spatial analysis

IEEE IGARSS’12

What is Augmented Reality?

Can it be useful for the GRS community?

Outline

1. Introduction to Augmented Reality

2. AR and GRS: the proposed concept

3. Augmenting the user’s view

4. Augmenting geo-data

5. Discussion and conclusion

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966

Nowadays, AR technology starts to run on consumer level mobile devices (e.g.,

smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008 2014

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008 2014 . . .

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

AR applications

Gaming

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

AR applications

Gaming Entertainment

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

AR applications

Gaming Entertainment Sport

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Urban planning

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Urban planning Tourism

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Urban planning Tourism Navigation

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Urban planning Tourism Navigation . . .

. . .

Augmented Reality for GRS Augmented Reality exploiting geo-data

Availability of massive archives of geo-referenced data (e.g., remote sensing,

geo-referenced social photo websites, GIS, geo-data)

A possible exploitation of geo-data

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmentation of a 2D map superimposing the corresponding 3D model

Effective integration of AR and geo-data is becoming feasible on consumer-

level portable devices

Augmented Reality for GRS Available applications showing geo-data on smart-phones

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Layar (since Jun 2009) Peak.AR (since Oct 2009)

Wikitude (since May 2009)

SwissPeaks (since Nov 2009)

Peaks Mixare (since Mar 2010)

Device-world registration is usually not accurate (e.g., errors in horizontal orientation of 30 deg if using the on-board compass)

Photo-world registration Accurate alignment of the device and the

surrounding panorama is necessary

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Captured Photograph DEM

2D Rendered DEM

Aligned Photograph

Salient feature extraction

Photo registration

Camera position (ϕ0, λ0, h0) and pose (α, β, γ) should be estimated

Vision based approaches operates by finding the best match between visual features and synthetic ones (from a 3D model)

Hardware information (e.g., magnetometer, compass, GPS) can be used as initial clue of the pose

After photo-world registration, each pixel in the image can be associated to geographic coordinates

Panorama rendering

AR for GRS: the proposed concept

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Reality augmentation

Augmenting the user’s view

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmenting the user’s view Once registered, geo-data can

be delivered to the user Geographical toponyms GIS layers Topographic maps 3D building models . . .

New modality to query and access geo-data

Fast interactive modality (e.g., touchscreen)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmenting the geoscientist’s view

Support tool for field campaigns: Real-time guidance for field surveys (e.g., ground truth collection) In-situ validation of remote sensing data and products (e.g., land cover maps)

Aid the interpretation of geo-referenced RS data data not in the visible range time series

Forest regions from Corine land

cover 2000 (25m resolution)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Topographic map

Reality augmentation

Augmenting the geo-data archives

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Augmenting the geo-data archives

Users’ portable devices become

ground-based sensors

r c

ϕ

λ

Digital Elevation Model

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

position pose focal length

(r, c) (ϕ, λ)

f(F, ϕ0, λ0, h0, α, β, γ, DEM)

Ground-based sensing

Extract Information on the land cover

November snowline

>1785 m

December snowline >1212 m

Oct. 2007

Nov. 2007

Dec. 2007

Snow cover map

0

0.2

0.4

0.6

0.8

1

86

0

10

60

12

60

14

60

16

60

18

60

20

60

22

60

24

60

26

60

28

60

30

60

Altitude (m)

Sno

w (

pe

rce

nta

ge)

Sept Nov Jan

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

Capture vertical surfaces Massive databases of geo-tagged images

(for free!) Near-live acquisition (e.g., for damage

assessment)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

Capture vertical surfaces Massive databases of geo-tagged images

(for free!) Near-live acquisition (e.g., for damage assessment)

Limited spatial coverage Uneven resolution Occlusions

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Discussion and conclusion

The effectiveness, quality and accuracy of the AR result depends on many factors: Geo-data projection and rendering are demanding operations and should be

performed fast Estimation of the camera pose and position for achieving a precise match for

accurately overlay virtual information on the scene and for acquiring information on the landscape

High-speed wireless transfer of data (i.e., optimize network transfers)

Conclusion Hardware and software specification of consumer-level portable devices are

constantly improving The exploitation of portable devices for AR applications has just started We proposed a concept that exploiting AR technology for GRS applications Thanks to an accurate registration of the device to the world it is possible to

perform visualization of geo-data on the mobile device interactively exploitation of the mobiles’ camera as a ground-based imaging sensors

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu

Introduction Discussion and conclusion Morphological and Attribute Profiles Spectral/spatial analysis

from http://www.gls-tedesco-a-berlino.de/2065.html

Thanks for your attention!

https://venturi.fbk.eu/ http://tev.fbk.eu/marmota/

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 dallamura@fbk.eu