Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in...

33
Mauro Dalla Mura, Michele Zanin, Claudio Andreatta, Paul Chippendale, Fondazione Bruno Kessler, Trento, Italy Augmented Reality: Fusing the Real and Synthetic Worlds

Transcript of Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in...

Page 1: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Mauro Dalla Mura, Michele Zanin, Claudio Andreatta, Paul Chippendale,

Fondazione Bruno Kessler, Trento, Italy

Augmented Reality: Fusing the Real and

Synthetic Worlds

Page 2: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

2 M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale

Morphological and Attribute Profiles Spectral/spatial analysis

IEEE IGARSS’12

What is Augmented Reality?

Page 3: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

3 M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale

Morphological and Attribute Profiles Spectral/spatial analysis

IEEE IGARSS’12

What is Augmented Reality?

Can it be useful for the GRS community?

Page 4: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Outline

1. Introduction to Augmented Reality

2. AR and GRS: the proposed concept

3. Augmenting the user’s view

4. Augmenting geo-data

5. Discussion and conclusion

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 5: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 6: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966

Nowadays, AR technology starts to run on consumer level mobile devices (e.g.,

smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 7: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 8: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 9: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008 2014

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 10: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality

Augmented Reality (AR) blends virtual and real scenes in both time and space

1950s 1966 2000 2008 2014 . . .

Nowadays, AR technology starts to run on consumer level mobile devices

(e.g., smartphones and tablets) in real-time

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 11: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 12: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 13: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 14: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 15: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Urban planning

Page 16: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Urban planning Tourism

Page 17: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Urban planning Tourism Navigation

Page 18: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR applications

Gaming Entertainment Sport Commercial

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Urban planning Tourism Navigation . . .

. . .

Page 19: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality for GRS Augmented Reality exploiting geo-data

Availability of massive archives of geo-referenced data (e.g., remote sensing,

geo-referenced social photo websites, GIS, geo-data)

A possible exploitation of geo-data

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Augmentation of a 2D map superimposing the corresponding 3D model

Effective integration of AR and geo-data is becoming feasible on consumer-

level portable devices

Page 20: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmented Reality for GRS Available applications showing geo-data on smart-phones

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Layar (since Jun 2009) Peak.AR (since Oct 2009)

Wikitude (since May 2009)

SwissPeaks (since Nov 2009)

Peaks Mixare (since Mar 2010)

Device-world registration is usually not accurate (e.g., errors in horizontal orientation of 30 deg if using the on-board compass)

Page 21: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Photo-world registration Accurate alignment of the device and the

surrounding panorama is necessary

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Captured Photograph DEM

2D Rendered DEM

Aligned Photograph

Salient feature extraction

Photo registration

Camera position (ϕ0, λ0, h0) and pose (α, β, γ) should be estimated

Vision based approaches operates by finding the best match between visual features and synthetic ones (from a 3D model)

Hardware information (e.g., magnetometer, compass, GPS) can be used as initial clue of the pose

After photo-world registration, each pixel in the image can be associated to geographic coordinates

Panorama rendering

Page 22: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

AR for GRS: the proposed concept

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 23: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Reality augmentation

Augmenting the user’s view

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 24: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmenting the user’s view Once registered, geo-data can

be delivered to the user Geographical toponyms GIS layers Topographic maps 3D building models . . .

New modality to query and access geo-data

Fast interactive modality (e.g., touchscreen)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 25: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmenting the geoscientist’s view

Support tool for field campaigns: Real-time guidance for field surveys (e.g., ground truth collection) In-situ validation of remote sensing data and products (e.g., land cover maps)

Aid the interpretation of geo-referenced RS data data not in the visible range time series

Forest regions from Corine land

cover 2000 (25m resolution)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Topographic map

Page 26: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Reality augmentation

Augmenting the geo-data archives

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 27: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Augmenting the geo-data archives

Users’ portable devices become

ground-based sensors

r c

ϕ

λ

Digital Elevation Model

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

position pose focal length

(r, c) (ϕ, λ)

f(F, ϕ0, λ0, h0, α, β, γ, DEM)

Page 28: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Ground-based sensing

Extract Information on the land cover

November snowline

>1785 m

December snowline >1212 m

Oct. 2007

Nov. 2007

Dec. 2007

Snow cover map

0

0.2

0.4

0.6

0.8

1

86

0

10

60

12

60

14

60

16

60

18

60

20

60

22

60

24

60

26

60

28

60

30

60

Altitude (m)

Sno

w (

pe

rce

nta

ge)

Sept Nov Jan

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 29: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 30: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

Capture vertical surfaces Massive databases of geo-tagged images

(for free!) Near-live acquisition (e.g., for damage

assessment)

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 31: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Ground-based sensors

Extract Information on the land cover Availability of multiple images on the

same area 3D models with photogrammetry Analysis of the land cover evolution

Capture vertical surfaces Massive databases of geo-tagged images

(for free!) Near-live acquisition (e.g., for damage assessment)

Limited spatial coverage Uneven resolution Occlusions

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 32: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Discussion and conclusion

The effectiveness, quality and accuracy of the AR result depends on many factors: Geo-data projection and rendering are demanding operations and should be

performed fast Estimation of the camera pose and position for achieving a precise match for

accurately overlay virtual information on the scene and for acquiring information on the landscape

High-speed wireless transfer of data (i.e., optimize network transfers)

Conclusion Hardware and software specification of consumer-level portable devices are

constantly improving The exploitation of portable devices for AR applications has just started We proposed a concept that exploiting AR technology for GRS applications Thanks to an accurate registration of the device to the world it is possible to

perform visualization of geo-data on the mobile device interactively exploitation of the mobiles’ camera as a ground-based imaging sensors

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]

Page 33: Augmented Reality · Augmented Reality Augmented Reality (AR) blends virtual and real scenes in both time and space 1950s 1966 2000 2008 2014 Nowadays, AR technology starts to run

Introduction Discussion and conclusion Morphological and Attribute Profiles Spectral/spatial analysis

from http://www.gls-tedesco-a-berlino.de/2065.html

Thanks for your attention!

https://venturi.fbk.eu/ http://tev.fbk.eu/marmota/

M. Dalla Mura, M. Zanin, C. Andreatta, P. Chippendale IEEE IGARSS’12 [email protected]