Estimation of Photo-Shoot Location with a Mobile...

National Institute of Advanced Industrial Science and TechnologyNational Institute of Advanced Industrial Science and Technology

Estimation of Photo-Shoot Location with a Mobile-Phone Camera Based on Virtualized-Reality

Environment Models

Jun NISHIDA12, Tomoya ISHIKAWA1, Koji MAKITA1, Masakatsu KOUROGI1, Jun YAMASHITA2,

Hideaki KUZUOKA2, Takeshi KURATA12

1Center for Service Research, AIST, Japan2University of Tsukuba, Japan

National Institute of Advanced Industrial Science and Technology

EBS, Post POS, MR Information Sharing• Real-virtual correspondences of products with POS

(Point-Of-Sales) systems– Facilitate modeling and designing the flow of the products by

not strongly relying on tacit knowledge.– Brought about drastic changes in retail, chain restaurant,

logistics, etc.– Realized EBS (Evidence-Based Service) to some extend.

• On the analogy…• One of the next key issues for service innovation How to Make better correspondence betweencustomers/employees/service processes/environmentand the computerized ones. Mixed-Reality Information Sharing technologies!

2


MR Information Sharing Technologies inHuman-Behavior Sensing and Visualization Research Team

3


Horizontal-Vertical Development ofMR Information Sharing Technologies

4


PDR（Pedestrian Dead-Reckoning）Estimates velocity vector, relative altitude, and actionsby measurements from waist-mounted sensor module.

Wearing sensor module on waist Easy to wear and maintain Easy to measure data for action recognition Relatively easily to apply for handheld

setting compared to shoe-mounted PDR based on Zero Velocity Updates (ZUPTs)

5

Sensor module・Accelerometers・Gyro-sensors・Magnetometers・Barometer

Handheld PDR

From PDR to PDRplus


Pedestrian Tracking System

Pedestrian dead-reckoning

RFIDreader

ActiveRFID tagID

IDRSSI

Acceleration /angular velocity

Video

3D model

Magneticvector

Magnetometer

Output of position/orientation

RF-based positioning

Surveillance camera

Atmospheric pressure

Barometer

TrajectorySensor/Data fusion(Particle filter)

Accelerometers/ gyro-sensors

Walking velocity

Position / Orientation

Trajectory matching/Velocity estimation

Absolute position

3D environment model

Velocity vector /Relative altitude /Action type

Sensor module

6

T. Ishikawa, M. Kourogi, T. Okuma, and T. Kurata, “Economic and Synergistic Pedestrian Tracking System for Indoor Environments”, In Proc. SoCPaR2009, pp.522-527, 2009.


Our goalEstimation of photo-shoot location with mobile-phone camera forInitialization and Drift Cancellation.

Pass (ground truth)Initialization

Drift Drift

Pass(estimated)Pass(estimated)

7

Cancellation Cancellation


8

Previous works~ Panorama-Based Annotation

G

Environmental mapA

B C D

E

A

B

C

F

Input frames

Position at whicha panorama is taken

PositionDirection

235 [deg]

5 [deg]

From theuser’s camera

Located Orientated

M. Kourogi, T. Kurata, et al., IWAR1999, ISWC2001

✔ A lot of panoramic images are needed for wide environments


Panorama-Based to Model Based

9

Input image Panoramic images

Input image Virtualized-Reality Environment Models

Matching

Matching


Design for coarse to fine localizationDatabase

Photo-realistic virtualized-reality environment models

Input Image

Fine estimation

SIFT

Edge

Result

Coarse estimation

SSD

ZNCC

yHistogram similarity

Coarse estim

ation result

10


Set up for experiments• Similarities for coarse estimation

– Histogram similarity in the HSV space (using H and S) – Sum of Squared Difference (SSD)– Zero-mean Normalized Cross-Correlation (ZNCC)

• Feature for fine estimation– Scale-Invariant Feature Transform (SIFT)

• Images for experiments– Real images (photos taken with an iPhone)– Generated images from a virtualized-reality model

11


Real images and generated images for experiments

Real images

Generated images

A B C

A B C

12


Image generation from a virtualized-reality model

g gPosition and orientation

used for image generation

Generated image from a virtualized-reality model

13


Generated image from a virtualized-reality model(generated by changing camera position)

14


Y

XSimilarity map (ZNCC) Real image taken with an

i-PhoneGenerated image from a virtualized-reality model

Similarity mapShooting position and orientation

of a real image

15


Experimental result : Histogram similarity in the HSV space (using H and S)Real images

Generated images Similarity of the nearest point is NOT high.

A B C

: Shooting position and orientation of a real image16


Experimental result : Similarity of Sum of Squared Difference (SSD)

Similarity of the nearest point is low.

B CReal images

Generated images

A



Experimental result : Similarity of Zero-mean Normalized Cross-Correlation (ZNCC)

・Similarity of the nearest point is comparatively high.

A B CReal images

Generated images



Generated image from a virtualized-reality model(generated by changing camera position)

19


Experimental result : Histogram similarity in the HSV space (using H and S)

-0.05

0

0.05

0.1

0.15

0.2

0.25

-100 -50 0 50 100

図5,Color 図6,Color 図7,ColorA B C

Difference of shooting orientations between real image and generated image (degree)

Similarity of the nearest point is NOT high.

Sim

ilarity

20


Experimental result : Similarity of Sum of Squared Difference (SSD)

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

-100 -50 0 50 100

図5,SSD 図6,SSD 図7,SSDA B C

Similarity of the nearest point is NOT high.


Sim

ilarity

21


Experimental result : Similarity of Zero-mean Normalized Cross-Correlation (ZNCC)

-0.3-0.2-0.1

00.10.20.30.40.5

-100 -50 0 50 100

図5,ZNCC 図6,ZNCC 図7,ZNCCA B C

Similarity of the nearest point is comparatively high.


Sim

ilarity

22


Sample result of SIFT matching

Real image Real image

Generated image(Almost same position)

Generated image(Different position)

23


Conclusion of coarse estimation

• HSV : NOT Appropriate– Similarity of the nearest point is NOT high.

• SSD : NOT Appropriate– Similarity of the nearest point is NOT high.

• ZNCC : Appropriate– Similarity of the nearest point is local maximal.

24


Future works

• More Experiments– in wide environments– with changing position and orientation in same time

• Observational study for fine estimation– More study with SIFT– Study with edge based method

25


Appendix 1 : Definitions of similarities

∗ ∗

∑ ∑ , ,

∑ ∑ , ∗∑ ∑ ,

∑ ∑ , ∑ ∑ ,

∑ ∑ , ∑ ∑ ,

26

National Institute of Advanced Industrial Science and Technology27

Appendix 2 : Similarity maps of ZNCC before and after the earthquake

Before

After

A



Before

After

B



Before

After

C

Estimation of Photo-Shoot Location with a Mobile...

Documents

Transcript of Estimation of Photo-Shoot Location with a Mobile...