National Institute of Advanced Industrial Science and TechnologyNational Institute of Advanced Industrial Science and Technology

Estimation of Photo-Shoot Location with a Mobile-Phone Camera Based on Virtualized-Reality

Environment Models

Jun NISHIDA12, Tomoya ISHIKAWA1, Koji MAKITA1, Masakatsu KOUROGI1, Jun YAMASHITA2,

Hideaki KUZUOKA2, Takeshi KURATA12

1Center for Service Research, AIST, Japan2University of Tsukuba, Japan

National Institute of Advanced Industrial Science and Technology

EBS, Post POS, MR Information Sharing• Real-virtual correspondences of products with POS

(Point-Of-Sales) systems– Facilitate modeling and designing the flow of the products by

not strongly relying on tacit knowledge.– Brought about drastic changes in retail, chain restaurant,

logistics, etc.– Realized EBS (Evidence-Based Service) to some extend.

• On the analogy…• One of the next key issues for service innovation How to Make better correspondence betweencustomers/employees/service processes/environmentand the computerized ones. Mixed-Reality Information Sharing technologies!

2

National Institute of Advanced Industrial Science and Technology

MR Information Sharing Technologies inHuman-Behavior Sensing and Visualization Research Team

3

National Institute of Advanced Industrial Science and Technology

Horizontal-Vertical Development ofMR Information Sharing Technologies

4

National Institute of Advanced Industrial Science and Technology

PDR（Pedestrian Dead-Reckoning）Estimates velocity vector, relative altitude, and actionsby measurements from waist-mounted sensor module.

Wearing sensor module on waist Easy to wear and maintain Easy to measure data for action recognition Relatively easily to apply for handheld

setting compared to shoe-mounted PDR based on Zero Velocity Updates (ZUPTs)

5

Sensor module・Accelerometers・Gyro-sensors・Magnetometers・Barometer

Handheld PDR

From PDR to PDRplus

National Institute of Advanced Industrial Science and Technology

Pedestrian Tracking System

Pedestrian dead-reckoning

RFIDreader

ActiveRFID tagID

IDRSSI

Acceleration /angular velocity

Video

3D model

Magneticvector

Magnetometer

Output of position/orientation

RF-based positioning

Surveillance camera

Atmospheric pressure

Barometer

TrajectorySensor/Data fusion(Particle filter)

Accelerometers/ gyro-sensors

Walking velocity

Position / Orientation

Trajectory matching/Velocity estimation

Absolute position

3D environment model

Velocity vector /Relative altitude /Action type

Sensor module

6

T. Ishikawa, M. Kourogi, T. Okuma, and T. Kurata, “Economic and Synergistic Pedestrian Tracking System for Indoor Environments”, In Proc. SoCPaR2009, pp.522-527, 2009.

National Institute of Advanced Industrial Science and Technology

Our goalEstimation of photo-shoot location with mobile-phone camera forInitialization and Drift Cancellation.

Pass (ground truth)Initialization

Drift Drift

Pass(estimated)Pass(estimated)

7

Cancellation Cancellation

National Institute of Advanced Industrial Science and Technology

8

Previous works~ Panorama-Based Annotation

G

Environmental mapA

B C D

E

A

B

C

F

Input frames

Position at whicha panorama is taken

PositionDirection

235 [deg]

5 [deg]

From theuser’s camera

Located Orientated

M. Kourogi, T. Kurata, et al., IWAR1999, ISWC2001

✔ A lot of panoramic images are needed for wide environments

National Institute of Advanced Industrial Science and Technology

Panorama-Based to Model Based

9

Input image Panoramic images

Input image Virtualized-Reality Environment Models

Matching

Matching

National Institute of Advanced Industrial Science and Technology

Design for coarse to fine localizationDatabase

Photo-realistic virtualized-reality environment models

Input Image

Fine estimation

SIFT

Edge

Result

Coarse estimation

SSD

ZNCC

yHistogram similarity

Coarse estim

ation result

10

National Institute of Advanced Industrial Science and Technology

Set up for experiments• Similarities for coarse estimation

– Histogram similarity in the HSV space (using H and S) – Sum of Squared Difference (SSD)– Zero-mean Normalized Cross-Correlation (ZNCC)

• Feature for fine estimation– Scale-Invariant Feature Transform (SIFT)

• Images for experiments– Real images (photos taken with an iPhone)– Generated images from a virtualized-reality model

11

National Institute of Advanced Industrial Science and Technology

Real images and generated images for experiments

Real images

Generated images

A B C

A B C

12

National Institute of Advanced Industrial Science and Technology

Image generation from a virtualized-reality model

g gPosition and orientation

used for image generation

Generated image from a virtualized-reality model

13

National Institute of Advanced Industrial Science and Technology

Generated image from a virtualized-reality model(generated by changing camera position)

14

National Institute of Advanced Industrial Science and Technology

Y

XSimilarity map (ZNCC) Real image taken with an

i-PhoneGenerated image from a virtualized-reality model

Similarity mapShooting position and orientation

of a real image

15

National Institute of Advanced Industrial Science and Technology

Experimental result : Histogram similarity in the HSV space (using H and S)Real images

Generated images Similarity of the nearest point is NOT high.

A B C

: Shooting position and orientation of a real image16

National Institute of Advanced Industrial Science and Technology

Experimental result : Similarity of Sum of Squared Difference (SSD)

Similarity of the nearest point is low.

B CReal images

Generated images

A

: Shooting position and orientation of a real image17

National Institute of Advanced Industrial Science and Technology

Experimental result : Similarity of Zero-mean Normalized Cross-Correlation (ZNCC)

・Similarity of the nearest point is comparatively high.

A B CReal images

Generated images

: Shooting position and orientation of a real image18

National Institute of Advanced Industrial Science and Technology

Generated image from a virtualized-reality model(generated by changing camera position)

19

National Institute of Advanced Industrial Science and Technology

Experimental result : Histogram similarity in the HSV space (using H and S)

-0.05

0

0.05

0.1

0.15

0.2

0.25

-100 -50 0 50 100

図5,Color 図6,Color 図7,ColorA B C

Difference of shooting orientations between real image and generated image (degree)

Similarity of the nearest point is NOT high.

Sim

ilarity

20

National Institute of Advanced Industrial Science and Technology

Experimental result : Similarity of Sum of Squared Difference (SSD)

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

-100 -50 0 50 100

図5,SSD 図6,SSD 図7,SSDA B C

Similarity of the nearest point is NOT high.

Difference of shooting orientations between real image and generated image (degree)

Sim

ilarity

21

National Institute of Advanced Industrial Science and Technology

Experimental result : Similarity of Zero-mean Normalized Cross-Correlation (ZNCC)

-0.3-0.2-0.1

00.10.20.30.40.5

-100 -50 0 50 100

図5,ZNCC 図6,ZNCC 図7,ZNCCA B C

Similarity of the nearest point is comparatively high.

Difference of shooting orientations between real image and generated image (degree)

Sim

ilarity

22

National Institute of Advanced Industrial Science and Technology

Sample result of SIFT matching

Real image Real image

Generated image(Almost same position)

Generated image(Different position)

23

National Institute of Advanced Industrial Science and Technology

Conclusion of coarse estimation

• HSV : NOT Appropriate– Similarity of the nearest point is NOT high.

• SSD : NOT Appropriate– Similarity of the nearest point is NOT high.

• ZNCC : Appropriate– Similarity of the nearest point is local maximal.

24

National Institute of Advanced Industrial Science and Technology

Future works

• More Experiments– in wide environments– with changing position and orientation in same time

• Observational study for fine estimation– More study with SIFT– Study with edge based method

25

National Institute of Advanced Industrial Science and Technology

Appendix 1 : Definitions of similarities

∗ ∗

∑ ∑ , ,

∑ ∑ , ∗∑ ∑ ,

∑ ∑ , ∑ ∑ ,

∑ ∑ , ∑ ∑ ,

26

National Institute of Advanced Industrial Science and Technology27

Appendix 2 : Similarity maps of ZNCC before and after the earthquake

Before

After

A

National Institute of Advanced Industrial Science and Technology28

Appendix 2 : Similarity maps of ZNCC before and after the earthquake

Before

After

B

National Institute of Advanced Industrial Science and Technology29

Appendix 2 : Similarity maps of ZNCC before and after the earthquake

Before

After

C