ComputerVisionLabSeoul National University

Keyframe-Based Real-Time Camera Tracking

Young Ki BAIK

Vision seminar : Mar. 5. 2010

Computer Vision Lab.Computer Vision Lab.

ComputerVisionLabSeoul National University

Reference

Keyframe-Based Real-Time Camera Tracking

- ICCV 2009- Zilong Dong, Guofeng Zhang, Jiaya Jia Hujun Bao

ComputerVisionLabSeoul National University

Outline

Introduction

System overview

Offline map building

Online camera tracking

Demonstration

Conclusion

Opitimal keyframe selection

Keyframe recognition using vocabulary tree

How to use multiple thread

ComputerVisionLabSeoul National University

What is the purpose?Real-time camera tracking

Known internal camera parameters The map of environment is given…

6-DOF camera pose estimation!!

ComputerVisionLabSeoul National University

System overviewReal-time camera tracking

Novel keyframe selection algorithm

How to build a map

How to estimate the camera pose

Offline mapping using the conventional MVG method

Online camera tracking

Efficient key frame recognition method

Parallel computing framework

ComputerVisionLabSeoul National University

Offline mapping stage

reference images

Image Acquisition Video data for the target environment

ComputerVisionLabSeoul National University

Offline mapping stage

Image Acquisition

Feature Extraction and Matching

SIFT descriptors

ComputerVisionLabSeoul National University

Offline mapping stage

Image Acquisition Conventional MVG

Feature Extraction and Matching

3D Map Building

ComputerVisionLabSeoul National University

Offline mapping stage

Keyframe set

Image Acquisition Keyframe selection

Feature Extraction and Matching

3D Map Building

Optimal Keyframe Selection For efficiency of map management

ComputerVisionLabSeoul National University

Offline mapping stage

Image AcquisitionVocabulary tree

Feature Extraction and Matching

3D Map Building

Optimal Keyframe Selection

Vocabulary Tree Construction

For fast keyframe selection

in tracking stage…

Contribution!!

ComputerVisionLabSeoul National University

Why keyframe?

Which one is good for a map?

To solve the scalability problem of the map

All references Keyframe

A small number of …

ComputerVisionLabSeoul National University

Optimal Keyframe Selection

select manually…

using heuristic method…

select optimal keyframe

Containing many salient features as possible

Reducing non-distinctiveness in matching

Features should be distributed evenly in keyframes

ComputerVisionLabSeoul National University

Optimal Keyframe Selection

Problem formulation… to find optimal

keyframes

Completeness Term Redundancy Term

weight

Saliency of features

Distribution of features

Distinctiveness

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

Definition

Reference images

Keyframes

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

Definition

Feature clusters

Reference image set

2D image position

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

Definition

Set of superior features

~Long-term surviving (or representative) features in reference

images

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

Saliency

~

The larger s(χ) means the higher saliency!

DoG map

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

Distribution

The smaller d (χ)→ Features are well distributed !!

Feature density

31x31 window

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionCompleteness Term

=3 : controls the sensitivity of feature density…: the superior feature set in the keyframe

set F

ComputerVisionLabSeoul National University

Optimal Keyframe SelectionRedundancy Term

NormalizationThe number of

keyframes which contain feature χ

Small Er(F) indicates low redundancy!

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

Let = 0Initialization

~

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Add Keyframe

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Add Keyframe

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Check

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Add Keyframe

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Add Keyframe

ComputerVisionLabSeoul National University

Keyframe selection

Algorithm

~

Check

ComputerVisionLabSeoul National University

Vocabulary Tree Construction

For fast keyframe recognition…

Root node

List

Branch = 8, Tree depth = 5

All FMean patch

All feature descriptor

ComputerVisionLabSeoul National University

Vocabulary Tree Construction

To build each node…

Root node

List

Clustering by K-means method

Branch = 8, Tree depth = 5

ComputerVisionLabSeoul National University

Vocabulary Tree Construction

To build each node…

Root node

List

Clustering by K-means method

List List

Computing mean patchBuilding keyframe list

Branch = 8, Tree depth = 5

ComputerVisionLabSeoul National University

Vocabulary Tree Construction

Finally…

Root node

List

Clustering by K-means method

List List

Computing mean patchBuilding keyframe list

List List List List

Branch = 8, Tree depth = 5

ComputerVisionLabSeoul National University

Online tracking stage

SIFT Feature Extraction

SIFT feature

Input image

ComputerVisionLabSeoul National University

Online tracking stage

SIFT Feature Extraction Vocabulary tree

Keyframe Recognition

ComputerVisionLabSeoul National University

Online tracking stage

SIFT Feature Extraction

Keyframe Recognition

Keyframe-based matching

Input image Selected F

ComputerVisionLabSeoul National University

Online tracking stage

SIFT Feature Extraction

Keyframe Recognition

Keyframe-based matching

Camera Pose Estimation

3-point algorithm

3-D map

2-Dpoint

ComputerVisionLabSeoul National University

Keyframe recognition

With SIFT patches of input image…

Root node

List List

List List List List

Matching input patch & mean

patches

Vocabulary tree Key frame voting memory

ComputerVisionLabSeoul National University

Keyframe recognition

With SIFT patches of input image…

Root node

List List

List List List List

Key frame voting memory

Move patch And Update voting memory

ComputerVisionLabSeoul National University

Keyframe recognition

With SIFT patches of input image…

Root node

List List

List List List List

Key frame voting memory

Select keyframe with largest voting value

ComputerVisionLabSeoul National University

Processing time

Process time per frame with a single thread…

ComputerVisionLabSeoul National University

Processing time

Using multiple-thread with pipeline

SIFT Feature

Extraction

Time axis

Select Keyframe

Keyframe

Matching

SIFT Feature

ExtractionPose

Estimation

Thread#1

Thread#2

Thread#3

Thread#4

Select Keyframe

Keyframe

Matching Pose Estimation

20 fps

First frame istime consuming!

ComputerVisionLabSeoul National University

Demonstration

ComputerVisionLabSeoul National University

Conclusion

Real-time camera tracking is presented !!

The optimal keyframe selection algorithm

Real-time tracking is presented!

Minimizing the energy formulation!

Fast keyframe recognition using vocabulary tree

Multiple thread with pipeline structure

LimitationIncremental updating of VT is not considered…

Proposed process is only available for known environment.

ComputerVisionLabSeoul National University

Q & A