Research Area B

Leif Kobbelt

Communication

System

Interface

Research Area B

2

Research Area B

3

Interface

System

Communication

A

C

D

• definition of „application“ within UMIC• find new and relevant applications (killer app)

• combine existing technology

• identify application profiles

• types of data

• amount of data

• latency requirements

• input / output devices

• golden demo• good : convey the message

• bad : no basis research, commercial competitors

4

Research Area B

Future Mobile Applications

5

fund

amen

tal

algor

ithm

s syste

m

desig

n

evalu

ation

com

mer

cializ

ation

Future Mobile Applications

6

fundam

enta

l

algorit

hms

syst

em

desig

n

eval

uatio

n

com

mer

cializ

ation

Future Mobile Applications

7

fundam

enta

l

algorit

hms

syst

em

desig

n

eval

uatio

n

com

mer

cializ

ation

communication

it security

computer graphics

computer vision

Future Mobile Applications

8

fundam

enta

l

algorit

hms

syst

em

desig

n

eval

uatio

n

com

mer

cializ

ation

software engineering

security

computer graphics & vision

interface design

Future Mobile Applications

9

fundam

enta

l

algorit

hms

syst

em

desig

n

eval

uatio

n

com

mer

cializ

ation

graphical UIs

prototypes

user studies

...

Visual Computing for Future Mobile Applications

Bastian Leibe

Target Scenario: Pedestrian Navigation

11

Aachen Cathedral

Mobile visual search Simply point the camera to any object/building of interest. Images are transmitted to a central server for recognition.

Target Scenario: Pedestrian Navigation

12

Aachen Cathedral

Mobile visual search Simply point the camera to any object/building of interest. Images are transmitted to a central server for recognition. Object-specific content is sent back to for visualization on the

mobile phone (mobile AR).

LocalizeMe Demo

13

P. Steingrube, T. Weyand, T. Sattler, A. Schmitz, B. Leibe, L. Kobbelt

Mobile Service Structure

14

Localization Service

Image Database

3D Model

Render Server

Information Service

Cultural Database

Internet

Compound

Application

MobileClient

ServerServerServer

User Interface

Localization: Large-Scale Image Matching

How can we perform this matching step efficiently?

15

Database with thousands (millions) of images

??

Mobile photo

Localization: Large-Scale Image Matching

16

Database with thousands (millions) of images

Mobile photo

Local features(~1000 per image) …

“Visual vocabulary”(~1M feature clusters)

Shortlist of candidate matches(~100 images)

Shortlist of candidate matches(~100 images)

Matching (nearest neighbors in 128D space)

Localization: Geometric Verification

17

Mobile photo (for each image in shortlist)

xAj

xBj

XjAssumption: corresponding 3D structure

T. Sattler, B. Leibe, L. Kobbelt, SCRAMSAC: Improving RANSAC’s Efficiency with a Spatial Consistency Filter. International Conference on Computer Vision, 2009.

Find a rigid geometric transformation to verify that the matched features correspond to the same 3D structure. Problem: efficient processing with many outliers.

Summary: Visualization Service

18

Image

Local Feature

Extractor

Local Feature

Database

Image Database

Feature Matching

ImageImageCandidate Matches

Geometric Verificatio

n

DetermineLocation

On mobile device

On server side

Mobile Service Structure

19

Localization Service

Image Database

3D Model

Render Server

Information Service

Cultural Database

Internet

Compound

Application

MobileClient

ServerServerServer

User Interface

World-Scale Mining for Content Creation

20

e.g. Wikipedia match

Mining geotagged images Extracted Image clusters

Automaticannotation

& verification

Frequent tags

Example: Automatic Landmark Detection

21

Matched images for Aachen city hall (subset)

T. Weyand, B. Leibe

How Does This Scale?

Feasibility study Pairwise matching on 500,000 geotagged images of Paris How many matching images can we find at a certain location?

Touristic sites and central roads are well-covered.

22

T. Weyand, B. Leibe

Mobile Service Structure

23

Localization Service

Image Database

3D Model

Render Server

Information Service

Cultural Database

Internet

Compound

Application

MobileClient

ServerServerServer

User Interface

Virtual City Model

24

Floor Plan Map (2D)

Street Graph (2D)

Height Field(3D)

Synthetic Textures

Photographic

Textures

Building Model

Landscape Model

Optimized Octree Data

Structure

Estate Plan(2D)

Results: Virtual Aachen Model

25

G. Fabritius, J. Kraßnigg, L. Krecklau, C. Manthei, A. Hornung, M. Habbecke, L. Kobbelt

Alternative 1: Mobile Rendering

26

Rendering quality ona regular PC

C. Schreder, A. Schmitz, L. Kobbelt

Mobile rendering Very limited memory! Need to precompute

octree structure Dynamic transmission

of geometry & textures

Alternative 2: Stream Rendering

27

Rendering on server

Internet

Current prototype Transmit images in

UDP packages Tradeoff: compression

vs. framerate

Mobile Service Structure

28

Localization Service

Image Database

3D Model

Render Server

Information Service

Cultural Database

Internet

Compound

Application

MobileClient

ServerServerServer

User Interface

Application Study: Looking Through Time

How did Aachen’s cathedral look in past centuries? Camera phone as a “magic lens” to reveal past building states How should the interface be designed for such an application? Mock-up prototype (using Motion Capture system for tracking) Evaluation in user studies

29

T. Palm, J. Borchers

Research in Novel Interaction Techniques

30

T. Karrer, M. Weiss, M. Wittenhagen, G. Herkenrath, J. Borchers

TWEND Twisting and bending as new

interaction gestures in futuremobile devices

E.g. for interaction with anelectronic map or an e-book.

PocketDRAGON Directly drag objects along their

movement trajectory to preciselynavigate to a specific point in avideo sequence

Mobile implementation madepossible through communicationwith server