EFFICIENT ROAD MAPPING VIA INTERACTIVE IMAGE SEGMENTATION

Presenter: Alexander Velizhev

CMRT’09ISPRS Workshop

O. Barinova, R. Shapovalov, S. Sudakov, A. Velizhev, A. Konushin

Introduction

• Roadway monitoring systems are widely-used for supervising road pavement surface and repair planning

Problem statement

• Analysis road pavement only by video sequences

Problem statement (2)

• Object types:– Lane marking – Road patches and defects

• Solution requirements:– High object detection rate – Maximum automation

Problem statement (3)Source image Expected result

Problem details

• Some real examples

Our algorithm outline1. Video rectification

2. Image preprocessing

3. Image segmentation

4. Features calculation

5. Interactive classification

Automaticofflinestage

Interactiveonlinestage

Video rectification

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

• Using of raw video has severe drawbacks:– Objects are represented with different

spatial resolution on the same frame– Projective distortions– Elongated objects exceed the bounds of

single frame

Video rectification (2)

Image preprocessing

Image segmentation

Features calculation

Interactive classification

• Video frames are converted to orthogonal projection and stitched to each other

Video rectification

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

Retinex transform

Contrast adjustment

Bilateral filter

Image preprocessing

Source image

Image segmentation

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

• Main goal is representing all objects of interest as different segments

• We use the hierarchical version of mean shift algorithm

Features calculation

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

• More than 100 various features are used for classification of segments

• Feature types:– Colour statistics (colour variance, Lab

components’ percentiles, ... )– Shape statistics (elongation, orientation,

area, …)– Difference with neighborhood of the

segments

Interactive classification

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

Interactive classification (2)

Image preprocessing

Image segmentation

Features calculation

Interactive classification

Video rectification

User manually marks object segments

Learning of cascade of classifiers

Automatic classification of the next road part

User corrects classification resultsEnd

Start

Cascade of classifiers

• Cascade of classifiers corresponds image segmentation levels

• We descend a hierarchy from large to small segments and reject segments that do not contain pixels of objects of interest

• Classifier training uses the data passed to a corresponding cascade layer by preceding version of cascade

Why do we use the cascade?

• To solve a problem of unbalanced classes

• To speed-up classification

Online learning

• We introduce an online version of the random forest algorithm

• Special class costing• The algorithm’s code is a part of our

open source “GML Balanced On-line Learning Toolkit ”– http://research.graphicon.ru/machine-learning/gml-

balanced-on-line-learning-toolkit-2.html

Why do we use online learning?

• We don’t need to store all training database in memory

• Short learning time• User actions immediately impact on

the classification results

How to measure system efficiency?• We are modeling “ideal” user actions

to measure the efficiency of the interactive classification

• Efficiency criterion:– a minimal number of mouse’s clicks for

making correct classification

Results

Source imageSegmented

imageAnalysis

result

Results (2)

Image part

Clicks

Manual classification

Interactive classification

Results (3)

Image part

Error,%

Summary

• We present a tool for efficient interactive mapping of road defects and lane marking

• Intensive use of computer vision methods on different stages of our data processing workflow increases usability of the tool

Weak points

• Image segmentation errors can degrade classifier and true object bounds cannot be extracted

• Algorithm is not robust to user mistakes

Future work

• Ultimate goal:Development of the universal semantic segmentation system which can be used for object extraction from large class of images

• Nearest plan:Improving the quality of image segmentation by integration colour and range data

CMRT’09ISPRS Workshop

Efficient road mapping via interactive image segmentation

O. Barinovaobarinova@graphics.cs.msu.ru

R. Shapovalovshapovalov@graphics.cs.msu.ru

S. Sudakovssudakov@graphics.cs.msu.ru

A. Konushinktosh@graphics.cs.msu.ru

A. Velizhevavelizhev@graphics.cs.msu.ru