How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 1 / 25

IJCNN 2017

Anchorage, AK, USA

How to Get Pavement Distress Detection Ready for Deep Learning? A Systematic Approach Markus Eisenbach

Ilmenau University of Technology (Germany)

Neuroinformatics and Cognitive Robotics Lab markus.eisenbach@tu-ilmenau.de www.tu-ilmenau.de/neurob

R. Stricker, D. Seichter, K. Amende, K. Debes, H.-M. Gross

Ilmenau University of Technology

Neuroinformatics and Cognitive Robotics Lab

Germany

M. Sesselmann,

D. Ebersbach

LEHMANN + PARTNER GmbH

Germany

U. Stoeckert

German Federal Road Research Institute (BASt)

Germany

ASINVOS Funded by BMBF under grant agr. no. 01IS15036

Project

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 2 / 25

IJCNN 2017

Anchorage, AK, USA

Motivation Why do we need Deep Learning for road condition assessment?

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 3 / 25

IJCNN 2017

Anchorage, AK, USA

What do you want?

Safe roads of course!

But this implies...

But this is the last thing I want

What do you want?

Safe roads of course!

But this implies...

But this is the last thing I want

If you would ask a driver...

Condition assessment necessary

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 4 / 25

IJCNN 2017

Anchorage, AK, USA

Is this road in good condition?

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 5 / 25

IJCNN 2017

Anchorage, AK, USA

Is this road in good condition?

Easy recognition due to focusing

No need to have a perfect detection system

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 6 / 25

IJCNN 2017

Anchorage, AK, USA

4 year cycle

13,000 km + 40,000 km 8,000 mi 25,000 mi

German road system

~ 7,500 km 4,700 mi

= 30,000 lane-km per year

(2,000 HD-images per lane-km)

? ? 18,600 lane-mi

= 30,000 lane-km per year

(2,000 HD-images per lane-km)

? ? 18,600 lane-mi

4 year cycle

~ 7,500 km 4,700 mi

13,000 km + 40,000 km 8,000 mi 25,000 mi

German road system

How many roads to analyze?

Analyze 60 M images / year

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 7 / 25

IJCNN 2017

Anchorage, AK, USA

How is it done at the moment?

manual labor lots of

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 8 / 25

IJCNN 2017

Anchorage, AK, USA

Outline What can you expect?

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 9 / 25

IJCNN 2017

Anchorage, AK, USA

Outline – What can you expect?

What do you need to train a useful distress detection systems?

GAPs dataset

Goal: Assist operators by Deep Learning

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 10 / 25

IJCNN 2017

Anchorage, AK, USA

Outline – What can you expect?

How good are the trained classifiers in crack detection?

Deep Learning techniques

First results

Goal: Assist operators by Deep Learning

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 11 / 25

IJCNN 2017

Anchorage, AK, USA

GAPs German Asphalt Pavement

distress dataset

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 12 / 25

IJCNN 2017

Anchorage, AK, USA

Standardized Data Acquisition Certified by the German Federal Highway Research Institute (BASt)

Standardized Data Acquisition Certified by the German Federal Highway Research Institute (BASt)

German Asphalt Pavement distress dataset (GAPs)

RMA*-specified labeling (coarse for ~1x1m cells)

and detailed labeling (tightly enclosing Bounding Boxes)

* RMA ... Road Monitoring and Assessment

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 13 / 25

IJCNN 2017

Anchorage, AK, USA

German Asphalt Pavement distress dataset (GAPs)

5 distress types (of 6 in FGSV-regulation*)

Crack 83%

Pothole 9%

Inlaid patch 2%

Applied patch 2%

Open joint 4%

* 6th distress type: Bleeding (was not present in data)

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 14 / 25

IJCNN 2017

Anchorage, AK, USA

1m Lane

3 German federal roads 1969 HD-images (8 bit) labeled

High resolution 1 Pixel = 1.2 mm x 1.2 mm

1m Lane

3 German federal roads 1969 HD-images (8 bit) labeled

High resolution 1 Pixel = 1.2 mm x 1.2 mm

German Asphalt Pavement distress dataset (GAPs)

Patches extracted 64x64 pixels ~ 8 cm x 8 cm (3.2 in x 3.2 in)

4 M intact road (subsampled)

1.3 M distress (mainly from )

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 15 / 25

IJCNN 2017

Anchorage, AK, USA

Installation (90GB!!):

pip install gaps-dataset

Python (MNIST-like import):

>>> from gaps_dataset import gaps

>>> gaps.download(login=‘your_login‘)

>>> x, y = load_chunk(0, ‘train‘)

Installation (90GB!!):

pip install gaps-dataset

Python (MNIST-like import):

>>> from gaps_dataset import gaps

>>> gaps.download(login=‘your_login‘)

>>> x, y = load_chunk(0, ‘train‘)

Website: (Version 2.0 will appear soon: 50k dataset, patch sizes)

https://www.tu-ilmenau.de/neurob/data-sets-code/gaps/

German Asphalt Pavement distress dataset (GAPs)

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 16 / 25

IJCNN 2017

Anchorage, AK, USA

Experiments How good are the trained

classifiers in crack detection?

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 17 / 25

IJCNN 2017

Anchorage, AK, USA

Detection of JPEG alignment

How hard is distress detection?

Detection of distress

vs

Is this patch aligned with a JPEG block?

Does this patch contain a crack?

Easier for

machines

Easier for

humans

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 18 / 25

IJCNN 2017

Anchorage, AK, USA

How did the Deep Learning approaches perform? Visual results

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 19 / 25

IJCNN 2017

Anchorage, AK, USA

How did the Deep Learning approaches perform? Visual results

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 20 / 25

IJCNN 2017

Anchorage, AK, USA

Performance Evaluation

Results in numbers

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 21 / 25

IJCNN 2017

Anchorage, AK, USA

8 Conv layer, 3 Max-pooling layer, 3 Fully connected layers 4 M weights

VGG-like

vs labeled distress

intact

3x

Which nets did we use?

8 Conv layer, 3 Max-pooling layer, 3 Fully connected layers 4 M weights

3x 3x VGG-like

For comparison: RCD* net (4x Conv, 4x Pool, 2x FC) 300k weights

* RCD ... Road Crack Detection

AlexNet-like

vs labeled distress

intact

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 22 / 25

IJCNN 2017

Anchorage, AK, USA

Features +

Boosting

0,3121

Machine Learning

CrackIT

0,4882

Compu. Vision

Features +

Boosting

0,3121

Machine Learning

CrackIT

0,4882

Compu. Vision

RCD 99x99

RCD 64x64

0,7184 0,6642

Deep Learning

0,7246

ASINVOS 64x64

Deep Learning outperforms classical approaches

F1-Score

Results in numbers

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 23 / 25

IJCNN 2017

Anchorage, AK, USA

Evaluation of regularization

Dropout

Batch normalization

Batch normalization + Dropout

Weight decay + Dropout

Max-norm regularization + Dropout

None ...not in the paper:

Adversarial Training

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 24 / 25

IJCNN 2017

Anchorage, AK, USA

Conclusion

How to Get Pavement Distress Detection Ready for Deep Learning?

M. Eisenbach et al. – Ilmenau University of Technology, Neuroinformatics & Cognitive Robotics Lab 25 / 25

IJCNN 2017

Anchorage, AK, USA

Conclusion How to Get Pavement Distress Detection Ready for Deep Learning? A Systematic Approach

Demand for Deep Learning assisted distress detection

GAPs dataset Regularization, Performance

D BN A