1

EnviroInfo Conference 2017

Disaster Management for Resilience

and Public Safety Workshop

Disaster Monitoring using UAV and Deep Learning

Andreas Kamilaris

13th September, 2017

Luxembourg

Problem

2

Monitoring and identification of disasters are crucial

for mitigating their effects on the environment and

on human population.

Motivation

3

Disaster monitoring can be facilitated by the use of

unmanned aerial vehicles (UAV), equipped with

camera sensors which can produce frequent aerial

photos of the areas of interest.

Motivation

4

Advantages of Drones:

• Small size

• Low cost of operation

• Exposure to dangerous environments

• High probability of mission success

• No risk of loss of aircrew resource

• High-resolution image sensing

• High operational flexibility

Motivation

5

Modern computer vision techniques:

• Artificial Neural Networks

• Scalable Vector Machines

• Multi-layer Perceptrons

• Random Forests

• Gaussian Mixture Models

• K-Nearest Neighbors

• Unsupervised feature learning

• Feature extraction techniques: Color, shape, texture

• Deep learning

Machine Learning-

based Approaches

Probabilistic

Modelling

Motivation

6

Advantages of Deep learning:

• Superior performance in terms of precision

• Perform classification and predictions particularly

well due to their structure.

• Flexible and adaptable

• No need for hand-engineered features

• Generalizes well

• Robust in low-resolution and -quality images.

Andreas Kamilaris and Francesc X. Prenafeta-Boldú, Deep Learning in Agriculture: A

Survey, Computers and Electronics in Agriculture Journal, 2017. [Under review]

Research Questions

7

Can drones and aerial image sensing be used for

real-time monitoring of physical areas and?

accurate identification of disasters?

Can deep learning be used in combination with

drones and aerial images for real-time disaster

monitoring/identification?

Deep Learning

8

Convolutional Neural Networks

Deep Learning

9

Convolutional Neural Networks

• Can be applied to any form of data, such as audio,

video, images, speech, and natural language.

• Various “successful” popular architectures: AlexNet,

VGG, GoogleNet, Inception-ResNet etc.

• Pre-trained weights

• Common datasets for pre-training CNN architectures

include ImageNet and PASCAL VOC.

• Many tools and platforms that allow researchers to

experiment with deep learning e.g. Keras, Theano.

General Idea

10Disaster!Nothing to

worry about!

State of the Art

11

No. Disaster Image source Accuracy

1.Fire (Kim, Lee, Park, Lee, &

Lee, 2016)Aerial photos

Human-like

judgement

2.Avalanche (Bejiga, Zeggada,

Nouffidj, & Melgani, 2017)Aerial photos 72-97%

3.Car accidents and fire (Kang

& Choo, 2016)CCTV cameras 96-99%

4. Landslides (Liu & Wu, 2016)Optical remote

sensing96%

5.

Landslides and flood (Amit,

Shiraishi, Inoshita, & Aoki,

2016)

Optical remote

sensing80-90%

Methodology

12

CNN Model: VGG architecture, pre-trained with the

ImageNet dataset of images.

Dataset: 544 aerial photos from Google images (min.

256x256 pixels), acquired using the query:

[Disaster]: earthquake, hurricanes, flood and fire.

[Landscape]: aerial views of cities, villages, forests and

rivers

[Disaster | Landscape] + "aerial view" + "drone"

Dataset

13

No. Image GroupNo. of

Images

Relevant Possible

Disaster

1. Buildings collapsed 101Earthquakes and

hurricanes

2. Flames or smoke 111 Fire

3. Flood 125

Earthquakes,

hurricanes and

tsunami

4. Forests and rivers 104 No Disaster

5. Cities and urban landscapes 103 No Disaster

Dataset: Disasters

14

Buildings collapsed

Flames or smoke

Flood

Dataset: Landscapes

15

Forests and rivers

Cities and urban landscapes

Setup

16

• 82% (444 images) of our dataset as training data

and 18% (100 images) as testing data.

• Random assignment of images in training/testing.

• Training procedure 20 minutes on a Linux

machine, testing 5 minutes for the 100 images.

• Learning rate: 0.001

• Used data augmentation techniques.

• 30 epochs

Results: Training Vs. Testing

17

83

84

85

86

87

88

89

90

91

92

82-18 70-30 75-25 85-15 90-10

Training Vs. Testing Percentage

Overa

ll P

recis

ion (

%)

Results: Training Vs. Precision

18

0

10

20

30

40

50

60

70

80

90

100

5 10 15 20 25 30 35

Ove

rall

Pre

cis

ion

(%

)

Number of Epochs

Results: Confusion Matrix

19

91% Precision

9% Error

Results: Analysis of Error

20

9% Error

Urban Vs. Buildings collapsed (4%) Urban Vs. Fire (2%)

Urban Vs. Flooding (1%)Flooding Vs. Buildings collapsed (2%)

Conclusion

21

Deep learning offers good precision and many benefits.

Can be successfully used in combination with UAV for

disaster monitoring/identification.

It has also some disadvantages:

• It takes (sometimes much) longer time to train.

• It requires the preparation and pre-labeling of a

dataset containing at least some hundreds of images.

Future Work

22

• Publish the dataset to the research community.

• Enhance the dataset with more images.

• Experiment with different architectures, platforms and

parameters.

• Increase overall precision to more than 95%.

• Perform a real-life case study with drones used for

monitoring some particular disaster e.g. indication of

fire.

Vision

23

Better disaster modelling,

especially when combining UAV

and deep learning with geo-

tagging of the events identified

and geospatial applications.

Facilitate the integration of relevant actors (i.e. action

forces/authorities, citizens/volunteers, other stakeholders)

in disaster management activities with regard to

communication, coordination and collaboration.

24

Many thanks for your attention!

Andreas Kamilaris

andreas.kamilaris@irta.cat