MULTIPLE DRONE CINEMATOGRAPHY · • A couple of Framing Shot Types deal with two or more subjects...
Transcript of MULTIPLE DRONE CINEMATOGRAPHY · • A couple of Framing Shot Types deal with two or more subjects...
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
MULTIPLE DRONE CINEMATOGRAPHYI. Mademlis1, I. Pitas1, A. Messina2
1 Aristotle University of Thessaloniki2 Rai - Centre for Research and Technological Innovation
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• H2020 ICT-26 2016
• Research and Innovation Actions on multiple-actor
systems
• 8 partners
• Univ. of Thessaloniki, University of Bristol, THALES, RAI,
Deutsche Welle, Istituto Superior Tecnico, Univ. of
Seville, Alerion
• 2017-2020
MULTIDRONE in a nutshell
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Improved multiple drone decisional autonomy, robustness and safety;
• Innovative, safe and fast multiple drone active perception and AV shooting
• Application and demonstration in three media production scenarios
Main Objectives
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• UAV cinematography is mainly derived from traditional
ground and aerial cinematography, but must also take
into account UAV-specific limitations, capabilities,
properties.
• A visual vocabulary of UAV cinematography can be
defined, consisting in:
• Shot types
• Combinations of framing shot types + UAV / camera
motion types.
© 2009, Jim Zuckerman
UAV Cinematography
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Composition principles:
• Central Composition.
• Rule of Thirds.
• Lighting rules.
• Depth-of-Field / Focus settings.
• Need to define a standardized UAV shot type taxonomy
UAV Cinematography
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
•
Actual drone footage and related articles / guidelines were used to this
end:
• A total of 8 framing (static) shot types and 26 UAV / camera motion types
suitable for UAV media production have been identified.
• Camera motion types were clustered into groups according to their
characteristics.
• Visually pleasing combinations of framing shot types and camera motion types
were identified.
UAV Cinematography
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Framing Shot Types are more or less those of traditional
cinematography.
• Most are defined based on the percentage of the video frame width /
height covered by the single target / subject.
FRAMING SHOT TYPEPercentage of frame width/height
covered by target
Extreme Long Shot (ELS) <5%
Very Long Shot (VLS) 5-20%
Long Shot (LS) 20-40%
Medium Shot (MS) 40-60%
Medium Close Up (MCU) 60-75%
Close Up (CU) >75%
Framing Shot Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• A couple of Framing Shot Types deal with two or more
subjects / targets:
• 2 Shot / 3 Shot: 2/3 subjects appear on frame, equally visible
(typically LS or MS).
• Over the Shoulder (OTS): Adapted from traditional
cinematography OTS. Main target fully visible, secondary target
visible at the video frame edge.
Framing Shot Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Example UAV shot types when shooting boat targets from the side.
Extreme Long Shot Long Shot
Medium Close Up Two Shot
Framing Shot Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• UAV / camera motion types can be considered as either
“scene-oriented” or “target-oriented”.
• Four groups of UAV / camera motion types were defined.
• Static shots (6). No UAV motion, target may or may not be
present:
• Static Shot (SS)
• Static Shot of Still Target (SSST)
• Static Shot of Moving Target (SSMT)
• Static Aerial Pan (SAP)
• Static Aerial Tilt (SAT).
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Dynamic shots (6). Moving UAV, no target:
• Moving Aerial Pan or Tilt (MAP, MAT)
• Pedestal / Elevator Shot (PS)
• Bird’s Eye Shot (BIRD)
• Moving Bird’s Eye Shot (MOVBIRD)
• Survey Shot (SURVEY)
• Fly-Through (FLYTHROUGH).
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Target Tracking shots (11). UAV motion depends on target
motion:
• Moving Aerial Pan with Moving Target (MAPMT)
• Moving Aerial Tilt with Moving Target (MATMT)
• Lateral Tracking Shot (LTS)
• Vertical Tracking Shot (VTS)
• Orbit (ORBIT)
• Fly-Over (FLYOVER)
• Fly-By (FLYBY).
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Target Tracking shots (continued)
• Chase/Follow Shot (CHASE)
• Descent (DESCENT)
• Descent-Over (DESCENTOVER)
• Ascent (ASCENT)
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Source: Youtube: "5 Drone Moves Every Flier
Should Know",
https://www.youtube.com/watch?v=1hz-
lkx4o6c
Lateral Tracking Shot (LTS)
• Camera stays focused on the
moving target.
• UAV flies sideways / in parallel to
the target, matching its speed.
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Source: Youtube: "Drone footage of Cycle-
Racing",
https://www.youtube.com/watch?v=vQ94R
4LC9ig
Chase/Follow (CHASE)
• Camera stays focused on the
moving target.
• UAV follows / leads the target
from behind / from the front,
matching its speed.
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Source: Youtube: "Drone Chasing
Horses",
https://www.youtube.com/watch?v=O0
uw8py9qmw
Orbit (ORBIT)
• Camera gimbal is slowly rotating,
so as to keep the still or moving
target properly framed.
• UAV circles around the target while
following its trajectory (if any).
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
UAV / Camera Motion Types
Static Aerial Tilt (SAT)
• UAV hovers.
• Camera gimbal rotates slowly
around the pitch axis in order to
capture the scene context.
Source: Youtube: "5 Drone Moves
Every Flier Should Know",
https://www.youtube.com/watch?v
=1hz-lkx4o6c
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
UAV / Camera Motion Types
Source: Youtube: "How to
film amazing aerials with
your drone |
DroneFilmSchool",
https://www.youtube.com/wat
ch?v=zmQcSdj7vbE
Moving Bird’s Eye Shot
(MOVBIRD)
• Camera remains stable facing
vertically down.
• UAV is slowly flying parallel to
the terrain with constant velocity.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
UAV / Camera Motion Types
Source: Youtube: "Manor
House Stables - Drone Film -
Horses",
https://www.youtube.com/wat
ch?v=56qOnxc28dw
copyright owner: M7 Aerial
Survey Shot
• Camera remains stable facing
ahead or backwards.
• UAV is slowly flying parallel to
the terrain with constant velocity.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Dynamic Target shots (3): a target exists, but UAV trajectory also depends on
other factors
• Constrained Lateral Tracking Shot (CONLTS)
• Pedestal/Elevator with Target (PST)
• Reveal Shot (REVEAL)Constrained Lateral Tracking Shot
(CONLTS)
• Camera remains stable, focused on the
moving target.
• UAV follows the target but it is
constrained to move onto a pre-defined
“flight plane” vertical to the ground plane.
• Useful in sports, e.g. football.
UAV / Camera Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Motion types involving 2 or more drones in orchestrated
motion can be also considered.
Multiple UAV Motion Types
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Target detection
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Target detection
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Target detection• Target/object examples: athletes, boats, biclycles.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Target detection
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection
• Single view object detection
• Deep learning (CNN) object detection.
• Light weight CNNs for object detection.
• Multiple view object detection.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection
• Object detection = classification + localization:
• Find what is in a picture as well as where it is.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection
• Input: an image.
• Output: bounding boxes containing depicted objects.
• Each image contains a different number of objects (outputs).
• Typical approach: train a specialized classifier and deploy
in sliding-window style to detect all object of that class.
• Very inefficient, quite ineffective.
• Goal: combine classification and localization into a single
architecture for multiple, multiclass object detection.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection with CNNs
• Deep Learning (DL) approach: train a classifier on, say,
1000 classes of ILSVRC.
• OverFeat (2013) was one of the first DL approaches to
object detection. Its convolutional method made multi-scale
sliding window efficient.
• Based on AlexNet architecture.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection with CNNs
Sermanet, Pierre, et al. "Overfeat: Integrated recognition, localization and detection using convolutional networks." International Conference on Learning
Representations (ICLR2014), CBLS, April 2014. 2014.
Overfeat: Object detection at
increasing image resolutions
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Object detection with CNNs
• Impact of Deep Learning.
• Pascal VOC
(object detection)
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Imaging for drone safety
• Human crowd detection for safe autonomous drones
• Emergency landing site detection.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd detection for safe autonomous drones
• Detect where crowd exists.
• Comply with legislation.
• Detect emergency landing points.
• Provide heatmaps of the estimated probability of crowd presence in
each location.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd detection for safe autonomous drones
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd detection for safe autonomous drones
• Limited previous efforts on crowd detection, using computer
vision techniques.
• Crowded scenes are considered in related research works
involving crowds, e.g.,:
• crowd understanding,
• crowd counting,
• human detection and tracking in crowds.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd detection for drone flight safety using CNNs
• In [1], a method utilizing Convolutional Neural Networks
(CNNs) for crowd detection is proposed.
• Two approaches:
• transforming a pre-trained CNN to a fast, fully-convolutional network,
• devising a two-loss-training model, enhancing the separability of the crowd
and non-crowd classes.
[1] Tzelepi, Maria, and Anastasios Tefas, "Human Crowd Detection for Drone Flight Safety Using Convolutional Neural Networks." in European
Signal Processing Conference (EUSIPCO), Kos, Greece, 2017.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd detection for drone flight safety using CNNs
• Provide lightweight models, as imposed by the computational
restrictions of the application.
• Effectively distinguish between crowded and non-crowded scenes.
• Provide crowd heatmaps to semantically enhance flight maps by
defining no-fly zones.
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd heatmaps
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Human crowd heatmaps
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
XML Schema
Production Planning Schema
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Conclusions
• MULTIDRONE is aiming at defining and developing
innovative multi-actor autonomous system for UAV-based
media production
• Many challenges, here we focused on two
• Drone cinematography principles for media production
planning
• Target detection for automated tracking using DL
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)