ReLive

Drone Mission Review in Virtual Reality

Ricardo Daniel Ribeiro Mota

Thesis to obtain the Master of Science Degree in

Information Systems and Computer Engineering

Supervisors: Prof. Alfredo Manuel Dos Santos Ferreira JuniorProf. Rui Miguel Carrasqueiro Henriques

Examination Committee

Chairperson: Prof. Francisco Antonio Chaves Saraiva de MeloSupervisor: Prof. Alfredo Manuel Dos Santos Ferreira Junior

Member of the Committee: Prof. Joao Antonio Madeiras Pereira

May 2019

Acknowledgments

I want to thank my parents for supporting me throughout my life and these last few years through

college not out of obligation but willingness to help and allowing me to become the best person I can be.

To my friends, thank you for being there for me even in the most stressful of times, but also with

helping me focus on what’s important. If I ever need to unwind or go on a rant I know I can count on

them.

Abstract

Drones are becoming more common and widespread, with this the need to review their actions also

increases as problems inevitably occur during their operation. These problems could be caused by a

malfunction in the unit, human operator error or even caused by harsh environments. As such, there is

a growing necessity of having a system that allows the users to investigate the root cause of the fail and

help to correct it.

With a review system it’s possible to get an overview of what the drone did and replicate it. We can

also show information captured by the drone’s sensors at a specific moment in time to better understand

what was happening.

In this thesis we research, prepare the building blocks and develop a post mission review system

applied to flying drones with focus on the user experience and interface, to better assess what hap-

pened and when, throughout the mission. The opportunity to create a new replay system allows us to

experiment with a review system in an immersive virtual reality environment, that is not restricted by the

small size of current monitors that clump together information in a single screen. In a virtual world we

can overcome this due to the near unlimited space there is to organize our data and virtual screens with

no need to spend time or money upgrading our real office space and monitors.

Keywords

Mission Review, Drone, Virtual Reality, Interface

iii

Resumo

Os drones estao a tornar-se cada vez mais comuns, com isto a necessidade de rever as suas acoes

aumenta pois problemas irao inevitavelmente acontecer durante a sua operacao. Estes problemas

podem ser causados por uma avaria do drone, erro humano ou mesmo por causas naturais. Como

tal, ha uma necessidade crescente de ter um sistema que permita os utilizadores investigar a causa

principal que levou a falha e ajudar a corrigi-la.

Com um sistema de revisao e possıvel ter uma visao geral das acoes que o drone fez durante a

missao e replica-las. Tambem e possıvel mostrar a informacao capturada pelos sensores do drone num

momento especıfico para perceber melhor o que estava a acontecer.

Nesta tese foi investigado e desenvolvido um sistema de revisao pos-missao aplicado a drones

voadores com enfase na interface e experiencia do utilizador, para melhor aferir o que aconteceu e

quando, durante a missao.

A oportunidade de criar um sistema de revisao permite-nos experimentar com um sistema num

ambiente imersivo de realidade virtual, que nao esta restringido pelo pequeno tamanho dos monitores

habituais, que agrupam informacao num unico ecra, reduzindo a sua visibilidade. Num mundo virtual,

podemos ultrapassar este limite devido ao espaco quase ilimitado que existe para organizar os nossos

dados e ecras virtuais, sem necessidade de perder tempo ou dinheiro a comprar monitores maiores ou

expandir o espaco do escritorio.

Palavras Chave

Revisao de Missao, Drone, Realidade Virtual, Interface

v

Contents

1 Introduction 2

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Document Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Related Work 9

2.1 Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Drone Control and Mission Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3 Proposed Solution 23

3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Implementation 29

4.1 Replay System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.1 Video Player . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.2 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1.3 UI Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1.4 Data Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.1.5 Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.1.6 Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.1.7 Data Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.1.8 Data Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2 Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2.1 Hangs and Crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2.2 Extra Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.3 Unity Duplicate Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.4 The Update Dilemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

vii

5 Evaluation 45

5.1 Technical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.1.1 Additional Screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.1.2 Additional Data Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.1.3 Data Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6 Conclusion 53

6.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

viii

List of Figures

1.1 Tekever drone monitoring application (work in progress) . . . . . . . . . . . . . . . . . . . 4

1.2 ReLive virtual reality environment concept (work in progress) . . . . . . . . . . . . . . . . 6

2.1 Dota 2 VR Hub spectator mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Remote tower immersive environment concept [1] . . . . . . . . . . . . . . . . . . . . . . 12

2.3 Virtual Command Presence six monitor workspace [2] . . . . . . . . . . . . . . . . . . . . 13

2.4 Bright and dark conditions in physical environment (left) vs virtual environment (right). . . 13

2.5 Virtual lumbar trainer [3] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.6 Exploring historic buildings (2001) [4] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.7 MIT campus 3D model in Unity3D loaded with geotagged tweets [5] . . . . . . . . . . . . 15

2.8 Single tweet interaction [5] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.9 Operator at the helm of a virtual ship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.10 Command Center of the Future developed by Ideum [6] . . . . . . . . . . . . . . . . . . . 17

2.11 Drone taking images at 60 meters (left), pilot and experts watching the live video (right) [7] 17

2.12 Interruption Assistance Interface [8] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.13 Event timeline showing event bookmarks [8] . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.14 Miner simulation with replay option [9] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.15 (Left) Standard monitor solution, (center) CATEC’s VR screen, (right) Oculus Rift [10] . . 19

2.16 Quadcopter programmed mission using GPS, data overlay by Dashware . . . . . . . . . . 20

3.1 Project architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.2 ReLive module architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1 ReLive’s scene overview, with camera field of view changed. . . . . . . . . . . . . . . . . 31

4.2 ReLive’s video player with timeline shown below. . . . . . . . . . . . . . . . . . . . . . . . 32

4.3 Timeline closeup with event indicators. Symbols showing (left) and minimalistic view (right) 32

4.4 Mission List Component. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.5 Event List Component. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

ix

4.6 Different scene data elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.7 User inside the sphere mockup. Rays sent from the camera, hitting the inside of the sphere. 36

4.8 Map during a mission, with two warning events (in yellow), . . . . . . . . . . . . . . . . . 37

4.9 Interpolation scenario for altitude in meters (m). A is the previous value, C the next one

and B is at the current time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.10 Original object’s properties (left). Duplicated object’s properties (right) . . . . . . . . . . . 42

5.1 Data metric grid of interface elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

x

List of Tables

5.1 Video files used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.2 Performance stats of one vs two screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

1

1Introduction

Contents

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Document Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2

This thesis is focused on the replay mission interface of flying drones and the way you can review its

actions and behavior after it has returned from the mission. The new system that will be used to review

drone missions after they are concluded will be in a head mounted virtual reality (VR) environment, this

work was proposed by Tekever to research how post-mission review of their drone fleet would work in a

different technological environment.

Tekever is a Portuguese tech company that specializes in hardware and software needed to create

and develop manned or unmanned aerial vehicles and new technologies from the ground up.

Our colleague, Ricardo Carvalho, is also partnering with Tekever for his thesis, some of the early

work that consists in using the current replay system endpoints and library will certainly overlap between

our work, and we will help each other with early implementation details and bugs. His work is also about

drones and the replay system but he will be tackling a different problem, collaboration between different

people in a room using augmented reality (AR).

1.1 Motivation

While on a mission the drone collects many types of data like the drone location, speed, video from the

cameras, infrared data from the sensors, signals from ships, and many more that can also depend on

the drone model.

These flying drones are used to survey the coastline, forests, and borders in partnership with national

and European entities, the tasks performed by the drone are called missions, they can be as simple as

sending it to a set destination or follow a passing ship detected by the camera.

The information captured during the mission such as video from the cameras and data from the sen-

sors, can be reviewed afterwards to help drone operators revisit important moments, train new operators

and detect and debug possible future or past problems that might not have been noticed the first time.

Nowadays flying drones are becoming more common and widespread. With this proliferation, errors

during the usage will inevitably happen, these problems could have a more technical nature (a fault

with the unit itself), a human nature (the pilot can cause the fault) or a hazard nature (the environment

itself is an obstacle). As such, there is a growing necessity of having a system that allows the users to

investigate the root cause of the fail and help to correct it.

3

1.2 Problem

There needs to be a way to review the state of the drone at any time during the recorded mission

without losing context or information about that moment, it could be before or at the time of an incident

to determine what happened or went wrong. So a comprehensive replay system is needed that allows

us to move freely backwards and forwards in time to better understand the circumstances of the drone.

The systems of today are full of information from the many sensors but that can be overwhelming to

the user if he’s not sure from the start what he needs to look for in the replay, the user might not quickly

find what he wants to review and that could be a problem if a time sensitive issue shows up, the two

dimensional overview nature of the current system and its many graphs and parameters does not help

the user to intuitively find what he needs.

Tekever’s current mission replay system is viewed on a single screen and uses a standard graphical

user interface divided in four parts, showing the drone camera, a 2D map with the drone location, one

with evolving data plots that show a given parameter as time moves forward and a list of all possible

parameters that will be shown as data plots (see Figure 1.1).

This system uses different files for the video and metadata of the sensors, the data time precision

is not very accurate so it’s harder to properly review a mission but a revamp of the data gather and

encoding is currently underway to correct the synchronization issue.

Figure 1.1: Tekever drone monitoring application (work in progress)

Visual clutter is a big concern in these types of systems because the physical monitor is of a fixed

4

size and to show different types of data at the same time in the screen some opt to use a window that

needs to be divided into smaller sized windows or viewports reducing their size and making it harder to

analyze the data.

Others may choose to create smaller extra windows to display additional information, but it also

clutters the user work and visual space and makes it harder to manage different win dows without hiding

others accidentally or needing to rearrange them after a new one is created.

A possible solution to the visual clutter would be to buy as many monitors as necessary to accommo-

date the current workflow of a specific user, despite solving the visual clutter by spreading the windows

to different screens.

This option introduces more variables and possible problems, now each user of the replay system

will need extra monitors and that costs money, the number of monitors might not be the same depending

on the workflow of the person using it, and the number of monitors might increase as time goes on and

new options are added to the program or if the workflow changes. In addition to this it also takes time

for the monitors to arrive and be setup properly.

1.3 Goals

The underlying application will need a rework in order to become a new type of re play system. For this,

we will have to research and think of new ways to display the mission status and the program as a whole

to better help the operators in reviewing their missions, for instance by changing their environment in

virtual reality to better suit their needs or the mission.

Our goal with this project is to build a more intuitive and immersive interface for the drone mission

replay system, it will be a virtual reality 3D application where the user will be able to replay a mission that

has already finished with a timeline control that can go backward and forward in time, show the captured

camera feeds and needed metadata, follow the drone’s position in space, a terrain view with contextual

information that helps the user navigate and more easily detect anomalies that may have happened

during the mission.

This virtual reality application will use a head mounted display immersing the user in the virtual world

created by us that can be modified at will or use built in presets, it can be setup so the user feels more

comfortable while using it by changing the environment around them to one that’s more familiar or liked

by the user.

The main environment will consist of a control center where the user has a giant screen in front

of them to monitor the most important part of the drone mission, the camera feed. Relevant informa-

tion about objects and important locations that are found on the video can be added on top for more

contextual information.

5

Figure 1.2: ReLive virtual reality environment concept (work in progress)

At the center we have a map of the current location and a drone position marker, to the sides we put

sensor data that can be represented in history graph form or in more metric specific ways that better

represent that type of data, for example a rotating compass. Also at the sides we can have mission

relevant information in the form of written list logs or visually marked on the map if this information

happens to be important points of interest or a trajectory that was planned beforehand (see Figure 1.2).

Information that is displayed on other virtual screens can be hidden to the side, out side of the user

field of view, so when the operator moves his head sideways that extra data is displayed to him without

overloading the main part with more visual clutter, these screens could be set to show up on the main

part if certain mission conditions are met as to warn the operator of a status change.

Not only is the visual part important but audio is also important when dealing with virtual reality to

create a more believable and immersive environment, the special conditions that triggered the screen

changing positions could instead be an audio warning that when played it alerted the operator to relevant

information. This could be less distracting to the operator as it would be only a reminder that new

information was available and not an active change done to the scene that could disrupt the mission

review.

1.4 Document Structure

This document is structured with the Chapter 2, Related Work, describing other types of work that help

to contextualize and improve the solution that will be created. In Chapter 3, Proposed Solution, we go

into further detail about how the solution will be developed, the tools and software used and more visual

representations of the interface. For Chapter 4 we demonstrate and explain the components that make

6

up our solution and how they come together. With Chapter 5, Evaluation, the application will be tested

on it’s scalability and performance when handling different information loads. Chapter 6, Conclusion, is

where we surmise the current work and what we learned.

7

8

2Related Work

Contents

2.1 Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Drone Control and Mission Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

9

10

In this section we will talk about work that already exists in different areas that are related to our project

and can help us to better develop a good and immersive interface. The works are divided in two sub

sections, we will first look at works that fall into the category of virtual reality that immerse the user in

their environment or task and for the second sub section we will look into non-immersive solutions for

drone mission visualization or control.

2.1 Virtual Reality

With the development of more powerful and readily available virtual reality headsets, a new market

emerges for virtual reality and that is games, some game are created with virtual reality headsets in

mind but others can be adapted to support this new technology used in games.

Dota 2 is an online multiplayer game, developed by Valve, where two teams of five players fight to

take down the enemy’s base, each player has to choose a character from a pool of 100 and because of

this the combination of heroes and strategy will vary between games. A game’s duration is not set and

only ends when one team’s base is destroyed, the average game time is around 40 minutes but some

can end very quickly while others can last more than an hour.

To improve at the game the player can take advantage of the replay system that allows him to review

any game that ever happened to fix their mistakes or learn from better players, or even spectate live

games that are being played at the moment In 2016, Valve announced an addition to the already existing

replay system, Dota VR Hub [11], a new virtual reality way to interact with the spectator and replay mode

(see Figure 2.1).

Figure 2.1: Dota 2 VR Hub spectator mode

11

Dota VR Hub includes a main screen where the game is being displayed, below the player in a table

like position is the minimap that shows the position of all characters and state of the game world, graphs

indicating the amount of gold and experience difference between the teams was placed at the sides,

there’s also a dropdown panel that shows current metrics to better determine how the players are doing,

a timeline to control the replay to go back and rewatch a certain play or skip ahead in time.

Each player also has a lifesize model dedicated to their character ingame that mimics their status

and actions during the game, it’s also possible to interact with the world, you can click on the ground to

teleport and move around the scene or even click on the minimap and jump into the game world and

see the action up close.

Figure 2.2: Remote tower immersive environment concept [1]

This first paper reviews possible scenarios where more immersion and the use of virtual reality head

mounted displays (HMD) can help in the subject of Air Traffic Control and Management (ATC/M) [1], the

field of Air Traffic Control is used to monitor the airfields, incoming aircrafts and their trajectory to ensure

that no accidents or delays happen.

An immersive environment for remote control towers can lower costs by reducing the operation and

maintenance costs of a real control tower, increase flexibility and capacity by adding more operators in

times of increased traffic, monitor more remote airfields with difficult physical access or ones with low

visibility climates that infrared sensors and a virtual environment can help human vision limitations with

extra sensors and information.

12

Figure 2.3: Virtual Command Presence six monitor workspace [2]

Virtual reality can be used to simulate different types of environments, Virtual Command Presence

(VCP) is a hybrid reality application that can create multiple high resolution virtual monitors in 3D space,

and these monitors function as video players that are capable of playing or streaming videos independent

of each other.

The user has the possibility of interacting with the displays by adding more, resizing, moving or even

deleting them, this allows the user to customize their workspace to their taste, it’s also possible to change

the environment to an office space, a deserted island, black void with stars in the distance and many

more.

Figure 2.4: Bright and dark conditions in physical environment (left) vs virtual environment (right).

[12] Uses immersive virtual reality to simulate a real office environment and then evaluates how the

performance of an individual differs when using the Immersive Virtual Environment (IVE) in comparison

13

to the real one. By testing over one hundred participants they determined that users performed to the

same level when using both environments with the same conditions, bright or dark lighting, this means

that a detailed and immersive environment produces the same results as the real one and workspace

changes that can affect employees could be tested using an IVE with the desired space modifications.

When changing the lighting conditions some participants reported better performance while others

reported worse, revealing that we can improve the users’ performance by changing their environment

and with an IVE it’s faster and more efficient to test possible changes than to change the real world

environment.

Figure 2.5: Virtual lumbar trainer [3]

Virtual reality is also used in other fields, in this published paper VR is applied to medicine to help

doctors and medical staff train and perfect a lumbar puncture procedure [3], by allowing them to repeat

the procedure without endangering a real patient. A 3D software loaded with trainer data in addition

to a machine that is able to provide the user with visual as well as realistic haptic feedback needed to

successfully complete the procedure, which can be done again and again if necessary.

Figure 2.6: Exploring historic buildings (2001) [4]

14

The need for virtual reality is not a new thing, many efforts have been done in this area in the past

20 years, one example of an older system is the Hellenic Cosmos cultural center in Athens in the year

2001 [4]. The cultural center used the CAVE system that projected a stereoscopic image in the room’s

walls and gave the people visiting the center special glasses to see the image as three dimensional.

The projection showed historic buildings that may not be possible to visit today and it was possible

to use a timeline to render the buildings as they were along their lifespan for a more interactive and

interesting exploration of buildings and monuments.

Figure 2.7: MIT campus 3D model in Unity3D loaded with geotagged tweets [5]

In the paper “Improving Big Data visual analytics with interactive virtual reality” [5] they researched

ways to display and interact with large amounts of geotagged information, in this case tweets associated

with a location, in virtual reality using a head mounted display Oculus Rift and the Leap Motion hand

controllers with the Unity3D engine. They chose to use a 3D model of the MIT campus and juxtaposed

the geotagged tweets on top of the 3D model.

Figure 2.8: Single tweet interaction [5]

It is possible to interact with a single tweet to see more information about it, it’s also possible to filter

15

all the tweets loaded into the application inside the virtual reality environment, the tweets matching the

criteria defined are displayed in a new room that also allows the user to interact with them to learn more

about the data.

Figure 2.9: Operator at the helm of a virtual ship

E2C2 codenamed Blueshark [13] is a project that merges many types of technologies from different

fields to create and research what the future of virtual reality interaction and collaboration might look

like in the context of the navy, instead of just using VR they are calling it VAPOR, Virtual Augmented

Physical Operational Realities.

One of its many uses is to control a ship, so to do that a virtual ship is used and the user is placed at

the helm with touch interface controls that also appear in the simulation (see Figure 2.9), it also supports

collaboration so more people can populate the ship even if they are not physically in the same room,

creating an immersive shared virtual world.

16

2.2 Drone Control and Mission Visualization

Figure 2.10: Command Center of the Future developed by Ideum [6]

We are used to seeing shiny and future looking interfaces in movies and science fiction shows that

are only created using visual effects for a particular scene, but in this case an interactive design firm,

Ideum, brought the interface to life [6].

It was build to be used as group mission planning and analysis simulation, so people can interact with

the simulation using touch controls to move the scene environment and gather more information from

objects and ships present. Features a mission analysis panel to display relevant mission information, a

timeline to move the simulation forwards and backwards in time and see the ships position and condition

change.

Figure 2.11: Drone taking images at 60 meters (left), pilot and experts watching the live video (right) [7]

17

”Mission review of aerial robotic assessment — Ammunition explosion cyprus 2011” [7] is a paper

created for the purpose of studying the ways aerial robots, drones, can assist in the assessment of

situations in this case after a catastrophic accident on a naval base. The drones we used to better view

the structure after the explosion and its condition, in this case the drones were used live at the local to

better inspect the structural integrity and high places like roofs where it’s impractical to send people.

Figure 2.12: Interruption Assistance Interface [8]

Figure 2.13: Event timeline showing event bookmarks [8]

With the paper “Assisting Interruption Recovery in Supervisory Control of Multiple Uavs” [8] Scott

et al. showed how an event replay interface can help to recover in interruption recovery times. They

tested two types of replay that proved effective, one where it would replay the event at 10 times the real

life speed (see Figure 2.12) and the other where extra event status bookmarks (see Figure 2.13) were

present on the event timeline and the user could jump to them by clicking on the bookmarks.

The sped up scenario could more easily give context and continuity between the UAVs actions, where

as the other could give a visual summary to the user where relevant events took place, while using the

bookmarks and event timeline status to give the user extra information at a glance it’s important to not

18

clutter the timeline with events as that could be detrimental to the replay experience.

Figure 2.14: Miner simulation with replay option [9]

Simulated 3D environments can be used to train workers to recognize future problems and avoid

accidents in the workplace, such is the case with the South African mining industry where on average

there’s one accident per day [9], by letting workers go through a series scenarios that contain common

mistakes that can be avoided, it prepares the workers to easily recognize and detect unusual patterns

that can be used to avoid accidents in the real world. With a system that can play different scenarios

leading to problems a replay system is key to rewind and go back to look at the situation in a different

perspective until the worker has learned what the cause was, that way a problem that may have been

hidden before can be discovered.

Figure 2.15: (Left) Standard monitor solution, (center) CATEC’s VR screen, (right) Oculus Rift [10]

In 2015 Ruiz et al. studied the effects that immersive displays have an effect in spatial knowledge

of multi-UAV operations [10]. The tested displays are displayed in the figure above where their Field of

Regard (FoR), part of the field of view that the display can influence, increases from left to right (see

Figure 2.15).

From their tests of multi-UAV spatial knowledge, that involved questions about future UAV positions,

19

path followed and other metrics, the more immersive displays with a higher FoR scored better than the

monitor solution. One thing to keep in mind was that despite this better score, if this total immersion

is not taken into account when building the environment and scenario as to not overload the user, the

operator’s workload could increase and affect the results.

Figure 2.16: Quadcopter programmed mission using GPS, data overlay by Dashware

Now that many users are able to buy off the shelf drones, a need arises to better visualize flight data

that is able to be shared between people. Dashware [14] created a solution that takes different types of

collected data from many activities like flying drones, cycling, skydiving and imposes it on the recorded

video to give more context to the person watching the video.

The interface overlay is hard coded into the video file to make it easier to share and comes with a

number of presets that are better suited to the activity displayed. These presets are made up of compo-

nents that can be altered and customized depending on what information the user needs displayed.

2.3 Discussion

While doing the research for this project we found many papers and articles that could be relevant, but

mainly focused on the one that could help explain or justify the benefits of virtual reality and replayability

in different fields.

Early our intention was to gather cases where their objective focused on using technology to help

prevent future problems. This was done by showing doctors practicing sensitive procedures with a

certain realism factor and feedback [3]. By using a drone to inspect a real disaster zone it was proven

20

that drones can be useful for live damage assessment [7] and that this could be translated to a post

mission review to better review the footage and sensor data captured by the drone.

Accidents can be a common occurrence in more dangerous workplaces, sometimes these accidents

are known before hand as they have happened before but if a worker as never encountered that specific

situation he might not be aware to prevent it, as is the case of the South African mining industry [9] that

worked to avoid these accidents by using a simulated work environment that will lead to a mistake or

accident and letting users explore the different scenarios to prepare them to better spot what’s wrong in

the given scenario and avoid the accident.

Then we searched for work that specialized in replay or review, to get to know already existing

solutions and how they are used and can help to develop a replay system application. Our oldest

referenced work is from 2001 that is used to explore historic buildings [4] and can let people explore

older monuments that do not exist anymore illustrating the use of a review system. A replay interface [8]

is helpful to the user by letting him find and assess a situation with a quick glance and determine

that where more worrying problems are appearing, also the South African mining industry example [9]

discussed before is also a review system example.

Immersive virtual reality work that contain useful use cases and smart implementations that can help

shape our own solution, this includes a airfield remote tower concept that features VR application to

airfield management [1], recreated virtual workplace environments where the user can in a more familiar

place [2] and [12], how to interact with large amounts of geotagged data and explore the virtual world [5],

control a virtual ship with different remote people [13] and a virtual reality system implemented in a game

to help spectate and replay matches [11].

In the interface side of things, a futuristic inspired application with an interactive interface [6] can help

developers realize that many of the previously unused concepts thought to be impractical in an interface

are now doable and user friendly. The potential of a visual solution with a higher field of regard can help

users in comparison with a normal monitor [10] and an interface with modular components that monitor

specific vehicle metrics [14] can be beneficial to users in their customizability.

21

22

3Proposed Solution

Contents

3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

23

24

A 3D virtual reality environment was created to immerse the user in the virtual world, it consists of

a simulated command center workspace where the user has the ability to monitor and replay the

drone’s mission.

3.1 Architecture

For our solution we use the Unity3D game development engine, it’s an engine we have used before, it

has extensive official support documentation and community support because of its widespread use, it

was also a Tekever requirement as the programming language C# is preferable and more compatible with

current Tekever’s programming environment where C# is prevalent in many of their developed systems.

With Unity we have the option to export to many different platforms if needed in the future, as this

engine has great support for many operating systems and virtual reality headsets, but for now we are

only interested in the Windows build version and we use the HTC Vive as our head mounted display

(HMD)/virtual reality headset, in alternative to other HMDs like the Oculus Rift because of previous

experiences with both headsets and the Vive providing more flexibility and fidelity when in use and while

developing games and applications.

In addition to Unity3D we plan to use the Valve created SteamVR Unity plugin that helps create

a layer of abstraction when dealing with Virtual Reality headsets and controllers to help with quicker

application development.

Figure 3.1: Project architecture overview

The mission replay system (see Figure 3.1) is available after the mission is over and the drone

has returned from its planned trajectory and concluded its goals, after a mission is concluded we have

access to the mission file (.ts) that contains two data streams, one is the video feed (h264 video stream)

from the onboard camera and the other is the timecoded data captured from sensors and payloads of

the drone that may be needed while reviewing the mission.

Our project is comprised of many different modules (see Figure 3.2), each with a specific function

that when working together create the ReLive mission review application. It starts by loading the mission

file into the application where the video and data is sorted by the Data Handling module according to

25

their timestamp and labeled for easier lookup when replaying the mission and showing a coherent state

at each moment in time.

The State and Replay module are responsible for ensuring the accuracy of the current state data

structures that are used by the various interface modules to show a visual representation of the scene

and data gathered, for example the current video frame, sensor history needed for the graph and the

drone location on the map.

A module created to alert the user of important events or changes is aptly named the Events module,

and will keep track of predefined or user custom metric thresholds and criteria that trigger a certain event

and notify the Interface module that is able to display the alert on the user’s field of view or play an alert

sound.

The Interaction module connects to the SteamVR and if necessary the Vive headset and controllers

directly to be able to respond to the user’s controller inputs and affect their environment by contacting

the Interface module which in turn will need to contact the State and Replay module to alter the current

effective time being reviewed and be put in a coherent state (see Fig. Figure 3.2), as the user can directly

change the timeline by pointing and clicking and interact with the virtual screens by adding more, resizing

and moving them. It’s also possible to change the state directly by using the controller hotkeys to rewind

or go forward in time.

Figure 3.2: ReLive module architecture

The main use case (see Figure 1.2) that was illustrated earlier to give the reader a visual image of

our concept). It is a control center where the user has a large screen in front of them to monitor the video

camera feed of the drone mission, relevant information about the drone, detected objects and important

locations that are found on the video can be added on top for more contextual information as to what the

drone was recording at the time.

26

At the center we have the map with the drone location, mission relevant information can be displayed

in the form of written list logs or visually marked on the map if this information happens to be important,

for example points of interest or a trajectory that was planned beforehand.

Below the central part of the scene there is a timeline showing the duration of the mission similar

to a video player timeline, and the user can go back and forward in time by using the controller or by

pointing to a specific part of the timeline, this timeline also has relevant information color coded or with

icons displayed, for instance if a ship is detected during the mission, instead of needing to look through

all the footage to find that particular ship detection, it is parsed beforehand from the available data and

displayed in the timeline at the specific time it is found to help the operator find relevant bits faster.

27

28

4Implementation

Contents

4.1 Replay System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.2 Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

29

30

This section discusses the implementation details and components needed for our solution and how

they work together to complete the replay system.

4.1 Replay System

The replay system consists mainly of two parts the interface and the backend that is responsible for data

handling logic. Let’s start with the interface:

Figure 4.1: ReLive’s scene overview, with camera field of view changed.

In Figure 4.1 we can see an overview of our scene, we shifted the camera perspective to show what

a user using a virtual reality headset could have in his field of view. This stretches the elements closer

to the edge of the screen but it is only for illustrations purposes, as the real use case won’t have the

camera perspective changed.

This scene example, has some elements added to the bottom part of the map, some of them had

their styles changed. Another graph added in the left side, and further to the left of it we have the big

map overview just outside the captured field of view.

4.1.1 Video Player

A big part of a replay system is watching the footage captured by the drone’s camera, so in our imple-

mentation created inside a 3D virtual world, it fills a large part of the scene and the user’s field of view

when looking in the video screen direction comparable to a cinema screen.

Mission information is used to populate our scene elements (see Figure 4.2), above the video screen

we put the name of the current mission being replayed, and the date and time of day it took place, at the

bottom we show the video’s current time and total time the drone recorded for this mission.

Unity’s built-in component handles the video file loading in different platforms for us, but we still need

to manage it with a Video Manager script that can speed up or slow down the video’s playback speed,

31

Figure 4.2: ReLive’s video player with timeline shown below.

that can jump forward and backward to an exact moment in time, this is called seeking. The module

doesn’t support reverse video playback but we can still go back by jumping back multiple times.

Using the video manager’s features we added on screen buttons to control the timeline changing the

current time of the video. The timeline is located below the main screen and shows a visual indicator

that represents how far along the video playback is, it’s also possible to jump to anywhere in the video

by clicking in the timeline.

This allows the user to quickly jump to their preferred time of the video to review a moment and

analyze it, greatly combining the timeline jumps with visual events on the timeline, the timeline’s clickable

area is bigger than it appears so it is easier to interact but still keeping the smaller look that it has.

Figure 4.3: Timeline closeup with event indicators. Symbols showing (left) and minimalistic view (right)

Figure 4.3 shows different visual event indicators that can appear on the timeline itself to create a

visual reference guide to when the events happen relative to other events or the current time. From left

32

to right is the warning event, current time marker, info event, mission start marker and finally a danger

event.

4.1.2 Events

Our solution has different types of time stamped events, these are extra ways to contextualize the cap-

tured data that occurs at a defined point in time, stored in the mission file.

The events types are:

• Mission: Represents the start of each mission step and helps convey the mission progress to the

user.

• Danger: Best used for dangerous values that were in the data, where a set threshold has been

exceeded and it is worth looking into.

• Warning: Can be a good complement to the danger event where a warning occurs before the

extreme conditions are met, is also used to show object sightings on the map

• Info: Its purpose is to show timed notes of the mission file and help guide the mission review. Can

also serve the purpose of showing objects on the map that may not demand the same urgency as

a warning

They can be displayed on the screen inside the timeline (see Figure 4.3), in their own Mission List or

Event List interface, placed on the map or as visual effects to the available modules in the scene to alert

the user of an important change in the data.

4.1.3 UI Elements

Mission List (see Figure 4.4) is a component created by us that lets the user quickly determine the

mission progress in a visual way, it shows the objectives and steps defined for the mission and highlights

the current one based on the video player’s time, this feature takes advantage of the Mission event type.

The component also allows user interaction to quickly jump in time to the chosen mission objective by

interacting with the mission step entry for a quick analysis of a certain part of the mission.

The Event List (see Figure 4.5) serves a similar purpose to the mission list but this time for the rest

of the event types, it lets the user know what is going to happen next in the mission at a quick glance.

When an event is taking place or a threshold is to be exceeded it is shown on the event list, they follow

the same color scheme as their timeline counterparts, interacting with the event’s list entry jumps you to

that moment in time.

33

(a) Default state (b) Mission in progress, current objectiveis selected

Figure 4.4: Mission List Component.

(a) Default state (b) All events highlighted

Figure 4.5: Event List Component.

Each event entry on the list shows the time when it happens during the mission. The entry has a short

description of the event, or a note that refers to a moment in the mission. If it’s an event generated by a

Databank condition it specifies which type of data was responsible for this event and the data value that

met the criteria, at the right side of the entry we place the reason for that event and the value condition

it exceeded. Currently the thresholds conditions we implemented are: “greater than” and “lower than” of

chosen values.

34

After an event’s time has been passed it still lingers for 5 seconds to give the user context to what

happened previously, this helps when jumping through the footage to review other sections of the mission

where events that occurred are not instantly hidden.

To not clutter the scene needlessly, the event list is a component that allows scrolling to view all the

other events that may not be showing at the start, as they will only happen later in the mission.

A script in the event list component is responsible for checking every event it has and checks to see if

it should be hidden at the current time according to the 5 second condition. Also when in the time range

of the event it should be highlighted for the user, the highlighted state of active events can be seen in

Figure 4.5.

4.1.4 Data Elements

Data that was recorded and stored also needs to be shown to the user while he is reviewing the mission.

We chose to have different ways to represent data visually because there isn’t a single solution that is

best for all types of information.

Our goal is for the user to be able to clearly distinguish which data it is with a quick glance by

choosing the appropriate component. It needs to have the data type’s name visible, the units for how it

is measured and of course the current value. They can also exist in the form of a graph that shows the

progress of those data values over time.

Figure 4.6: Different scene data elements.

The user is able to place more data elements on the scene during mission review thanks to our object

spawner component. This component is responsible for the creation and placement of data objects or

graphs, in Figure 4.6 we can see different styled variations of our elements, their value is changed

according to their corresponding Databank. The bar next to each one of them, also changes with the

value. Growing its white component until it reaches the max possible value of the Databank, and when

the Databank is at its lowest value we can’t see it.

We want to be able to place objects around the user in a circular fashion, so a 3D oval shaped

object, sphere that’s being scaled non uniformly, exists centered on the user position and encompasses

the scene, then an invisible ray is cast from the user’s position in the direction he was pointing and when

it hits the sphere we know where to place our intended object. After that, it rotates to face the user so

35

it can be seen even in every position it is placed even if the user decides to place it directly on top or

below his position.

Figure 4.7: User inside the sphere mockup. Rays sent from the camera, hitting the inside of the sphere.

This effect proved difficult at first because of the user’s position being inside the sphere (see Fig-

ure 4.7), the raycast was not being detected and it could be because of the way a 3D object geometry

mesh is represented. Raycast hits to the backside of a mesh are not detected, this meant that when

inside the sphere every polygon face of the sphere is the backside to the user, so we had no collisions

after casting our ray.

Many solutions were tried, using unity’s own scripting to invert every polygon of the sphere making

it face the user, another attempt was to invert the polygon’s normals instead of the mesh polygon but

both ways had no noticeable effect. We even tried shaders to invert the object but they weren’t working.

The solution was to use a Unity plugin called ProBuilder that actually works and performs geometry

operations to objects, it correctly inverted the polygons and the collision was detected.

4.1.5 Map

An important feature to have is a map so the user can follow the mission, with a birds’ eye view of the

location. With the top-down view, we can provide additional spatial context to what is happening on the

screen, how the mission progresses and how each event relates to the overall picture and other events.

To have a map inside Unity we used a plugin created by Mapbox, a service that specializes in maps

for many different platforms, tools and use cases. It handles the loading and creation of a map in our

scene and is customizable, we can choose the starting location and the map tiles around it appear on

our map.

36

We can even choose other themes instead of having a real life satellite image, we can use choose

between minimalistic, cartoonish, dark or change its colors to our choice or even browse the community

created themes and pick the one that fits best. It can highlight roads, create buildings or show information

about different points of interest in that area and stores terrain elevation data which can be used on the

generated map.

Figure 4.8: Map during a mission, with two warning events (in yellow),

For our main case, we want the default look, photo captured from a satellite, no need for buildings

as they don’t look very good outside of large cities, and they would interfere with our placed objects.

Mapbox also provides an easy way to place objects on the map given their GPS coordinates, making

our life much easier when moving the map and keeping the objects where they should relative to the

map.

The drone is placed on the map at the currently recorded location coordinates, we follow the drone

in our map to give it focus when the user is not interacting with the map. The user can move the map to

explore other areas, instead of the one where the drone is, to gain more insight on the mission.

Other events are also placed on the map (see fig. 4.8), such as the mission steps locations of which

the current objective has a pulsating circle indicator, sightings appear on the map for the time span that

they were detected and then are removed when the sighting time limit is reached. A trail is left behind

by the drone through the places it passed on the map, making it easier to know the trajectory the drone

took.

Besides the main central map, we also have a larger map overview to the side, its purpose is to

37

better show the mission progress if the mission takes place between large distances and its scope can’t

be captured by the more close-up central map.

4.1.6 Controls

Interaction with our mission review system can be done in two ways, the first being with a normal setup

of a keyboard and mouse. Pointing and clicking to interact with the objects on the screen as you would

on a previous review system, while also being able to use the mouse to look around the scene like you

would a game.

This method is still supported because it greatly helped throughout the creation of our program with

the scene prototyping, placement of components and with fixing problems during development.

Another method of interaction is with the Virtual reality headset and its handheld controller. Where

the user can interact with the world using the controller as a pointer to simulate a point and click but in a

3D virtual world. Looking around in this virtual reality mode comes more natural to the user as he simply

needs to move his head to explore the scene, much like in real life.

The mission review is meant to be done without leaving its starting position but the user can also

move around the scene if he wants to have a closer look at the scene, for instance, the map.

While targeting a Data element the user is able to change its state with the directional controls, on

the controller or the keyboard. We can change its visual style to another element, change the Databank

it represents or move it.

When the user is not interacting with anything, he is able to control the video player playback speed,

playing state and jump forward or backward.

4.1.7 Data Metrics

A replay system needs to be able to show the state at a moment in time for that the needed information

that was recorded and now is going to be displayed needs to be well organized.

The current drone sensors record data at fixed time intervals to preserve storage space and band-

width and we could just develop a system that also replays at fixed intervals but that solution wouldn’t

scale well in the future if the drone started to record some sensor data more frequently than others or if

it recorded in irregular intervals.

So we decided to go the simpler way and store each data set as it is recorded with its timestamp

stored along with data value, making it a more flexible system that supports as many or few data points

as it needs. The data set entity we created is called a Databank.

The most common data types that need to be stored are number values, GPS positions, dates and

written words, they can be represented by strings, floats and a combination of both.

38

As different types of data have important values that need to be monitored or taken into account, a

Databank also supports thresholds that can be defined and then be used to alert the user that a specific

data set might be interesting to look at the time the threshold was exceeded as it might suggest that

something important happened. For example: if we are dealing with the altitude Databank, an altitude

below a given limit or even 0 meters might be a catastrophic problem for that mission.

4.1.8 Data Interpolation

We have the data storage structure implemented, but now what happens in between the measured time

intervals? See Figure 4.9.

Figure 4.9: Interpolation scenario for altitude in meters (m). A is the previous value, C the next one and B is at thecurrent time.

There’s some options that we can choose from and they all have their use cases, such as:

• Show no data: when there’s nothing recorded in that time frame we guarantee that operator

reviewing the mission just sees what the drone actually recorded, for example if tracking a ship’s

beacon we see the actual position and timing when it was detected with no lingering effects.

• Show the most recent recorded value: like the previous option the operator also sees real values

that were recorded, it could be used to always show a ship’s last known position or for instance

show a diagnostics sensor last recorded error rate

• Show the linear interpolated value between the previous and next value: this option creates

possible values between two real values, proportionally to how closely we are to one of the points,

providing a smoother experience for the replay user

• Develop a more complex interpolation method: this last option is the one with more potential

for future work as a better interpolation model can be modeled specifically with a data set in mind

could produce better and more coherent resulting values for that data in particular

When the mission is created it’s possible to choose which method will be used for that data type,

with the default being the linear interpolation (lerp), it provides the smoother experience with no jumping

values from time to time.

39

4.2 Difficulties

Development is not always a straight forward progress from start to finish, sometimes there are events

that halt progress or even roll it back. They could be human or machine induced, many of which are

impossible to predict and account for at the start, in this section we will discuss the ones generated by

our chosen tools.

Unity, for many of its good points and advantages as a content creation tool, there are always draw-

backs. It is a balancing act between good features that save us time during development and inconsistent

bugs that seem to appear out of Unity’s will to not cooperate.

4.2.1 Hangs and Crashes

Every software that is created needs to run and be debugged at some point, with Unity it is run each

time we want to play the scene we put together and we can debug it by choosing to attach our debugger

in Visual Studio, our chosen development environment. When playing the scene there was a small

percentage chance that it would crash and shut down Unity and an even higher chance of crashing if we

were trying to debug our program and stepping through the code.

The problem is that with a crash any progress made to our scene is lost if not saved beforehand,

when testing changes to see what could be better for our solution sometimes these changes stack up

and we never expect that a crash is right around the corner. There were times that luck was on our side

but some times the crashes were very unexpected and some progress was lost, it was never so large

that it ruined our day but it was an annoyance all the same, and if it wasn’t a straight crash where the

unity program would shut down, it was a complete program hang that did not respond to any actions, this

could be considered worse than a crash because it gave us hope that unity would recover but when it

didn’t it was worse because it forced us to forcibly terminate the program ourselves, effectively crashing

and losing progress anyway.

A developer usually is only blindsided by the first crash, and after that, he looks for ways to prevent it

from happening again and possibly recover what was lost. When a crash happens, the natural thing to

do is try to open it again and see if we can recover anything, with Unity there’s a file saved when a crash

happens but amazingly it’s removed at the program’s startup, so we had no luck there.

Our first solution would be a timed interval auto save feature so that if a crash happens there’s not a

whole lot of progress being lost, unfortunately, Unity still doesn’t have an auto save feature, a surprising

turn of events. The community has requested it for many years but it’s still not available in Unity, thankfully

it’s possible to expand Unity’s functionality to force a project save when an action is performed, in this

case, we chose to save it when clicking play before our scene is run, mitigating our lost progress that

occurs when running our program.

40

Unfortunately the script responsible for the pseudo auto-save feature was not properly installed the

first time so we got another scare sometime later, since then the hotkey: Ctrl + S that saves the project

became our best friend, as we couldn’t really trust the script even tho it was correctly installed this time.

So we started pressing it every time something changed and it became a reflex.

4.2.2 Extra Problems

One instance of Unity screwing up was even more surprising, now the project was saving every time it

was being run and on that particular moment it runs but it was complaining and spewing errors every

second.

Some objects were not properly set and their references were empty, this could happen when creat-

ing an object for the first time but we had not touched that object in quite some time as it was the main

one responsible for the screen and other effects. After examining it further, it wasn’t just a few objects,

all of the objects references were empty.

Many emotions were felt at that moment, the project was just automatically saved and now no ref-

erences were set, it was possible to redo all the references but it would be a long and tedious job and

some might be set incorrectly thus breaking other functionality.

We remembered that sometime before this dreaded event happened a backup tool was run when

starting this work’s session and that backup file should be problem free, despite being a bit outdated. So

we decided to open both files and compared them to see the extent of the damage, that truly surprised

us, the file now had miraculously regained all the references and was working without problems.

That’s when we realized that Unity likes to pull pranks from time to time and a restart will set it

straight, no need to waste time trying to solve ghost problems that don’t really exist.

4.2.3 Unity Duplicate Problem

An odd problem that kept showing up while creating and modifying the scene in Unity was when dupli-

cating existing objects already in the scene. Their transform properties like position and scale would be

changed to strange float values that were very close to the original duplicated value but not the same

(see Figure 4.10), when dealing with nested objects it’s not advisable to have any unknown factors that

could give Unity a reason to not place the objects where we want them.

Such a simple action that just creates a copy of another object and doesn’t work as it should that’s

just one of the quirks of dealing with Unity, the values were not always the same as in the example but

we were able to replicated it in our first try showing that’s a really common bug and was always in the

back of our minds when changing the scene.

41

Figure 4.10: Original object’s properties (left). Duplicated object’s properties (right)

4.2.4 The Update Dilemma

During the program’s development, the tools used received many minor and major updates and any

experienced developer would caution against major updates to the development tools unless a new

feature added is really beneficial to the workflow.

Unity is known for breaking features with minor and specially major version updates and there wasn’t

any particular feature that we required from the newer versions. But the .Net Framework used by our

version of Unity that defined the programming language features was very old and when creating code

for the data handling part of the replay system, the need arised for a newer version that was standard

for a few years. The update was a toggle in the Unity settings and seemed simple enough, but this had

effects that would only be felt a couple of bugs later.

GPS positions used for the location part of the program started to break and weren’t being displayed

properly, no code relevant to the locations was changed that could cause them to break. The previous

simple framework updated had a fun side effect: the decimal number separator on the newer framework

versions depended upon the user’s localization setting.

Previously the decimal separator was the point used in English speaking countries that had more

influence in the creation of programming language’s. The new update supported many different countries

preferences and set them by default based on the user’s system and the operating system used for

development was Portuguese where the comma is the decimal separator, breaking the functionality that

depended in decimal values as GPS positions tend to do. The fix was to change Unity’s localization

to English and almost like a tradition the fix is a community created script that can be used to alter its

settings and not a build in option.

A fun ghost bug that the localization fix created was one where the same string placed in different

functions that were used as a key to a dictionary was generating an error when called from one function

but working correctly from the other, implying that the word didn’t exist sometimes. “Informatica” was

the string causing the problem, at first we thought that the accent in the “a” might’ve been typed wrong,

but after being copy pasted it still wasn’t working and even when using the same variable, it only worked

42

when the string had no accents and just simple ASCII characters.

This was because the functions were in different folders, one had the localization fix to English and

the other did not, the fix of pasting the localization script to every new folder created during the project

didn’t seem feasible so strings that could pose a localization problem are not supported. In the future

to support these types of strings and keep the newer framework support a possible solution could be

changing the internal representation of GPS positions to not need the localization support script, this

was tried before when the location problem first showed itself but didn’t seem like a good solution, after

all the problems that happened maybe it was the right choice.

In short: changing the net framework version will have unforeseen consequences in the strangest

places, and proceed at your own risk.

43

44

5Evaluation

Contents

5.1 Technical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

45

46

A review system will deal with many different types of missions, some that involve a larger volume of

data to be processed than originally intended.

So to evaluate this project’s implementation and how it could scale with different amounts of data

being processed and displayed at a time. We will test its performance when dealing with a scenario

where the user wants to see that data displayed simultaneous on the scene.

This will also help us understand how the program deals with more extreme use cases and to figure

out which future upgrades and feature additions are feasible on the current solution or if they need

modifications before being implemented.

5.1 Technical Evaluation

Our system that was used to benchmark the tests is slightly above average performing system, a desktop

with a combination of older and newer parts. System specifications:

• Processor (CPU): Intel Core i5 2500k @ 4.0Ghz

• Graphics Card (GPU): Nvidia GTX 1060 3GB

• Memory (RAM): 16GB DDR3

• Storage: 250GB Crucial MX200 SSD and 1TB Hard disk drive 7200RPM

An equivalent priced system made with brand new parts can easily out perform it, making our solution

even better suited for the future and possible application feature upgrades.

5.1.1 Additional Screens

Unity’s video processing implementation is helpful because it supports different formats abstracting the

details on how it is done. For loading videos it uses streaming after being provided an URL, can also

be a local folder path, to locate the video file. But unity’s video playback is very taxing on the system’s

processor.

Our current solution has a single video screen, because of the requirement that the drone only has

one camera to record the footage. In the future different drone models could be used that have more

video feeds recorded, so we test how adding a screen affects the frames per second (fps) of our program.

Comparing it to a popular media player program, VLC only uses about 10 to 15% of our system’s

CPU to play a video. Where unity uses 20% of the CPU without the video, and an extra 30% just to play

our video, making it near 50% total when running our whole scene. VLC is close to a third more efficient

for playing video than unity, so a future VLC integration with Unity could prove beneficial.

47

In the future, if more than four videos need to be played in the scene at the same time, a different

solution needs to be considered. Either not using unity’s own implementation, or scaling a less important

screen to a less performance impactful resolution.

When testing performance we need to unlock the frame per second settings of Unity, as it tries to

synchronize the engine’s updates per second with our monitor refresh rate, if tests were conducted while

this was turned on they wouldn’t be valid because it limits the performance.

It’s common for monitors to have only 60 hertz (Hz) or update 60 times per second, with VR it’s said

that to have a smooth experience a minimum of 90Hz is required. Making Unity’s attempt to limit our

update rate to 60 times a second not suitable for VR or testing. Fortunately, the community provides

scripts to try and unlock the frame rate because the built-in setting were not working as intended in

uncapping the limit.

Table 5.1: Video files used

Video Total Time Size Resolution Average FPSA 1 minute 90 MB 1080p 199B 5 minutes 1.8 GB 1080p 187C 2.5 hours 2.24 GB 1080p 208D 2 hours 3.11 GB 720p 212

Now we will start testing how the application handles different types of videos. These tests were

done on the editor to speed up the testing but the percentage differences when tested with a compiled

program build were in the same range.

Playing the videos for the tests, we noticed that at the start there could be a period of caching

where the frames per second would start in a lower range, but gradually climb to the recorded values of

Table 5.2 after a few seconds or after skipping ahead in the video. Average frames per second recorded

were not perfectly stuck to that value, they oscillated around that value but could be 5 to 10 frames above

or below at any given time.

Jumping ahead in the timeline has an immediate performance cost at the moment the skip is made,

when jumping through the timeline there is a sharp drop of 50-70 fps that instantly recovers (within 20

ms) to the normal recorded range. The performance drop exists in all four files tested and it is due to the

way unity handles video files, no obvious fixes exist.

Looking at Table 5.2, video file B is the densest file meaning each frame has more image information,

with 1.8GB for its 5 minute total time and has the lowest average frames per second. Making it seem

that the denser a video file is, the lowest its measured performance and that’s exactly what we see in

the recorded performance information. Unity prefers displaying a less dense video file, and is able to run

faster. It also prefers lower resolution files, such as the Video file D, but in this case, the performance

gain is not that significant.

48

We also tested putting the files in different types of storage disks, predicting that the file on the faster

SSD would allow the program to run faster than one stored in the slower Hard Disk Drive (HDD). But no

noticeable difference was measured in their CPU usage or frames per second.

Table 5.2: Performance stats of one vs two screen

Number of Videos CPU Usage % Average FPS1 46 to 53 2152 60 to 66 200

Extra video player screens are a possible improvement path in the future so we tested to see how

another different video affected our program performance. Adding another screen increased the CPU

load by 10% and there was a decrease of about 15 fps, a 7% decrease. If there’s a need to replay

footage from different cameras, using the unity video approach we can get to 4 screens while maintaining

a smooth performance.

Regarding screen size, despite already having a large screen we can’t predict if it’s the perfect size

for all future missions so we tested changing its size. We scaled it to 10 times the normal size, testing a

very extreme possible scenario and it only reduced our performance by about 15 frames per second. So

if needed we can increase the screen size without worrying about causing the performance to degrade

and reach unpleasant levels.

At the end of this round of testing we ran some of the tests to compare the results to a compiled build

version and the performance increased, it added 80 to 130 fps depending on the test case, making it

even better and nowhere close the 90hz undesirable limit.

5.1.2 Additional Data Elements

Databanks are used to store the recorded sensor information and after they can be displayed on screen

with a DataElements component. In this section we will test how adding more interface components

affects the performance of our program.

Our scene already starts with 6 data interface components, we created a grid that spawns the desired

amount of components. The first test will spawn 25 elements in a 5 by 5 grid, and the second one will

have 49 elements in a 7 by 7 grid (see Figure 5.1), this last case is an extreme case that’s unlikely to

happen as that amount of elements on screen at the same time is very distracting. It is used to assess

the performance change when adding extra elements.

Each element is requesting its interpolated value one time per frame and updating itself on screen.

When we test the 25 grid array we get an average of 160 FPS, changing to the 49 grid we get 110

FPS, both are around 200 FPS when the visual elements are disabled.

49

(a) 25 Data Elements in a 5x5 grid. (b) 49 Data Elements in a 7x7 grid.

Figure 5.1: Data metric grid of interface elements.

These values we are seeing, are starting to get close to the 90 FPS limit we talked about earlier so we

check the compiled build version to see if we would have problems and the values increased significantly.

The grid with 25 elements increased from 160 to 250 FPS and the 49 elements one increased from 110

to 185FPS, out of the lower intended performance limit and into a smooth stable range.

5.1.3 Data Frequency

Data frequency is also a parameter which can be tested, how many drone data points do we store per

minute. First, we tested with 10 per minute, having real information recorded once every 6 seconds,

then we tested with 60 points per minute, or once per second and the performance stayed the same.

This could be because we are already interpolating values many times per second, and having more

points of real data doesn’t affect how many we need to consider to interpolate. It just changes which two

points we use when interpolating, but having a faster recording will mean we can display more accurate

data if the drone can record it that fast with no loss to performance.

Different source videos from Table 5.1 did not have a significant impact on the performance of data

frequency based on the video file time, not considering the already measured impact from the previous

testing.

Moving the map only has a performance impact when panning results in an empty tile that needs to

be fetched and loaded from the server. This loading of tiles lowers the performance into undesired levels

below the 90 FPS range into the 50s or even small hangs, but it’s the way Mapbox loads new information

so we can’t avoid it.

In this chapter we tested different scenarios that could expand our application’s functionality, by

50

adding more video screens, increase the number of data elements on the scene and raise the data

recorded rate. All of them proved successful while keeping the program functional and within satisfying

performance levels.

51

52

6Conclusion

Contents

6.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

53

54

This project started with the challenge of developing a new flying drone mission review system in

virtual reality as an alternative to the current simple non-3D window graphical user interface applications.

We were able to list some limitations of the current solution and ways to improve it with our solution.

Our application was created with a virtual world in mind, and the user is able to review the footage

captured during a mission, while having an overview of its progression via the Mission list that tracks the

objective progress. The Event list is used to observe detailed information about coming events and the

central map for a positional overview on how each event relates to one another and how far along the

drone has come since the start.

The video timeline gives additional insight into the mission, as the user knows how long it will take to

complete, but also for the information that an event will happen around a certain part of the mission just

from a simple symbol placed in the timeline.

Interface elements were created to facilitate the examination of different types of sensor data and

can be arranged in a way that better helps the user in their review.

Data interpolation is used to smooth out parts of the mission review where data might not have been

collected or the interval between each data point stored is too large and a stuttering review experience

is avoided.

Putting it all together, we have a review system that is able to handle increasing amounts of data, up-

date their values hundred times per second while displaying the values and graphic elements changing

on the scene at the same time. It also deals well with small and large file video files where total times

can reach hours with no noticeable performance difference.

Our solution has the capability to be expanded with future iterations by having more video screens

with different files playing. It can support faster-stored data interval rates where the time gap between

data points stored is much smaller, preparing it for a future where sensor recorded data requires a more

precise measure.

6.1 Future Work

For ways that the solution could be improved, adding an actual reverse playback would be a valuable

feature, the program could pre-reverse the mission footage file at the start and keep it loaded in the

background for the time when the user puts the playback in reverse, it would then change the displayed

video files and just play the reversed video.

Advanced heuristics or machine learning could be used on top of the data to detect possible points

of interest to the mission review and mark them as special events in addition to current ones.

There’s a possibility to implement live functionality in addition to post mission review, the unity video

handler already supports a URL for the video. If using an online URL is not viable, it could be receiving

55

chunks of the video, smaller files rendered or cut and playing them one after the other to simulate

a continuous video. A network component would receive the streaming data in the order they were

recorded and populate the Databanks with the corresponding data.

A clever interpolation can be implemented that is based on the type of data it is trying to interpolate.

This new method could use more than 2 points to estimate the correct value or even based on the

value’s momentum, if it was increasing too fast previously, it might reach the next value faster than a

linear interpolation formula would predict.

Choose other program windows and integrate them with the virtual scene, so it’s easier to adopt extra

functionality from other programs or display web pages.

Having a changeable interpolation rate between different types of data, currently it is running on

every possible frame. It could be reduced to 60 times a second if the number of Databanks grows

exponentially to reduce program CPU load when the amount of data gets too extreme. This rate could

be based on a priority queue, and the bottom ones were given lower interpolation rates, updated fewer

times per second and the highest priority would be updated every frame.

One aspect that wasn’t explored in this work was having different rooms or environments and could

be interesting to explore.

Would be interesting to add conditions to each mission objective, and use metrics to check if they

are completed. Not just rely on the mission file, this addition could detect situations where the human

operator didn’t notice that the mission step objective condition was met and didn’t set it to complete until

later. Mitigating a possible human error in the process of automating mission progress detection.

56

Bibliography

[1] M. Cordeil, T. Dwyer, and C. Hurter, “Immersive Solutions for Future Air Traffic Control and

Management,” New York, NY, USA, 2016. [Online]. Available: http://doi.acm.org/10.1145/3009939.

3009944

[2] Virtual command presence. http://eyecreateworlds.com/index.php/virtual-command-presence.

Accessed: 2018-05-08. [Online]. Available: http://eyecreateworlds.com/index.php/

virtual-command-presence

[3] M. Farber, F. Hummel, C. Gerloff, and H. Handels, “Virtual reality simulator for the training of lumbar

punctures,” Methods of Information in Medicine, vol. 48, no. 5, pp. 493–501, may 2009.

[4] L. Mosaker, “Visualising historical knowledge using virtual reality technology,” Digital Creativity,

vol. 12, no. 1, pp. 15–25, Mar. 2001. [Online]. Available: https://www.tandfonline.com/doi/abs/10.

1076/digc.12.1.15.10865

[5] A. Moran, V. Gadepally, M. Hubbell, and J. Kepner, “Improving Big Data visual analytics with in-

teractive virtual reality,” in 2015 IEEE High Performance Extreme Computing Conference (HPEC),

Sep. 2015, pp. 1–6.

[6] Command Center of the Future. http://www.ideum.com/portfolio/command-center-of-the-

future. Accessed: 2018-05-08. [Online]. Available: http://www.ideum.com/portfolio/

command-center-of-the-future

[7] M. Angermann, M. Frassl, and M. Lichtenstern, “Mission review of aerial robotic assessment —

Ammunition explosion cyprus 2011,” in 2012 IEEE International Symposium on Safety, Security,

and Rescue Robotics (SSRR), Nov. 2012, pp. 1–6.

[8] S. D. Scott, S. Mercier, M. L. Cummings, and E. Wang, “Assisting Interruption Recovery in

Supervisory Control of Multiple Uavs:,” Proceedings of the Human Factors and Ergonomics

Society Annual Meeting, Nov. 2006. [Online]. Available: http://journals.sagepub.com/doi/citedby/

10.1177/154193120605000518

57

[9] E. van Wyk and R. de Villiers, “Virtual Reality Training Applications for the Mining Industry,”

in Proceedings of the 6th International Conference on Computer Graphics, Virtual Reality,

Visualisation and Interaction in Africa, ser. AFRIGRAPH ’09. New York, NY, USA: ACM, 2009,

pp. 53–63. [Online]. Available: http://doi.acm.org/10.1145/1503454.1503465

[10] J. J. Ruiz, A. Viguria, J. R. Martinez-de Dios, and A. Ollero, “Immersive displays for building spa-

tial knowledge in multi-UAV operations,” in 2015 International Conference on Unmanned Aircraft

Systems (ICUAS), Jun. 2015, pp. 1043–1048.

[11] Spectating dota 2 in vr. Accessed: 2018-05-15. [Online]. Available: https://www.polygon.com/2016/

7/29/12325418/dota-2-vr-valve-vive

[12] A. Heydarian, J. P. Carneiro, D. Gerber, B. Becerik-Gerber, T. Hayes, and W. Wood, “Immersive

virtual environments versus physical built environments: A benchmarking study for building design

and user-built environment explorations,” Automation in Construction, vol. 54, pp. 116–126, Jun.

2015. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0926580515000606

[13] Blueshark. Accessed: 2018-05-08. [Online]. Available: http://e2c2.ict.usc.edu/

[14] Dashware. Accessed: 2018-05-14. [Online]. Available: http://www.dashware.net

58