Neuroflight: Next Generation Flight ControlFirmware

William Koch, Renato Mancuso, Azer BestavrosBoston University

Boston, MA 02215wfkoch, rmancuso, best@bu.edu

Abstract—Quadcopter aircraft are now mainstream and usedin a number of contexts, including surveillance, aerial mappingand surveying, natural disaster recovery, and search and rescueoperations. In the last decade, seminal results have been ac-complished in autonomous and semi-autonomous flight. Worksin this domain have extensively addressed challenges in stateestimation, trajectory tracking, localization, and the like. Suchhigh-level control objectives rely on, and are directly impactedby, the behavior of the inner-most control loop, namely attitudecontrol. Remarkably however, attitude control has received little-to-no attention in recent years. Flight control implementationspredominantly use the classical PID algorithm. PID is easyto implement, which resonates well with resource constrainedquadcopters. Nonetheless, PID has important limitations in termsof adaptability. Also, its performance is only as good as the tune.

In this work, we investigate if performing better than PID ispossible, while still abiding to the typical resource constraintsof small-scale quadcopters. For this purpose, we introduceNeuroflight, the first open-source neural network-based flightcontroller firmware. We present our toolchain for training a neu-ral network in simulation and compiling it to run on embeddedhardware. We discuss the main challenges faced when jumpingfrom simulation to reality along with the proposed solutions. Ourevaluation shows that the neural network controller can executeat over 2kHz on an ARM Cortex-M7 processor. We demonstratevia flight tests that a quadcopter running Neuroflight can achievestable flight and execute aerobatic maneuvers.

I. INTRODUCTION

Recently there has been explosive growth in user-levelapplications developed for unmanned aerial vehicles (UAVs).However little innovation has been made to the UAV’s low-level attitude flight controller which still predominantly usesclassic PID control. Although PID control has proven to besufficient for a variety of applications, it falls short in dynamicflight conditions and environments (e.g., in the presence ofwind, payload changes and voltage sags). In these cases, moresophisticated control strategies are necessary, that are able toadapt and learn. The use of neural networks (NNs) for flightcontrol (i.e., neuro-flight control) has been actively researchedfor decades to overcome limitations in other control algorithmssuch as PID control. However the vast majority of researchhas focused on developing autonomous neuro-flight controllerautopilots capable of tracking trajectories [1], [2], [3], [4], [5],[6], [7], [8]. A simple autopilot consists of an outer loop andan inner loop. The outer loop is responsible for generating

attitude1 and thrust command inputs to follow a specifictrajectory. The inner loop is responsible for maintaining stableflight and for reaching the attitude set points over time throughdirect manipulation of the aircraft’s actuators. Unlike the outerloop, the inner attitude control loop is mandatory for bothautonomous and manual flight. Moreover, the performance ofthe attitude controller have an important impact on the abilityto reach higher-level control objectives. This work is the firstto explore the adoption of neuro-flight control as an alterna-tive to PID for inner loop flight control. Because this workpioneers the adoption of NN attitude controllers, we herebystudy its performance independently from trajectory planning.Nonetheless, we acknowledge that studying the implication ofa better attitude controller on higher-level objectives is neededand planned as future work.

Recently an OpenAI gym [9] environment called GYMFC-V1 [10] was released. Via GYMFC-V1 it is possible to trainNNs attitude control of a quadcopter in simulation usingreinforcement learning (RL). Neuro-flight controllers trainedwith Proximal Policy Optimization (PPO) [11] were shown toexceed the performance of a PID controller. Nonetheless theattitude neuro-flight controllers were not validated in the realworld, thus it remained as an open question if the NNs trainedin GYMFC-V1 are capable of flight. As such, this work makesthe following contributions:

1) We introduce Neuroflight, the first open source neuro-flight controller firmware for multi-rotors and fixed wingaircraft. The NN embedded in Neuroflight replaces at-titude control and motor mixing commonly found intraditional flight control firmwares (Section III).

2) To train neuro-flight controllers capable of stable flightin the real world we introduce GYMFC-V1.5, a modifiedenvironment addressing several challenges in making thetransition from simulation to reality (Section IV).

3) We propose a toolchain for compiling a trained NN torun on embedded hardware. To our knowledge this isthe first work that consolidates a neuro-flight attitudecontroller on a microcontroller, rather than a multi-purpose onboard computer, thus allowing deployment onlightweight micro-UAVs (Section V).

1Defined as the orientation of the aircraft in terms of its angular velocityfor each roll, pitch, and yaw axis.

arX

iv:1

901.

0655

3v2

[cs

.RO

] 1

6 Se

p 20

19

4) Lastly, we provide an evaluation showing the NN canexecute at over 2kHz on an Arm Cortex-M7 processorand flight tests demonstrate that a quadcopter runningNeuroflight can achieve stable flight and execute aer-obatic maneuvers such as rolls, flips, and the Split-S (Section VI). Source code for the project can be foundat https://github.com/wil3/neuroflight and videos of ourtest flights can be viewed at https://wfk.io/neuroflight.

The goal of this work is to provide the community with astable platform to innovate and advance development of neuro-flight control design for UAVs, and to take a step towardsmaking neuro-flight controllers mainstream. In the future wehope to establish NN powered attitude control as a convenientalternative to classic PID control for UAVs operating in harshenvironments or that require particularly competitive set pointtracking performance (e.g., drone racing).

II. BACKGROUND AND RELATED WORK

Since the dawn of fly-by-wire, flight control algorithmshave continued to advance to meet increasing performancedemands [12], [13], [14]. In recent years a significant amountof research has investigated the use of NNs for flight controlwhich has advantages over classical control methods thanksto their ability to learn and plan.

Various efforts have demonstrated stable flight of a quad-copter through mathematical models using neuro-flight con-trollers to track trajectories. Online learning methods [2], [3]can learn quadcopter dynamics in real-time. Yet this requiresan initial learning period and flight performance behavior canbe unpredictable for rare occurring events. Offline supervisedlearning [1] can construct pre-trained neuro-flight controllerscapable of immediate flight. However realistic data can beexpensive to obtain and inaccuracies from the true aircraft canresult in suboptimal control policies.

RL is an alternative to supervised learning for offline learn-ing. It is ideal for sequential tasks in continuous environments,such as control and thus an attractive option for training neuro-flight controllers. RL consists of an agent (i.e., NN) interactingwith an environment to learn a task. At discrete time steps theagent performs an action (e.g., writes control signals to aircraftactuators) in the environment. In return the agent receives thecurrent state of the environment (obtained from various aircraftsensors which typically becomes the input to the NN) and anumerical reward representing the action’s performance. Theagent’s objective is to maximize its rewards.

Over time there has been a number of successes transferringcontrollers trained with RL to a UAVs onboard computer to au-tonomously track trajectories in the real world. Flight has beenachieved for both helicopters [4], [5], [6] and quadcopters [7],[8]. Unfortunately none of these works have published anycode thereby making it difficult to reproduce results and tobuild on top of their research. Furthermore evaluations areonly in respect to the accuracy of position therefore it is stillunknown how well attitude is controlled. Of the open sourceflight control firmwares currently available every single oneuses PID control [15].

Koch et. al [10] proposed an RL environment, GYMFC-V1,for developing attitude neuro-flight controllers which exceedaccuracy of a PID controller in regards to angular velocityerror. The GYMFC-V1 environment uses the Gazebo simula-tor [16], a high fidelity physics simulator, which contains adigital replica, or digital twin, of the aircraft, fixed about itscenter of mass to the simulation world one meter above theground allowing the aircraft to freely rotate in any direction.The angular velocity Ω(t) = [Ωφ(t),Ωθ(t),Ωψ(t)] for eachroll, pitch, and yaw axis of the aircraft is controlled bywriting pulse width modulation (PWM) values to the aircraftactuators. The agent is trained using episodic tasks (i.e., atask that has a terminal state). At the beginning of an episodictask a desired angular velocity Ω∗(t) is randomly sampled.The goal of the agent is to achieve this velocity in a finiteamount of time starting from still. At each time step an actiona(t) = [a0(t), . . . , aN−1(t)] is provided by the agent where Nis the number of aircraft actuators to be controlled (e.g., N = 4for a quadcopter) and ai(t) ∈ [1000, 2000] represents thePWM value. The agent is returned the state x(t) = (e(t), ω(t))where e(t) = Ω∗(t) − Ω(t) is the angular velocity errorand ω(t) = [ω0(t), . . . , ωN−1(t)] is the angular velocity ofeach actuator (e.g., for a quadcopter the RPM of the motor).Additionally a negative reward r is returned representing theangular velocity error. However evaluations were preformedin simulation thus it was unknown if neuro-flight controllerstrained by GYMFC-V1 could control a quadcopter in the realworld.

In this work we pick up where GYMFC-V1 left off. Weexplain in Section IV how without several modifications aNN trained with GYMFC-V1 will not be able to achievestable flight. With these modifications addressed in GYMFC-V1.5 we were able to generate attitude neuro-flight controllerscapable of high precision flight in the real world.

III. NEUROFLIGHT OVERVIEW

Neuroflight is a fork of Betaflight version 3.3.3 [17],a high performance flight controller firmware used exten-sively in first-person-view (FPV) multicopter racing. InternallyBetaflight uses a two-degree-of-freedom PID controller (notto be confused with rotational degrees-of-freedom) for atti-tude control and includes other enhancements such as gainscheduling for increased stability when battery voltage islow and throttle is high. Betaflight runs on a wide varietyof flight controller hardware based on the Arm Cortex-Mfamily of microcontrollers. Flight control tasks are scheduledusing a non-preemptive cooperative scheduler. The main PIDcontroller task consists of multiple subtasks, including: (1)reading the remote control (RC) command for the desiredangular velocity, (2) reading and filtering the angular velocityfrom the onboard gyroscope sensor, (3) evaluating the PIDcontroller, (4) applying motor mixing to the PID output toaccount for asymmetries in the motor locations (see [10] forfurther details on mixing), and (5) writing the motor controlsignals to the electronic speed controller (ESC).

Neuroflight replaces Betaflight’s PID controller task witha neuro-flight controller task. This task uses a single NNfor attitude control and motor mixing. The architecture ofNeuroflight decouples the NN from the rest of the firmwareallowing the NN to be trained and compiled independently.An overview of the architecture is illustrated in Fig. 1. Thecompiled NN is then later linked into Neuroflight to producea firmware image for the target flight controller hardware.

To Neuroflight, the NN appears to be a generic functiony(t) = f(x(t)). The input x(t) = [e(t),∆e(t)] where ∆e(t) =e(t) − e(t − 1). The output y(t) = [y0, . . . , yN−1] where Nis the number of aircraft actuators to be controlled and yi ∈[0, 1] is the control signal representing the percent power tobe applied to the ith actuator. This output representation isprotocol agnostic and is not compatible with NNs trained withGYMFC-V1 whose output is the PWM to be applied to theactuator. PWM is seldomly used in high performance flightcontrol firmware and has been replaced by digital protocolssuch as DShot for improved accuracy and speed [17].

At time t, the NN inputs are resolved; Ω∗(t) is read from theRX serial port which is either connected to a radio receiver inthe case of manual flight or an onboard companion computeroperating as an autopilot in the case of autonomous flight,and Ω(t) is read from the gyroscope sensor. The NN is thenevaluated to obtain the control signal outputs y(t). Howeverthe NN has no concept of thrust (T), therefore to achievetranslational movement the thrust command must be mixedinto the NN output to produce the final control signal outputto the ESC, u(t). The logic of throttle mixing is to uniformlyapply additional power across all actuators proportional to theavailable range in the NN output, while giving priority toachieving Ω∗(t). If any output value is over saturated (i.e.,∃yi(t) : yi(t) ≥ 1) no additional throttle will be added. Theinput throttle value is scaled depending on the available outputrange to obtain the actual throttle value:

T(t) = T(t) (1−maxiyi(t)) (1)

where the function max returns the max value from the NNoutput. The readjusted throttle value is then proportionallyadded to each NN output to form the final control signaloutput:

ui(t) = T(t) + yi(t). (2)

IV. GYMFC-V1.5

In this section we discuss the enhancements made toGYMFC-V1 to create GYMFC-V1.5. These changes primarilyconsist of a new state representation and reward system.

A. State Representation

GYMFC-V1 returns the state (e(t), ω(t)) to the agent ateach time step. However not all UAVs have the sensors tomeasure motor velocity ω(t) as this typically involves digitalESC protocols. Even in an aircraft with compatible hardware,including the absolute motor velocity as an input to the NNintroduces additional challenges. This is because a NN trained

on absolute RPMs does not easily transfer from simulationto the real world, unless an accurate propulsion subsystemmodel is available for the digital twin. A mismatch between thephysical propulsion system (i.e., motor/propeller combination)and the digital twin will result in the inability to achievestable flight. Developing an accurate motor models is time-consuming and expensive. Specialized equipment is requiredto capture the relations between voltage, power consumption,temperature, torque, and thrust.

To address these issues we investigated training using al-ternative environment states that do not rely on any specificcharacteristic of the motor(s). We posited that reducing theentire state to just angular velocity errors would carry enoughinformation for the NN to achieve stable flight. At the sametime, we expected that the obtained NN would transfer wellto the real aircraft. Thus, our NN is trained by replacing ω(t)with the error differences ∆e(t). To identify the performanceimpact of this design choice, we trained two NNs. A firstNN was trained in with ω(t) in input. Its behavior wascompared to a second NN trained in an environment thatprovides ∆e(t) instead. Both NNs were trained with PPOusing hyperparameters from [10] for 10 million steps. Aftertraining, each NN was validated against 10 never before seenrandom target angular velocities. Results show the NN trainedin an environment with x(t) = (e(t),∆e(t)) experienced onaverage 45.07% less error with only an increase of 3.41% inits control signal outputs.

In RL the interaction between the agent and the environmentcan be formally modeled as a Markov Decision Process (MDP)in which the probability that the agent transitions to the nextstate depends on its current state and action to be taken.The behavior of the agent is defined by its policy whichis essentially a mapping of states to actions. There may bemultiple different state representations that are able to mapto actions resulting in similar performance. For instance, itemerged from our experiments that using a history of errorsas input to the NN also led to satisfactory performance. Thisapproach has the disadvantage of requiring a state history tableto be maintained, which ultimately made the approach lessdesirable.

The intuition why a state representation comprised of onlyangular velocity errors works can be summarized as follows.First, note that a PD controller (a PID controller with theintegrative gain set to zero) is also a function computed overthe angular velocity error. Because an NN is essentially auniversal approximator, the expectation is that the NN wouldalso be able to find a suitable control strategy based on thesesame inputs.

Albeit required, however, modifying the environment statealone is not enough to achieve stable flight. The RL taskalso needs to be adjusted. Training using episodic tasks, inwhich the aircraft is at rest and must reach an angular velocitynever exposes the agent to scenarios in which the quadcoptermust return to still from some random angular velocity. Withthe new state input consisting of the previous state, this is asignificant difference from GYMFC-V1 which only uses the

Δ𝑒(𝑡)

simulator

RL training

synthesis & compilation

Neural Network Flight

Controller 𝑦 𝑡 = 𝑓(𝑥 𝑡 )

fly

motor signals

real-time

+

- Ω Ω∗

𝑒(𝑡)

𝑒(𝑡 − 1)

-

+

gyro

radio RX

Throttle Mix

𝑢(𝑡)

𝑇(𝑡)

𝑦(𝑡)

Fig. 1: Overview of the Neuroflight architecture.

current state. For this purpose, a continuous task is constructedto mimic real flight, continually issuing commands.2 This taskrandomly samples a command and sets the target angularvelocity to this command for a random amount of time. Thiscommand is then followed by an idle (i.e., Ω∗ = [0, 0, 0])command to return the aircraft to still for a random amount oftime. This is repeated until a max simulation time is reached.

B. Reward System

In the context of RL, reward engineering is the process ofdesigning a reward system in order to provide the agent witha numerical signal that they are doing the right thing [18].Reward engineering is a particularly difficult problem. Asreward systems increase in complexity, they may presentunintended side affects resulting in the agent behaving in anunexpected manner.

GYMFC-V1.5 reinforces stable flight behavior through ourreward system defined as:

r = re + ry + r∆. (3)

The agent is penalized for its angular velocity error, similarto GYMFC-V1, along each axis with:

re = −(e2φ + e2

θ + e2ψ). (4)

However we have identified the remaining two variablesin the reward system as critical for transferability to the realworld and achieving stable flight. Both rewards are a functionof the agents control output. First ry rewards the agent forminimizing the control output, and next, r∆ rewards the agentfor minimizing oscillations.

The rewards as a function of the control signal are able toaid in the transferability by compensating for limitations in thetraining environment and unmodeled dynamics in the motormodel.

2Technically this is still considered an episodic task since the simulationtime is finite. However in the real world flight time is typically finite as well.

Minimizing Output Oscillations In the real world highfrequency oscillations in the control output can damage mo-tors. Rapid switching of the control output causes the ESCto rapidly change the angular velocity of the motor drawingexcessive current into the motor windings. The increase in cur-rent causes high temperatures which can lead to the insulationof the motor wires to fail. Once the motor wires are exposedthey will produce a short and “burn out” the motor.

The reward system used by GYMFC-V1 is strictly afunction of the angular velocity error. This is inadequate indeveloping neuro-flight controllers that can be used in thereal world. Essentially this produces controllers that closelyresemble the behavior of an over-tuned PID controller. Thecontroller is stuck in a state in which it is always correctingitself, leading to output oscillation.

In order to construct networks that produce smooth controlsignal outputs, the control signal output must be introducedinto the reward system. This turned out to be quite challenging.Ultimately we were able to construct NNs outputting stablecontrol outputs with the inclusion of the following reward:

r∆ = β

N−1∑i=0

max0,∆ymax − (∆yi)2. (5)

This reward is only applied if the absolute angular velocityerror for every axis is less than some threshold (i.e., the errorband). This allows the agent to be signaled by re to reach thetarget without the influence from this reward. Maximizing r∆

will drive the agent’s change in output to zero when in theerror band. To derive r∆, the change in the control output yifrom the previous simulation step is squared to magnify theeffect. This is then subtracted from a constant ∆ymax definingan upper bound for the change in the control output. The maxfunction then forces a positive reward, therefore if (∆yi)

2

exceeds the limit no reward will be given. The rewards foreach control output N − 1 are summed and then scaled bya constant β, where β > 0. Using the same training andvalidation procedure previously discussed, we found a NN

trained in GYMFC-V1.5 compared to GYMFC-V1 resultedin a 87.95% decrease in ∆y.

Minimizing Control Signal Output Values Recall fromSection II, that the GYMFC-V1 environment fixes the aircraftto the simulation world about its center of mass, allowing itto only perform rotational movements. Due to this constraintthe agent can achieve Ω∗ with a number of different controlsignal outputs (e.g., when Ω∗ = [0, 0, 0] this can be achievedas long as y0 ≡ y1 ≡ y2 ≡ y3). However this poses asignificant problem when transferred to the real world as anaircraft is not fixed about its center of mass. Any additionalpower to the motors will result in an unexpected change intranslational movement. This is immediately evident whenarming the quadcopter which should remain idle until RCcommands are received. At idle, the power output (typically4% of the throttle value) must not result in any translationalmovement. Another byproduct of inefficient control signalsis a decreased throttle range (Section III). Therefore it isdesirable to have the NN control signals minimized while stillmaintaining the desired angular velocity. In order to teach theagent to minimize control outputs we introduce the rewardfunction:

ry = α (1− y) (6)

providing the agent a positive reward as the output decreases.Since yi ≤ 1 we first compute the average output y. Next 1−yis calculated as a positive reward for low output usage whichis scaled by a constant α, where α > 0. NNs trained usingthis reward experience on average a 90.56% decrease in theircontrol signal output.

Challenges and Lessons Learned The fundamental chal-lenge we faced was managing high amplitude oscillations inthe control signal. In stochastic continuous control problemsit is standard for the network to output the mean from a Gaus-sian distribution [11], [19]. However this poses problems forcontrol tasks with bounded outputs such as flight control. Thetypical strategy is to clip the output to the target bounds yetwe have observed this to significantly contribute to oscillationsin the control output.

Through our experience we learned that due to the outputbeing stochastic (which aids in exploration), the rewardsmust encapsulate the general trend of the performance andnot necessarily at a specific time (e.g., the stochastic outputnaturally oscillates). Additionally we found the reward systemmust include performance metrics other than (but possibly inaddition to) traditional time domain step response character-istics (e.g., overshoot, rise time, settling time, etc.). Given theagent initially knows nothing, there is no step response toanalyze. In future work we will explore the use of goal basedlearning in an attempt to develop a hybrid solution in whichthe agent learns enough to track a step response, then usetraditional metrics for fine tuning.

Although our reward system was sufficient in achievingflight, we believe this is still an open area of research worthexploring. In addition to aforementioned rewards, we experi-mented with several other rewards including penalties for over

GymFC

freeze_graphtransform_graph

OpenAI Baselines

tfcompileArm Toolchain

compiled

STAGE 1 Training

STAGE 2 Optimization

STAGE 3 Compilation

frozen

RL

checkpoint

Neuroflight source

optimized

Digital Twin

firmware image

Fig. 2: Overview of the Neuroflight toolchain. Our maincontributions are in the gray boxes while boxes with dashedborders indicate minor modifications to existing software.

saturation of the control output (i.e., if the network outputexceeded the clipped region), control output jerk (i.e., changein acceleration), and the number of oscillations in the output.When combining multiple rewards, balancing and tuning theweight can be an exercise of its own. For example if penalizingfor number of oscillations or jerk this can lead to an outputthat resembles a low frequency square wave if penalizingthe amplitude is not considered. We also experimented withpositive binary rewards for agent continually making progressto the desired setpoint however this appear to not provide theagent with enough information to converge.

V. TOOLCHAIN

In this section we introduce our toolchain for building theNeuroflight firmware. Neuroflight is based on the philosophythat each flight control firmware should be customized for thetarget aircraft to achieve maximum performance. To train aNN optimal attitude control of an aircraft, a digital represen-tation (i.e., a digital twin) of the aircraft must be constructedto be used in simulation. This work begins to address howdigital twin fidelity affects flight performance, however it isstill an open question that we will address in future work.The toolchain displayed in Fig. 2 consists of three stages andtakes as input a digital twin and outputs a Neuroflight firmwareunique to the digital twin. In the remainder of this section wewill discuss each stage in detail.

A. Training

The training stage takes as input a digital twin of an aircraftand outputs a NN trained attitude control of the digital twincapable of achieving stable flight in the real world. Ourtoolchain can support any RL library that interfaces withOpenAI environment APIs and allows for the NN state to besaved as a Tensorflow graph. Currently our toolchain uses RLalgorithms provided by OpenAI baselines [20] which has beenmodified to save the NN state. In Tensorflow, the saved stateof a NN is known as a checkpoint and consists of three files

describing the structure and values in the graph. Once traininghas completed, the checkpoint is provides as input to Stage 2:Optimization.

B. Optimization

The optimization stage is an intermediate stage betweentraining and compilation that prepares the NN graph to berun on hardware. The optimization stage (and compilationstage) require a number of Tensorflow tools which can allbe found in the Tensorflow repository [21]. The first step inthe optimization stage is to freeze the graph. Freezing thegraph accomplishes two tasks: (1) condenses the three check-point files into a single Protobuf file by replacing variableswith their equivalent constant values (e.g., numerical weightvalues) and (2) extracts the subgraph containing the trainedNN by trimming unused nodes and operations that wereonly used during training. Freezing is done with Tensorflow’sfreeze_graph.py tool which takes as input the checkpointand the output node of the graph so the tool can identify andextract the subgraph.

Unfortunately the Tensorflow input and output nodes are notdocumented by RL libraries (OpenAI baselines [20], Stablebaselines [22], TensorForce [23]) and in most cases it isnot trivial to identify them. We reverse engineered the graphproduced by OpenAI baselines (specifically the PPO1 imple-mentation) using a combination of tools and cross referencingwith the source code. A Tensorflow graph can be visuallyinspected using Tensorflow’s Tensorboard tool. OpenAI base-lines does not support Tensorboard thus we created a scriptto convert a checkpoint to a Probobuf file and then usedTensorflow’s import_pb_to_tensorboard.py tool toview the graph in Tensorboard. Additionally we used Tensor-flow’s summarize_graph tool to summarize the inputs andoutputs of the graph. Ultimately we identified the input nodeto be “pi/ob”, and the output to be “pi/pol/final/BiasAdd”.

Once the graph is frozen, it is optimized to run on hard-ware by running the Tensorflow transform_graph tool.Optimization provided by this tool allows graphs to executefaster and reduce its overall footprint by further removingunnecessary nodes. The optimized frozen ProtoBuf file isprovided as input to Stage 3: Compilation.

C. Compilation

A significant challenge was developing a method to in-tegrate a trained NN into Neuroflight to be able to run onthe limited resources provided by a microcontroller. The mostpowerful of the microcontrollers supported by Betaflight andNeuroflight consists of 1MB of flash memory, 320KB ofSRAM and an ARM Cortex-M7 processor with a clock speedof 216MHz [24]. Recently there has been an increase in inter-est for running NNs on embedded devices but few solutionshave been proposed and no standard solution exists. We foundTensorflow’s tool tfcompile to work best for our toolchain.tfcompile provides ahead-of-time (AOT) compilation ofTensorflow graphs into executable code primarily motivatedas a method to execute graphs on mobile devices. Normally

executing graphs requires the Tensorflow runtime which isfar too heavy for a microcontroller. Compiling graphs usingtfcompile does not use the Tensoflow runtime which resultsin a self contained executable and a reduced footprint.

Tensorflow uses the Bazel [25] build system and expectsyou will be using the tfcompile Bazel macro in yourproject. Neuroflight on the other hand is using make withthe GNU Arm Embedded Toolchain. Thus it was necessaryfor us to integrate tfcompile into the toolchain by callingthe tfcompile binary directly. When invoked, an objectfile representing the compiled graph and an accompanyingheader file is produced. Examining the header file we iden-tified three additional Tensorflow dependencies that mustbe included in Neuroflight (typically this is automaticallyincluded if using the Bazel build system): the AOT run-time (runtime.o), an interface to run the compiled func-tions (xla_compiled_cpu_function.o), and runningoptions (executable_run_options.o) for a total of24.86 KB. In Section VI we will analyze the size of thegenerated object file for the specific neuro-flight controller.

To perform fast floating point calculations Neuroflight mustbe compiled with ARM’s hard-float application binary inter-face (ABI). Betaflight core, inherited by Neuroflight alreadydefines the proper compilation flags in the Makefile howeverit is required that the entire firmware must be compiled withthe same ABI meaning the Tensorflow graph must also becompiled with the same ABI. Yet tfcompile does notcurrently allow for setting arbitrary compilation flags whichrequired us to modify the code. Under the hood, tfcompileuses the LLVM backend for code generation. We were able toenable hard floating points through the ABIType attribute inthe llvm::TargetOptions class.

VI. EVALUATION

In this section we evaluate Neuroflight controlling a highperformance custom FPV racing quadcopter named NF1. Firstand foremost, we show that it is capable of maintaining stableflight. Additionally, we demonstrate that the synthesized NNcontroller is also able to stabilize the aircraft even whenexecuting advanced aerobatic maneuvers. Images of NF1 andits entire build log have been published to RotorBuilds [26].

A. Firmware Construction

We used the Iris quadcopter model included with theGazebo simulator (which is also used by GYMFC-V1) withmodifications to the motor model to more accurately reflectNF1 for our digital twin. The digital twin motor model usedby Gazebo is quite simple. Each control signal is multipliedby a maximum rotor velocity constant to derive the targetrotor velocity while each rotor is associated with a PIDcontroller to achieve this target rotor velocity. We obtained anestimated maximum 33,422 RPMs for our propulsion systemfrom Miniquad Test Bench [27] to update the maximum rotorvelocity constant. We also modified the rotor PID controller(P=0.01, I=1.0) to achieve a similar throttle ramp.

(a) Screenshot of the Iris quadcopter flying in simulation. (b) Still frame of the FPV video footage acquired during a test flight.

Fig. 3: Flight in simulation (left) and in the real world (right).

Iris NF1Weight 1282g 432gWheelbase 550mm 212mmPropeller 10:47x2 51:52x3Motor 28x30 850Kv 22x04 2522Kv

Battery 3-cell 3.5Ah LiPo 4-cell 1.5Ah LiPoFlight Controller F4 F7

TABLE I: Comparison between Iris and NF1 specifications.

NF1 is in stark contrast with the Iris quadcopter modelused by GYMFC-V1 which is advertised for autonomous flightand imaging [28]. We have provided a visual comparison inFig. 4 and a comparison between the aircraft specifications inTable I. In this table, weight includes the battery, while thewheelbase is the motor to motor diagonal distance. Propellerspecifications are in the format “LL:PPxB” where LL is thepropeller length in inches, PP is the pitch in inches and Bis the number of blades. Brushless motor sizes are in theformat “WWxHH” where WW and HH is the stator widthand height respectively. The motors Kv value is the motorvelocity constant and is defined as the inverse of the motorsback-EMF constant which roughly indicates the RPMs per volton an unloaded motor [29]. Flight controllers are classifiedby the version of the embedded ARM Cortex-M processorprefixed by the letter ‘F’ (e.g., F4 flight controller uses anARM Cortex-M4).

Our NN architecture consists of 6 inputs, 4 outputs, 2 hiddenlayers with 32 nodes each using hyperbolic tangent activationfunctions resulting in a total of 1344 tunable weights.The network outputs the mean of a Gaussian distributionwith a variable standard deviation as defined by PPO forcontinuous domains [11]. Training was performed with theOpenAI Baseline version 0.1.4 implementation of PPO1 dueto its previous success in [10] which showed PPO to outperform Deep Deterministic Policy Gradient (DDPG) [30],and Trust Region Policy Optimization (TRPO) [31] in regardsto attitude control in simulation. A picture of the quadcopterduring trained in GYMFC-V1.5 can be seen in Fig. 3a.

(a) Iris (b) NF1

Fig. 4: Iris simulated quadcopter compared to the NF1 realquadcopter.

The reward system hyperparameters used were α = 300,β = 0.5, and ∆ymax = 1002 and the PPO hyperparametersused are reported in Table II. The reward hyperparameter∆ymax is defined as the maximum delta in the output we arewilling to accept, while α and β were found through experi-mentation to find the desired balance between minimizing theoutput and minimizing the output oscillations. The discountand Generalized Advantage Estimate (GAE) parameters weretaken from [11] while the remaining parameters were foundusing random search. The agent was particularity sensitive tothe selection of the horizon and minibatch size. To account forsensor noise in the real world we added noise to the angularvelocity measurements which was sampled from a Gaussiandistribution with µ = 0 and σ = 5. The standard deviationwas obtained by incrementing σ until it began to impact thecontrollers ability to track the set point. We observed this toreduce motor oscillations in the real world.

Each training task/episode ran for 30 seconds in simulation.The simulator is configured to take simulation steps every 1mswhich results in a total of 30,000 simulation steps per episode.Training ran for a total of 10 million time steps (333 episodes)on a desktop computer running Ubuntu 16.04 with an eight-core i7-7700 CPU and an NVIDIA GeForce GT 730 graphics

Hyperparameter ValueHorizon (T) 500Adam stepsize 1× 10−4 × ρ

Num. epochs 5Minibatch size 32Discount (γ) 0.99GAE parameter (λ) 0.95

TABLE II: PPO Hyperparameters where ρ is linearly annealedover the course of training from 1 to 0.

card which took approximately 11 hours. However trainingconverged much earlier at around 1 million time steps (33episodes) in just over an hour (Fig. 5). We trained a totalof three NNs which each used a different random seed forthe RL training algorithm and selected the NN that receivedthe highest cumulative reward to use in Neuroflight. Fig. 5shows a plot of the cumulative rewards of each training episodefor each of the NNs. The plot illustrates how drastic trainingepisodes can vary simply due to the use of a different seed.

The optimization stage reduced the frozen Tensorflow graphof the best performing NN by 16% to a size of 12KB.The graph was compiled with Tensorflow version 1.8.0-rc1and the firmware was compiled for the MATEKF722 targetcorresponding to the manufacturer and model of our flight con-troller MATEKSYS Flight Controller F722-STD. Our flightcontroller uses the STM32F722RET6 microcontroller with512KB flash memory, and 256KB of SRAM.

We inspected the .text, .data and .bss section headersof the firmware’s ELF file to derive a lower bound of thememory utilization. These sections totalled 380KB, resultingin at least 74% utilization of the flash memory. Graph op-timization accounted for a reduction of 280B, all of whichwas reduced from the .text section. Although in terms ofmemory utilization the optimization stage was not necessary,this however will be more important for larger networks inthe future. Comparing this to the parent project, Betaflight’ssections totalled 375KB.

Using Tensorflow’s benchmarking tool we performed onemillion evaluations of the graph with and without optimizationand found the optimization processes to reduce execution timeon average by 1.1µs.

B. Timing Analysis

Running a flight control task with a fast control rate allowsfor the use of a high speed ESC protocol, reducing writelatency to the motors and thus resulting in higher precisionflight. Therefore it is critical to analyze the execution time ofthe neuro-flight control task so the optimal control rate of thetask can be determined. Once this is identified it can be used toselect which ESC protocol will provide the best performance.We collect timing data for Neuroflight and compare this toits parent project Betaflight. Times are taken for when thequadcopter is disarmed and also armed under load for thecontrol algorithm (i.e., evaluation of the NN and PID equation)and also the entire flight control task which in addition to the

0 50 100 150 200 250 300Episode

2.0

1.5

1.0

0.5

0.0

Cum

ulat

ive

Rewa

rd

1e9

Trial 1Trial 2Trial 3

Fig. 5: Cumulative rewards for each training episode.

WCET (µs) BCET (µs) Var. Window (%)

DisarmedNF 204 194 4.9BF 14 9 35.7

ArmedNF 210 195 7.1BF 15 9 40.0

TABLE III: Timing analysis of the control algorithm used inNeuroflight (NF) and Betaflight (BF).

control algorithm includes reading the gryo, reading the RCcommands and writing to the motors.

We instrumented the firmware to calculate the timing mea-surement and wrote the results to an unused serial port onthe flight control board. Connecting to the serial port onthe flight control board via an FTDI adapter we are ableto log the data on an external PC running minicom. Werecorded 5,000 measurements and report the worst-case ex-ecution time (WCET), best-case execution time (BCET) andthe variability window in Table III for the control algorithmand Table IV for the control task. The variability window iscalculated as the difference between the WCET and BCET,normalized by the WCET, i.e., (WCET−BCET)/WCET. Thisprovides indication of how predicable is the execution of theflight control logic, as it embeds information about the relativefluctuation of execution times. Two remarks are importantwith respect to the results in Table III. First, the NN traversalcompared to PID is about 14x slower (armed case), albeitthe predictability of the controller increases. It is important toremember that, while executing the PID is much simpler thanevaluating an NN, our approach allows removing additionallogic that is required by a PID, such as motor mixing. Thus,a more meaningful comparison needs to be performed bylooking at the overall WCET and predictability of the wholeflight control task, which we carry out in Table IV. Second,because NN evaluation always involve the same exact steps, animprovement in terms of predictability can be observed underNeuroflight.

The timing analysis reported in Table IV reveals that the

WCET (µs) BCET (µs) Var. Window (%)

DisarmedNF 244 229 6.1BF 58 45 22.4

ArmedNF 423 263 37.8BF 238 78 67.2

TABLE IV: Timing analysis of the flight control task used inNeuroflight (NF) and Betaflight (BF).

neuro-flight control task has a WCET of 423 µs which wouldallow for a max execution rate of 2.4kHz. However in Neu-roflight (and in Betaflight), the flight control task frequencymust be a division of the gyro update frequency, thus with4kHz gyro update and a denominator of 2, the neuro-flightcontrol task can be configured to execute at 2kHz. To put thisinto perspective this is 8 times faster3 than the popular PX4firmware [32].

Furthermore this control rate is 40 times faster than the tra-ditional PWM ESC protocol used by commercial quadcopters(50Hz [33]) thereby allowing us to configure Neuroflight touse the ESC protocol DShot600 which has a max frequencyof 37.5kHz [34].

Given the simplicity of the PID algorithm it came as nosurprise that the Betaflight flight control task is faster, yetthis is only by a factor of 1.78 when armed. As we can seecomparing Table III to Table IV the additional subprocessestasks are the bottleneck of the Betaflight flight control ask.However referring to the variability window, the Neuroflightcontrol algorithm and control task are far more stable thanBetaflight. The Betaflight flight control task exhibits littlepredictability when armed.

Recent research has shown there is no measurable improve-ments for control task loop rates that are faster than 4kHz [33].Our timing analysis has shown that Neuroflight is close of thisgoal. To reach this goal there are three approaches we cantake: (1) Support future microcontrollers with faster processorspeeds, (2) experiment with different NN architectures toreduce the number of arithmetic operations and thus reducethe computational time to execute the NN, and (3) optimizethe flight control sub tasks to reduce the flight control task’sWCET and variability window. In future work we immediatelyplan to explore (2) and (3), results obtained in these directionswould not depend on the specific hardware used in the finalassembly.

C. Power Analysis

The flight controller affects power consumption directly andindirectly. The direct power draw is a result of the executionof the control algorithm/task, while the indirect power drawis due to the generated control signals which determines theamount of power the ESC will draw.

As a first attempt to understand and compare the powerconsumption of a NN based controller to a standard PIDcontroller, we performed a static power analysis. For NF1running Neuroflight, we connected a multimeter inline with

3According to the default loop rate of 250Hz.

Voltage (V) Current (A) Power (W)

DisarmedNF 16.78 0.37 6.21BF 16.78 0.37 6.21

ArmedNF 16.78 0.67 11.24BF 16.78 0.6 10.07

TABLE V: Power analysis of Neuroflight (NF) compared toBetaflight (BF).

the battery power supply to measure the current draw andreport the measurements for both when the quadcopter is dis-armed (direct power consumption) and armed idling (indirectpower consumption), similarly done to our timing analysis.We then take the same measurements for the NF1 runningBetaflight (PID control). Results reported in Table V showthere is no change using the NN based controller in regardsto direct power draw of the control algorithm. This result wasexpected as the flight control firmware does not execute sleepinstructions. However for the indirect power draw, there isa measurable 70mA (approximately 11%) increase in currentdraw for the NN controller. It is important to remember thisparticular NN controller has been trained to optimize its abilityto track a desired angular velocity. Thus the increase in currentdraw does not come as a surprise as the control signals willbe required to switch quickly to maintain the set point whichresults in increased current draw.

An advantage a NN controller has over a traditional PIDcontroller is that it has the ability to optimize its performancebased on a number of conditions and characteristics, suchas power consumption. In the future we will investigatealternative optimization goals for the controller and instrumentNF1 with sensors to record power consumption in flight toperform a thorough power analysis.

D. Flight Evaluation

To test the performance of Neuroflight we had an expe-rienced drone racing pilot conduct test flights for us. TheFPV videos of the test flights can be viewed at https://wfk.io/neuroflight. A still image extracted from the FPV video feedshows the view point of the pilot of one of the test flights canbe seen in Fig. 3b. In FPV flying the aircraft has a camerawhich transmits the analog video feed back to the pilot who iswearing goggles with a monitor connected to a video receiver.This allows the pilot to control the aircraft from the perspectiveof the aircraft.

Neuroflight supports real-time logging during flight allow-ing us to collect gyro and RC command data to analyze howwell the neuro-flight controller is able to track the desiredangular velocity. We asked the pilot to fly a mix of basicmaneuvers such as loops and figure eights and advancedmaneuvers such as rolls, flips, dives and the Split-S. To executea Split-S the pilot inverts the quadcopter and descends ina half loop dive, exiting the loop so they are flying in theopposite horizontal direction. Once we collected the flightlogs we played the desired angular rates back to the NN inthe GYMFC-V1.5 environment to evaluate the performance

Fig. 6: Flight test log demonstrating Neuroflight tracking a desired angular velocity in the real world compared to in simulation.Maneuvers during this flight are annotated.

NN Controller (PPO)Metric Roll (φ) Pitch(θ) Yaw (ψ) AverageMAE 15 21 11 15MSE 1,010 595 282 629IAE 15,094 20,946 11,203 15,748ISE 1,005,923 592,318 281,548 626,596ITAE 911,269 1,246,625 678,602 945,499ITSE 52,904,984 36,036,125 18,023,064 35,654,724

TABLE VI: Performance metrics of NN controller from flightin the real world. Metric is reported for each individual axis,along with the average. Lower values are better.

in simulation. Comparison between the simulated and realworld performance is illustrated in Fig. 6 while specificmaneuvers that occur during this test flight are annotated. Wecomputed various control measures of the flight performanceincluding the Mean Absolute Error (MAE), and Mean SquaredError (MSE), as well as the descrete form of the IntegralAbsolute Error (IAE), Integral Squared Error (ISE), IntegralTime-weighted Absolute Error (ITAE), and Integral Time-weighted Squared Error (ITSE). These values are reported inTable VI for the real flight and in Table VII for the simulatedflight .

As we can see there is an increase in error transferring fromsimulation from reality, however this was expected becausethe digital twin does not perfectly model the real system. TheMAE is not significant however there is a large increase inerror for the integral measurements. A partial explanation forthis is if we refer to Fig. 6 (particularly the pitch axis) wecan see the controller is consistently off by about 10 degrees

which will continually add error to these measurements.The increased error on the pitch axis appears to be due to

the differences in frame shape between the digital twin andreal quadcopter, which are both asymmetrical but in relationto a different axis. This discrepancy may have resulted in pitchcontrol lagging in the real world as more torque and power isrequired to pitch in our real quadcopter.

A more accurate digital twin model can boost accuracy.Furthermore, during this particular flight wind gusts exceeded30mph, while in the simulation world there are no externaldisturbances acting upon the aircraft. In the future we planto deploy an array of sensors to measure wind speed so wecan correlate wind gusts with excessive error. Nonetheless, asshown in the video, stable flight can be maintained demon-strating the transferability of a NN trained with our approach.

PID vs NN Control. Next we performed an experiment tocompare the performance of the NN controller used in Neu-roflight to a PID controller in simulation using the GYMFC-V1.5 environment. Although other control algorithms mayexist in literature that out perform PID, of the open sourceflight controllers avail be for benchmarking, every single oneuses PID [15]. A major contribution of this work is providingthe research community an additional flight control algorithmfor benchmarking.

The PID controller was first tuned in the simulator usingthe classical Ziegler-Nichols method [35] and then manuallyadjusted to reduce overshoot to obtained the following gainsfor each axis of rotation: Kφ = [0.032029, 0, 0.000396],Kθ = [0.032029, 0, 0.000396], Kψ = [0.032029, 0, 0], whereKaxis = [Kp,Ki,Kd] for each proportional, integrative, and

NN Controller (PPO) PIDMetric Roll (φ) Pitch(θ) Yaw (ψ) Average Roll (φ) Pitch(θ) Yaw (ψ) AverageMAE 2.89 1.52 4.07 2.83 3.90 5.26 3.42 4.20MSE 23.23 5.59 27.2 18.67 34.81 45.59 20.55 33.65IAE 2,887.55 1,522.62 4,071.63 2,827.26 3,904.83 5,258.45 3,423.00 4,195.42ISE 23,226.56 5,588.60 27,203.42 18,672.86 34,811.32 45,589.85 20,548.85 33,650.01ITAE 179,944.50 93,339.33 261,947.22 178,410.35 236,408.09 320,204.59 217,343.01 257,985.23ITSE 1,499,075.54 369,576.98 1,893,954.11 1,254,202.21 2,100,576.25 2,927,031.10 1,419,390.55 2,148,999.30

TABLE VII: Performance metric comparison of the simulation evaluation for the NN controller and PID controller. Metric isreported for each individual axis, along with the average. Lower values are better.

−400

−200

0

Rol

l(d

eg/s

)

Ω∗φPID

NN

−200

−100

0

100

Pit

ch(d

eg/s

)

Ω∗θPID

NN

20.6 20.8 21.0 21.2 21.4

Time (s)

−100

0

100

200

Yaw

(deg

/s)

Ω∗ψPID

NN

(a) Roll

−200

0

Rol

l(d

eg/s

)

Ω∗φPID

NN

−200

−100

0

100

Pit

ch(d

eg/s

)

Ω∗θPID

NN

101.25 101.50 101.75 102.00 102.25 102.50 102.75 103.00 103.25

Time (s)

−100

0

100

Yaw

(deg

/s)

Ω∗ψPID

NN

(b) Split-S

Fig. 7: Performance comparison of the NN controller versus aPID controller tracking a desired angular velocity in simulationto execute the Split-S and roll aerobatic maneuvers.

derivative gains, respectively. It took approximately a half hourto manually tune the 9 gains with the bottleneck being the timeto execute the simulator in order to obtain the parameters tocalculate Ziegler-Nichols. In comparison to training a NN viaPPO, there is not a considerable overhead difference given thisis an offline task. In fact the tuning rate by PPO is significantlyfaster by a factor of 75.

The RC commands from the real test flight where thenreplayed back to the simulator similar to the previous exper-iment, however this time using the tuned PID controller. Azoomed in comparison of the NN and PID controller trackingthe desired angular velocity for two aerobatic maneuvers isshown in Fig. 7. Although the performance is quite close,we can most visibly the NN controller tracking the pitch axisduring a Split-S maneuver more accurately.

We also computed the same control measurements for thePID controller and reported them in Table VII. Results show,on average, the NN controller to out perform the PID controllerfor every one of our metrics.

It is important to note PID tuning is a challenging taskand the PID controller’s accuracy and ability to control thequadcopter is only as good as the tune. The NN controller onthe other hand did not require any manually tuning, insteadthrough RL and interacting with the aircraft over time it is ableto teach itself attitude control. As we continue to the reduce thegap between simulation and the real world, the performance ofthe NN controller will continue to improve in the real world.

VII. FUTURE WORK AND CONCLUSION

In this work we introduced Neuroflight, the first open-sourceneuro-flight control firmware for remote piloting multi-coptersand fixed wing aircraft and its accompanying toolchain. Thereare four main directions we plan to pursue in future work.

1) Digital twin development. In this work we synthesizedour NN using an existing quadcopter model that didnot match NF1. Although stable flight was achieveddemonstrating the NNs robustness, comparison betweenthe simulated flight verse the actual flight is evidenceinaccuracies in the digital twin has a negative affect inflight control accuracy. In future work we will developan accurate digital twin of NF1 and investigate how thefidelity of a digital twin affects flight performance in aneffort to reduce costs during development.

2) Adaptive and predictive control. With a stable platformin place we can now begin to harness the NN’s true

potential. We will enhance the training environment toteach adaptive control to account for excessive sensornoise, voltage sag, change in flight dynamics due tohigh throttle input, payload changes, external disturbancessuch as wind, and propulsion system failure.

3) Continuous learning. Our current approach trains NNsexclusively using offline learning. However, in order toreduce the performance gap between the simulated andreal world, we expect that a hybrid architecture involvingonline incremental learning will be necessary. Onlinelearning will allow the aircraft to adapt, in real-time,and to compensate for any modelling errors that existedduring synthesis of the NN during offline (initial) training.Given the payload restrictions of micro-UAVs and weightassociated with hardware necessary for online learningwe will investigate methods to off-load the computationalburden of incremental learning to the cloud.

4) NN architecture development Several performancebenefits can be realized from an optimal network architec-ture for flight control including improved accuracy (Sec-tion VI-D) and faster execution (Section VI-B). In futurework we plan to explore recurrent architectures utilizinglong short-term memory (LSTM) to improve accuracy.Additionally we will investigate alternative distributionssuch as the beta function which is naturally bounded [19].Furthermore we will explore the use of the rectified linearunit (ReLU) activation functions to increase executiontime which is more computationally efficient than thehyperbolic tangent function.

The economic costs associated with developing neuro-flightcontrol will foreshadow its future, determining whether itsuse will remain confined to special purpose applications, orif it will be adopted in mainstream flight control architectures.Nonetheless, we strongly believe that Neuroflight is a majormilestone in neuro-flight control and will provide the requiredfoundations for next generation flight control firmwares.

REFERENCES

[1] J. F. Shepherd III and K. Tumer, “Robust neuro-control for a microquadrotor,” in Proceedings of the 12th annual conference on Geneticand evolutionary computation. ACM, 2010, pp. 1131–1138.

[2] C. Nicol, C. Macnab, and A. Ramirez-Serrano, “Robust neural networkcontrol of a quadrotor helicopter,” in Electrical and Computer Engineer-ing, 2008. CCECE 2008. Canadian Conference on. IEEE, 2008, pp.001 233–001 238.

[3] T. Dierks and S. Jagannathan, “Output feedback control of a quadrotoruav using neural networks,” IEEE transactions on neural networks,vol. 21, no. 1, pp. 50–66, 2010.

[4] J. A. Bagnell and J. G. Schneider, “Autonomous helicopter control usingreinforcement learning policy search methods,” in Robotics and Automa-tion, 2001. Proceedings 2001 ICRA. IEEE International Conference on,vol. 2. IEEE, 2001, pp. 1615–1620.

[5] H. J. Kim, M. I. Jordan, S. Sastry, and A. Y. Ng, “Autonomous helicopterflight via reinforcement learning,” in Advances in neural informationprocessing systems, 2004, pp. 799–806.

[6] P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng, “An application ofreinforcement learning to aerobatic helicopter flight,” in Advances inneural information processing systems, 2007, pp. 1–8.

[7] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotorwith reinforcement learning,” IEEE Robotics and Automation Letters,vol. 2, no. 4, pp. 2096–2103, 2017.

[8] S. R. B. dos Santos, S. N. Givigi, and C. L. N. Junior, “An experimentalvalidation of reinforcement learning applied to the position control ofuavs,” in 2012 IEEE International Conference on Systems, Man, andCybernetics (SMC). IEEE, 2012, pp. 2796–2802.

[9] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schul-man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprintarXiv:1606.01540, 2016.

[10] W. Koch, R. Mancuso, R. West, and A. Bestavros, “Reinforcementlearning for uav attitude control,” ACM Transactions on Cyber-PhysicalSystems, vol. 3, no. 2, p. 22, 2019.

[11] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox-imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347,2017.

[12] D. J. Leith and W. E. Leithead, “Survey of gain-scheduling analysis anddesign,” International journal of control, vol. 73, no. 11, pp. 1001–1025,2000.

[13] N. Hovakimyan, C. Cao, E. Kharisov, E. Xargay, and I. M. Gregory,“L1 adaptive control for safety-critical systems,” IEEE Control Systems,vol. 31, no. 5, pp. 54–104, 2011.

[14] D. Lee, H. J. Kim, and S. Sastry, “Feedback linearization vs. adaptivesliding mode control for a quadrotor helicopter,” International Journalof control, Automation and systems, vol. 7, no. 3, pp. 419–428, 2009.

[15] E. Ebeid, M. Skriver, K. H. Terkildsen, K. Jensen, and U. P. Schultz,“A survey of open-source uav flight controllers and flight simulators,”Microprocessors and Microsystems, vol. 61, pp. 11–20, 2018.

[16] N. Koenig and A. Howard, “Design and use paradigms for gazebo, anopen-source multi-robot simulator,” in Intelligent Robots and Systems,2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ International Confer-ence on, vol. 3. IEEE, 2004, pp. 2149–2154.

[17] “BetaFlight,” https://github.com/betaflight/betaflight, 2018, accessed:2018-11-25.

[18] D. Dewey, “Reinforcement learning and the reward engineering princi-ple,” in 2014 AAAI Spring Symposium Series, 2014.

[19] P.-W. Chou, D. Maturana, and S. Scherer, “Improving stochastic policygradients in continuous control with deep reinforcement learning usingthe beta distribution,” in Proceedings of the 34th International Confer-ence on Machine Learning-Volume 70. JMLR. org, 2017, pp. 834–843.

[20] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford,J. Schulman, S. Sidor, and Y. Wu, “Openai baselines,” https://github.com/openai/baselines, 2017.

[21] “Tensorflow:An Open Source Machine Learning Framework for Every-one,” https://github.com/tensorflow/tensorflow/, 2018, accessed: 2018-11-25.

[22] A. Hill, A. Raffin, R. Traore, P. Dhariwal, C. Hesse, O. Klimov,A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu,“Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.

[23] M. Schaarschmidt, A. Kuhnle, and K. Fricke, “Tensorforce: Atensorflow library for applied reinforcement learning,” Web page,2017, accessed: 2018-12-07. [Online]. Available: https://github.com/reinforceio/tensorforce

[24] “STM32F745VG,” https://www.st.com/en/microcontrollers/stm32f745vg.html, 2018, accessed: 2018-11-25.

[25] “Bazel - a fast, scalable, multi-language and extensible build system,”https://bazel.build/, 2018, accessed: 2018-11-25.

[26] “NF1: Neuroflight Test Aircraft 1,” https://rotorbuilds.com/build/15163,2018, accessed: 2018-11-25.

[27] “Motor Data Explorer,” https://www.miniquadtestbench.com/motor-explorer.html, 2018, accessed: 2018-11-25.

[28] “Iris Quadcopter,” 2018, accessed: 2018-12-07. [Online]. Available:http://www.arducopter.co.uk/iris-quadcopter-uav.html

[29] “Brushless motor constant explained,” http://learningrc.com/motor-kv/,2015, accessed: 2018-12-07.

[30] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,D. Silver, and D. Wierstra, “Continuous control with deep reinforcementlearning,” arXiv preprint arXiv:1509.02971, 2015.

[31] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trustregion policy optimization,” in International Conference on MachineLearning, 2015, pp. 1889–1897.

[32] L. Meier, D. Honegger, and M. Pollefeys, “Px4: A node-based mul-tithreaded open source robotics framework for deeply embedded plat-forms,” in 2015 IEEE international conference on robotics and automa-tion (ICRA). IEEE, 2015, pp. 6235–6240.

[33] M. Abdulrahim, T. Bates, T. Nilson, J. Bloch, D. Nethery, and T. Smith,“Defining flight envelope requirements and handling qualities criteriafor first-person-view quadrotor racing,” in AIAA Scitech 2019 Forum,2019, p. 0825.

[34] O. Liang, “Looptime and Flight Controller,” https://oscarliang.com/best-looptime-flight-controller/, 2018, accessed: 2018-11-25.

[35] J. G. Ziegler and N. B. Nichols, “Optimum settings for automaticcontrollers,” trans. ASME, vol. 64, no. 11, 1942.