Model-Based Stochastic Search for Large Scale Optimization ofMulti-Agent UAV Swarms

David D. Fan1, Evangelos Theodorou2, and John Reeder3

Abstract— Recent work from the reinforcement learningcommunity has shown that Evolution Strategies are a fast andscalable alternative to other reinforcement learning methods.In this paper we show that Evolution Strategies are a specialcase of model-based stochastic search methods. This class ofalgorithms has nice asymptotic convergence properties andknown convergence rates. We show how these methods canbe used to solve both cooperative and competitive multi-agent problems in an efficient manner. We demonstrate theeffectiveness of this approach on two complex multi-agent UAVswarm combat scenarios: where a team of fixed wing aircraftmust attack a well-defended base, and where two teams ofagents go head to head to defeat each other†.

I. INTRODUCTION

Reinforcement Learning is concerned with maximizingrewards from an environment through repeated interactionsand trial and error. Such methods often rely on variousapproximations of the Bellman equation and include valuefunction approximation, policy gradient methods, and more[1]. The Evolutionary Computation community, on the otherhand, have developed a suite of methods for black boxoptimization and heuristic search [2]. Such methods havebeen used to optimize the structure of neural networks forvision tasks, for instance [3].

Recently, Salimans et al. have shown that a particularvariant of evolutionary computation methods, termed Evo-lution Strategies (ES) are a fast and scalable alternative toother reinforcement learning approaches, solving the difficulthumanoid MuJoCo task in 10 minutes [4]. The authorsargue that ES has several benefits over other reinforcementlearning methods: 1) The need to backpropagate gradientsthrough a policy is avoided, which opens up a wider classof policy parameterizations; 2) ES methods are massivelyparallelizable, which allows for scaling up learning to larger,more complex problems; 3) ES often finds policies whichare more robust than other reinforcement learning methods;and 4) ES are better at assigning credit to changes in thepolicy over longer timescales, which enables solving taskswith longer time horizons and sparse rewards. In this workwe leverage all four of these advantages by using ES tosolve a problem with: 1) a more complex and decipherablepolicy architecture which allows for safety considerations; 2)a large-scale simulated environment with many interacting

1 Inst. for Robotics and Intelligent Machines, Georgia Institute ofTechnology. david.fan@gatech.edu

2 Dept. of Aerospace Engineering, Georgia Institute of Technology.evangelos.theodorou@gatech.edu

3 SPAWAR Systems Center Pacific, San Diego, CA, USA.john.d.reeder@navy.mil

† Video at http://goo.gl/dWvQi7

Fig. 1: The SCRIMMAGE multi-agent simulation environ-ment. In this scenario, blue team fixed-wing agents attackred team quadcopter defenders. White lines indicate missedshots.

elements; 3) multiple sources of stochasticity including vari-ations in intial conditions, disturbances, etc.; and 4) sparserewards which only occur at the very end of a long episode.

A common critique of evolutionary computation algo-rithms is a lack of convergence analysis or guarantees.Of course, for problems with non-differentiable and non-convex objective functions, analysis will always be diffi-cult. Nevertheless, we show that the Evolution Strategiesalgorithm proposed by [4] is a special case of a class ofmodel-based stochastic search methods known as Gradient-Based Adaptive Stochastic Search (GASS) [5]. This class ofmethods generalizes many stochastic search methods such asthe well-known Cross Entropy Method (CEM) [6], CMA-ES [7], etc. By casting a non-differentiable, non-convexoptimization problem as a gradient descent problem, one canarrive at nice asymptotic convergence properties and knownconvergence rates [8].

With more confidence in the convergence of EvolutionStrategies, we demonstrate how ES can be used to efficientlysolve both cooperative and competitive large-scale multi-agent problems. Many approaches to solving multi-agentproblems rely on hand-designed and hand-tuned algorithms(see [9] for a review). One such example, distributed ModelPredictive Control, relies on independent MPC controllerson each agent with some level of coordination betweenthem [10], [11]. These controllers require hand-designing

arX

iv:1

803.

0110

6v2

[cs

.MA

] 1

1 A

ug 2

018

dynamics models, cost functions, feedback gains, etc. andrequire expert domain knowledge. Additionally, scaling thesemethods up to more complex problems continues to bean issue. Evolutionary algorithms have also been tried asa solution to multi-agent problems; usually with smaller,simpler environments, and policies with low complexity [12],[13]. Recently, a hybrid approach combining MPC and theuse of genetic algorithms to evolve the cost function for ahand-tuned MPC controller has been demonstrated for a UAVswarm combat scenario [14].

In this work we demonstrate the effectiveness of ourapproach on two complex multi-agent UAV swarm combatscenarios: where a team of fixed wing aircraft must attacka well-defended base, and where two teams of agents gohead to head to defeat each other. Such scenarios have beenpreviously considered in simulated environments with lessfidelity and complexity [15], [14]. We leverage the compu-tational efficiency and flexibility of the recently developedSCRIMMAGE multi-agent simulator for our experiments(Figure 1) [16]. We compare the performance of ES againstthe Cross Entropy Method. We also show for the competitivescenario how the policy learns over time to coordinate astrategy in response to an enemy learning to do the same.We make our code freely available for use (https://github.com/ddfan/swarm_evolve).

II. PROBLEM FORMULATION

We can pose our problem as the non-differentiable, non-convex optimization problem

θ∗ = argmax

θ∈Θ

J(θ) (1)

where Θ ⊂ Rn, a nonempty compact set, is the spaceof solutions, and J(θ) is a non-differentiable, non-convexreal-valued objective function J : Θ→ R. θ could be anycombination of decision variables of our problem, includingneural network weights, PID gains, hardware design param-eters, etc. which affect the outcome of the returns J. Forreinforcement learning problems θ usually represents theparameters of the policy and J is an implicit function of thesequential application of the policy to the environment. Wefirst review how this problem can be solved using Gradient-Based Adaptive Stochastic Search methods and then showhow the ES algorithm is a special case of these methods.

A. Gradient-Based Adaptive Stochastic Search

The goal of model-based stochastic search methods isto cast the non-differentiable optimization problem (1) asa differentiable one by specifying a probabilistic model(hence ”model-based”) from which to sample [8]. Let thismodel be p(θ |ω) = f (θ ;ω),ω ∈Ω, where ω is a parameterwhich defines the probability distribution (e.g. for Gaussiandistributions, the distribution is fully parameterized by themean and variance ω = [µ,σ ]). Then the expectation of J(θ)over the distribution f (θ ;ω) will always be less than theoptimal value of J, i.e.∫

Θ

J(θ) f (θ ;ω)dθ ≤ J(θ ∗) (2)

The idea of Gradient-based Adaptive Stochastic Search(GASS) is that one can perform a search in the spaceof parameters of the distribution Ω rather than Θ, for adistribution which maximizes the expectation in (2):

ω∗ = argmax

ω∈Ω

∫Θ

J(θ) f (θ ;ω)dθ (3)

Maximizing this expectation corresponds to finding a distri-bution which is maximally distributed around the optimal θ .However, unlike maximizing (1), this objective function cannow be made continuous and differentiable with respect toω . With some assumptions on the form of the distribution,the gradient with respect to ω can be pushed inside theexpectation.

The GASS algorithm presented by [8] is applicable to theexponential family of probability densities:

f (θ ;ω) = expωᵀT (θ)−φ(θ) (4)

where φ(θ) = ln∫

exp(ωᵀT (θ))dθ , and T (θ) is the vectorof sufficient statistics. Since we are concerned with showingthe connection with ES which uses parameter perturbationssampled with Gaussian noise, we assume that f (θ ;ω) isGaussian. Furthermore, since we are concerned with learninga large number of parameters (i.e. weights in a neuralnetwork), we assume an independent Gaussian distributionover each parameter. Then, T (θ) = [θ ,θ 2]ᵀ ∈ R2n and ω =[µ/σ2,−1/nσ2]ᵀ ∈ R2n, where µ and σ are vectors of themean and standard deviation corresponding to the distribu-tion of each parameter, respectively.

Algorithm 1 Gradient-Based Adaptive Stochastic Search

Require: Learning rate αk, sample size Nk, initial policyparameters ω0 = [µ0,σ

20 ]

ᵀ, smoothing function S(), smallconstant γ > 0.

1: for k = 0,1, · · · do2: Sample θ i

kiid∼ f (θ ;ωk), i = 1,2, · · · ,Nk.

3: Compute returns wik = S(J(θ i

k)) for i = 1, · · · ,Nk.4: Compute variance terms Vk = Vk + γI, eq (5),(6)5: Calculate normalizer η = ∑

Nki=1 wi

k.6: Update ωk+1:

7: ωk+1← ωk +αk1η

V−1k

Nk

∑i=1

wik

([θ i

k(θ i

k)2

]−[

µ

σ2 +µ2

])8: end for

We present the GASS algorithm for this specific set ofprobability models (Algorithm 1), although the analysis forconvergence holds for the more general exponential familyof distributions. For each iteration k, The GASS algorithminvolves drawing Nk samples of parameters θ i

kiid∼ f (θ ;ωk), i=

1,2, · · · ,Nk. These parameters are then used to sample thereturn function J(θ i

k). The returns are fed through a shapingfunction S(·) : R→R+ and then used to calculate an updateon the model parameters ωk+1.

The shaping function S(·) is required to be nondecreasingand bounded from above and below for bounded inputs,with the lower bound away from 0. Additionally, the set

argmaxθ∈Θ S(J(θ)) must be a nonempty subset of theset of solutions of the original problem argmaxθ∈Θ J(θ).The shaping function can be used to adjust the explo-ration/exploitation trade-off or help deal with outliers whensampling. The original analysis of GASS assumes a moregeneral form of Sk(·) where S can change at each iteration.For simplicity we assume here it is deterministic and un-changing per iteration.

GASS can be considered a second-order gradient methodand requires estimating the variance of the sampled param-eters:

Vk =1

Nk−1

Nk

∑i=1

T (θ ik)T (θ

ik)

ᵀ

− 1N2

k −Nk

(Nk

∑i=1

T (θ ik)

)(Nk

∑i=1

T (θ ik)

)ᵀ

. (5)

In practice if the size of the parameter space Θ is large, as isthe case in neural networks, this variance matrix will be ofsize 2n×2n and will be costly to compute. In our work weapproximate Vk with independent calculations of the varianceon the parameters of each independent Gaussian. With aslight abuse of notation, consider θ i

k as a scalar element ofθ i

k. We then have, for each scalar element θ ik a 2×2 variance

matrix:

Vk =1

Nk−1

Nk

∑i=1

[θ i

k(θ i

k)2

][θ i

k (θ ik)

2]

− 1N2

k −Nk

(Nk

∑i=1

[θ i

k(θ i

k)2

])( Nk

∑i=1

[θ i

k (θ ik)

2])

. (6)

Theorem 1 shows that GASS produces a sequence ofωk that converges to a limit set which specifies a set ofdistributions that maximize (3). Distributions in this set willspecify how to choose θ ∗ to ultimately maximize (1). Aswith most non-convex optimization algorithms, we are notguaranteed to arrive at the global maximum, but using prob-abilistic models and careful choice of the shaping functionshould help avoid early convergence into suboptimal localmaximum. The proof relies on casting the update rule in theform of a generalized Robbins-Monro algorithm (see [8],Thms 1 and 2). Theorem 1 also specifies convergence ratesin terms of the number of iterations k, the number of samplesper iteration Nk, and the learning rate αk. In practice Theorem1 implies the need to carefully balance the increase in thenumber of samples per iteration and the decrease in learningrate as iterations progress.Assumption 1

i) The learning rate αk > 0, αk → 0 as k → ∞, and∑

∞k=0 αk = ∞.

ii) The sample size Nk = N0kξ , where ξ > 0; also αk andNk jointly satisfy α/

√Nk = O(k−β ).

iii) T (θ) is bounded on Θ.iv) If ω∗ is a local maximum of (3), the Hessian of∫

ΘJ(θ) f (θ ;ω)dθ is continuous and symmetric negative

definite in a neighborhood of ω∗.

Theorem 1Assume that Assumption 1 holds. Let αk = α0/kα for 0 <α < 1. Let Nk = N0kτ−α where τ > 2α is a constant. Thenthe sequence ωk generated by Algorithm 1 converges to alimit set w.p.1. with rate O(1/

√kτ).

B. Evolutionary Strategies

We now review the ES algorithm proposed by [4] andshow how it is a first-order approximation of the GASSalgorithm. The ES algorithm consists of the same twophases as GASS: 1) Randomly perturb parameters with noisesampled from a Gaussian distribution. 2) Calculate returnsand calculate an update to the parameters. The algorithm isoutlined in Algorithm 2. Once returns are calculated, they aresent through a function S(·) which performs fitness shaping[17]. Salimans et al. used a rank transformation function forS(·) which they argue reduced the influence of outliers ateach iteration and helped to avoid local optima. It is clear that

Algorithm 2 Evolution Strategies

Require: Learning rate αk, noise standard deviation γ , initialpolicy parameters θ0, smoothing function S().

1: for k = 0,1, · · · do2: Sample ε1, · · · ,εn ∼N (0, In×n)3: Compute returns wi

k = S(J(θk + γεi)) for i = 1, · · · ,Nk

4: Update θk+1:5: θk+1← θk +αk

1Nkγ ∑

Nki=1 wi

kεi6: end for

the ES algorithm is a sub-case of the GASS algorithm whenthe sampling distribution is a point distribution. We can alsorecover the ES algorithm by ignoring the variance terms online 7 in Algorithm 1. Instead of the normalizing term η , ESuses the number of samples Nk. The small constant in GASSγ becomes the variance term in the ES algorithm. The updaterule in Algorithm 2 involves multiplying the scaled returns bythe noise, which is exactly θ i

k−µ in Algorithm 1. We see thatES enjoys the same asymptotic convergence rates offered bythe analysis of GASS. While GASS is a second-order methodand ES is only a first-order method, in practice ES usesapproximate second-order gradient descent methods whichadapt the learning rate in order to speed up and stabilizelearning. Examples of these methods include ADAM, RM-SProp, SGD with momentum, etc., which have been shownto perform very well for neural networks. Therefore we cantreat ES a first-order approximation of the full second-ordervariance updates which GASS uses. In our experiments weuse ADAM [18] to adapt the learning rate for each parameter.As similarly reported in [4], when using adaptive learningrates we found little improvement over adapting the varianceof the sampling distribution. We hypothesize that a first ordermethod with adaptive learning rates is sufficient for achievinggood performance when optimizing neural networks. Forother types of policy parameterizations however, the fullsecond-order treatment of GASS may be more useful. It isalso possible to mix and match which parameters require a

-+ PID

𝑥𝑟𝑒𝑓𝑦𝑟𝑒𝑓𝑧𝑟𝑒𝑓

𝑢(𝑡)

𝑓( Ԧ𝑜(𝑡); 𝜃)

Safety Logic

Ԧ𝑜(𝑡)

Ally states

Sensing Neural Network

Ԧ𝑥(𝑡)

Fig. 2: Diagram of each agent’s policy. Nearby ally states and sensed enemies, base locations, etc. along with the agent’sown state are fed into a neural network which produces a reference target in relative xyz coordinates. The target is fed intothe safety logic block which checks for collisions with neighbors or the ground. It produces a reference target which is fedto the PID controller, which in turn provides low-level controls for the agent (thrust, aileron, elevator, rudder).

full variance update and which can be updated with a first-order approximate method. We use the rank transformationfunction for S(·) and keep Nk constant.

C. Learning Structured Policies for Multi-Agent Problems

Now that we are more confident about the convergenceof the ES/GASS method, we show how ES can be used tooptimize a complex policy in a large-scale multi-agent envi-ronment. We use the SCRIMMAGE multi-agent simulationenvironment [16] as it allows us to quickly and in parallelsimulate complex multi-agent scenarios. We populate oursimulation with 6DoF fixed-wing aircraft and quadcopterswith dynamics models having 10 and 12 states, respectively.These dynamcis models allow for full ranges of motionwithin realistic operating regimes. Stochastic disturbancesin the form of wind and control noise are modeled asadditive Gaussian noise. Ground and mid-air collisions canoccur which result in the aircraft being destroyed. We alsoincorporate a weapons module which allows for targetingand firing at an enemy within a fixed cone projecting fromthe aircraft’s nose. The probability of a hit depends on thedistance to the target and the total area presented by the targetto the attacker. This area is based on the wireframe modelof the aircraft and its relative pose. For more details, see ourcode and the SCRIMMAGE simulator documentation.

We consider the case where each agent uses its own policyto compute its own controls, but where the parameters of thepolicies are the same for all agents. This allows each agentto control itself in a decentralized manner, while allowingfor beneficial group behaviors to emerge. Furthermore, weassume that friendly agents can communicate to share stateswith each other (see Figure 2). Because we have a largenumber of agents (up to 50 per team), to keep communicationcosts lower we only allow agents to share information locally,i.e. agents close to each other have access to each other’sstates. In our experiments we allow each agent to sense thestates of the closest 5 friendly agents for a total of 5∗10= 50

incoming state messages.Additionally, each agent is equipped with sensors to detect

enemy agents. Full state observability is not available here,instead we assume that sensors are capable of sensing anenemy’s relative position and velocity. In our experimentswe assumed that each agent is able to sense the nearest5 enemies for a total of 5 ∗ 7 = 35 dimensions of enemydata (7 states = [relative xyz position, distance, and relativexyz velocities]). The sensors also provide information abouthome and enemy base relative headings and distances (anadditional 8 states). With the addition of the agent’s own state(9 states), the policy’s observation input ~o(t) has a dimensionof 102. These input states are fed into the agent’s policy: aneural network f (~o(t);θ) with 3 fully connected layers withsizes 200, 200, and 50, which outputs 3 numbers representinga desired relative heading [xre f ,yre f ,zre f ]. Each agent’s neuralnetwork has more than 70,000 parameters. Each agent usesthe same neural network parameters as its teammates, butsince each agent encounters a different observation at eachtimestep, the output of each agent’s neural network policywill be unique. It may also be possible to learn uniquepolicies for each agent; we leave this for future work.

With safety being a large concern in UAV flight, wedesign the policy to take into account safety and controlconsiderations. The relative heading output from the neuralnetwork policy is intended to be used by a PID controllerto track the heading. The PID controller provides low-level control commands u(t) to the aircraft (thrust, aileron,elevator, rudder). However, to prevent cases where the neuralnetwork policy guides the aircraft into crashing into theground or allies, etc., we override the neural network headingwith an avoidance heading if the aircraft is about to collidewith something. This helps to focus the learning processon how to intelligently interact with the environment andallies rather than learning how to avoid obvious mistakes.Furthermore, by designing the policy in a structured andinterpretable way, it will be easier to take the learned policy

directly from simulation into the real world. Since theneural network component of the policy does not producelow-level commands, it is invariant to different low-levelcontrollers, dynamics, PID gains, etc. This aids in learningmore transferrable policies for real-world applications.

III. EXPERIMENTS

We consider two scenarios: a base attack scenario wherea team of 50 fixed wing aircraft must attack an enemy basedefended by 20 quadcopters, and a team competitive taskwhere two teams concurrently learn to defeat each other. Inboth tasks we use the following reward function:

J = 10× (#kills)+50× (#collisions with enemy base)−1e−5× (distance from enemy base at end of episode)

(7)

The reward function encourages air-to-air combat, as wellas suicide attacks against the enemy base (e.g. a swarm ofcheap, disposable drones carrying payloads). The last termencourages the aircraft to move towards the enemy duringthe initial phases of learning.

A. Base Attack Task

Fig. 3: Snapshot of base attack task. The goal of the bluefixed wing team (lower left) is to attack the red base (reddot, upper right) while avoiding or attacking red quadcopterguards.

In this scenario a team of 50 fixed-wing aircraft mustattack an enemy base defended by 20 quadcopters (Figure3). The quadcopters use a hand-crafted policy where in theabsence of an enemy, they spread themselves out evenly tocover the base. In the presence of an enemy they targetthe closest enemy, match that enemy’s altitude, and firerepeatedly. We used Nk = 300,γ = 0.02, a time step of 0.1seconds, and total episode length of 200 seconds. Initialpositions of both teams were randomized in a fixed area atopposide ends of the arena. Training took two days with fullparallelization on a machine equipped with a Xeon Phi CPU(244 threads).

We found that over the course of training the fixed-wing team learned a policy where they quickly form a V-formation and approach the base. Some aircraft suicide-attack the enemy base while others begin dog-fighting (see

0 500 1000 1500 2000 2500Iteration

0

250

500

750

1000

1250

Score

ESCEM

(a) Training

0 500 1000 1500 2000 2500Iteration

0

250

500

750

1000

1250

Score

ESCEM

(b) Testing

Fig. 4: Scores per iteration for the base attack task. Top:Scores earned by perturbed policies during training. Scoresare on average lower because they result from policies whichare parameterized by randomly peturbed values. Bottom:Scores during the course of training earned by the updatedpolicy parameters. Red curve is Evolution Strategies algo-rithm, blue is Cross Entropy Method. Bold line is the median,shaded areas are 25/75 quartile bounds.

supplementary video1). We also compared our implementa-tion of the ES method against the well-known cross-entropymethod (CEM). CEM performs significantly worse than ES(Figure 4). We hypothesize this is because CEM throws outa significant fraction of sampled parameters and thereforeobtains a worse estimate of the gradient of (3). Comparisonagainst other full second-order methods such as CMA-ES orthe full second-order GASS algorithm is unrealistic due tothe large number of parameters in the neural network andthe prohibitive computational difficulties with computing thecovariances of those parameters.

B. Two Team Competitive Match

The second scenario we consider is where two teams eachequipped with their own unique policies for their agentslearn concurrently to defeat their opponent (Figure 5). Ateach iteration, Nk = 300 simulations are spawned, eachwith a different random perturbation, and with each teamhaving a different perturbation. The updates for each policyare calculated based on the scores received from playingthe opponent’s perturbed policies. The result is that each

1http://https://goo.gl/dWvQi7

Fig. 5: Snapshot of two team competitive match. The goalof both teams is to defeat all enemy planes while sufferingminimum losses, or to attack the opponent’s base. Red linesindicate successful firing hit.

team learns to defeat a wide range of opponent behaviorsat each iteration. We observed that the behavior of thetwo teams quickly approached a Nash equilibrium whereboth sides try to defeat the maximum number of opponentaircraft in order to prevent higher-scoring suicide attacks (seesupplementary video). The end result is a stalemate withboth teams annihilating each other, ending with tied scores(Figure 6). We hypothesize that more varied behavior couldbe learned by having each team compete against some pastenemy team behaviors or by building a library of policiesfrom which to select from, as frequently discussed by theevolutionary computation community [19].

IV. CONCLUSION

We have shown that Evolution Strategies are applicablefor learning policies with many thousands of parametersfor a wide range of complex tasks in both the competi-tive and cooperative multi-agent setting. By showing theconnection between ES and more well-understood model-based stochastic search methods, we are able to gain in-sight into future algorithm design. Future work will includeexperiments with optimizing mixed parameterizations, e.g.optimizing both neural network weights and PID gains. Inthis case, the second-order treatment on non-neural networkparameters may be more beneficial, since the behavior of thesystem may be more sensitive to perturbations of non-neuralnetwork parameters. Another direction of investigation couldbe optimizing unique policies for each agent in the team.Yet another direction would be comparing other evolutionarycomputation strategies for training neural networks, includ-ing methods which use a more diverse population [20], ormore genetic algorithm-type heuristics [21].

REFERENCES

[1] Y. Li, “Deep Reinforcement Learning: An Overview,” ArXiv e-prints,Jan. 2017.

[2] K. Stanley and B. Bryant, “Real-time neuroevolution in the NEROvideo game,” IEEE transactions on, 2005. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=1545941

[3] O. J. Coleman, “Evolving Neural Networks for Visual Processing,”Thesis, 2010.

0 100 200 300 400 500 600 700Iteration

0

100

200

300

400

500

Scor

e

Team 1Team 2

(a) Training

0 100 200 300 400 500 600 700Iteration

0

100

200

300

400

500

Scor

e

Team 1Team 2

(b) Testing

Fig. 6: Scores per iteration for the team competition task.Top: Scores earned by each team with policies parameterizedby randomly peturbed values during training. Bottom: Scoresearned when testing the trained architectures using updatedpolicy parameters. Red and blue curves show scores for team1 and 2 respectively.

[4] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever, “EvolutionStrategies as a Scalable Alternative to Reinforcement Learning,” ArXive-prints, Mar. 2017.

[5] J. Hu, “Model-based stochastic search methods,” in Handbook ofSimulation Optimization. Springer, 2015, pp. 319–340.

[6] S. Mannor, R. Rubinstein, and Y. Gat, “The cross entropy method forfast policy search,” in Machine Learning-International Workshop ThenConference-, vol. 20, no. 2, 2003, Conference Proceedings, p. 512.

[7] N. Hansen, “The CMA evolution strategy: A tutorial,” CoRR, vol.abs/1604.00772, 2016. [Online]. Available: http://arxiv.org/abs/1604.00772

[8] E. Zhou and J. Hu, “Gradient-based adaptive stochastic search for non-differentiable optimization,” IEEE Transactions on Automatic Control,vol. 59, no. 7, pp. 1818–1832, 2014.

[9] L. Panait and S. Luke, “Cooperative multi-agent learning: Thestate of the art,” Autonomous Agents and Multi-Agent Systems,vol. 11, no. 3, pp. 387–434, 2005. [Online]. Available: http://link.springer.com/10.1007/s10458-005-2631-2

[10] J. B. Rawlings and B. T. Stewart, “Coordinating multiple optimization-based controllers: New opportunities and challenges,” Journal ofProcess Control, vol. 18, no. 9, pp. 839–845, 2008.

[11] W. Al-Gherwi, H. Budman, and A. Elkamel, “Robust distributed modelpredictive control: A review and recent developments,” The CanadianJournal of Chemical Engineering, vol. 89, no. 5, pp. 1176–1190,2011. [Online]. Available: http://doi.wiley.com/10.1002/cjce.20555

[12] G. B. Lamont, J. N. Slear, and K. Melendez, “UAV swarm missionplanning and routing using multi-objective evolutionary algorithms,” inIEEE Symposium Computational Intelligence in Multicriteria DecisionMaking, no. Mcdm, 2007, Conference Proceedings, pp. 10–20.

[13] A. R. Yu, B. B. Thompson, and R. J. Marks, “Competitive evolution oftactical multiswarm dynamics,” IEEE Transactions on Systems, Man,

and Cybernetics Part A:Systems and Humans, vol. 43, no. 3, pp. 563–569, 2013.

[14] D. D. Fan, E. Theodorou, and J. Reeder, “Evolving cost functionsfor model predictive control of multi-agent uav combat swarms,” inProceedings of the Genetic and Evolutionary Computation ConferenceCompanion, ser. GECCO ’17. New York, NY, USA: ACM, 2017,pp. 55–56. [Online]. Available: http://doi.acm.org/10.1145/3067695.3076019

[15] U. Gaerther, “UAV swarm tactics: an agent-based simulationand Markov process analysis,” 2015. [Online]. Available: https://calhoun.nps.edu/handle/10945/34665

[16] K. J. DeMarco. (2018) SCRIMMAGE multi-agent robotics simulator.[Online]. Available: http://www.scrimmagesim.org/

[17] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, andJ. Schmidhuber, “Natural evolution strategies.” Journal of MachineLearning Research, vol. 15, no. 1, pp. 949–980, 2014.

[18] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza-tion,” arXiv preprint arXiv:1412.6980, 2014.

[19] K. O. Stanley and R. Miikkulainen, “Competitive coevolution throughevolutionary complexification,” Journal of Artificial Intelligence Re-search, vol. 21, pp. 63–100, 2004.

[20] E. Conti, V. Madhavan, F. Petroski Such, J. Lehman, K. O. Stanley,and J. Clune, “Improving Exploration in Evolution Strategies for DeepReinforcement Learning via a Population of Novelty-Seeking Agents,”ArXiv e-prints, Dec. 2017.

[21] F. Petroski Such, V. Madhavan, E. Conti, J. Lehman, K. O. Stanley, andJ. Clune, “Deep Neuroevolution: Genetic Algorithms Are a Competi-tive Alternative for Training Deep Neural Networks for ReinforcementLearning,” ArXiv e-prints, Dec. 2017.