Coevolution of Human-Competitive Robocode Tanks Using Genetic

Programming with Exogenous Fitness

Jason Owens & Ron Bowers

Why?

• Interesting

• GP and coevolution are particularly intriguing aspects of EC.

• We are using EC to generate an entertaining product.

• Challenging

• GP and coevolution are difficult to use.

• To date, no one has succeeded with coevolution in Robocode.

• Relevant

• Our employer is developing adaptive, agent-based combat simulations.

• We aren't working on them.

Coevolution

• Angeline and Pollock [1993] argue that competitive fitness functions are more effective than independent fitness functions.

• A competitive fitness function is dependent on the current state of the population, rather than relative to a global optimum.

• Generally, competitive fitness is measured through competition betweenpopulation members.

• These competitions can range from testing each member against each other to a carefully structured tournament

• Floreano, Nolfi and Mondada [1999] found that competitive coevolution can generate efficient chasing and escaping strategies in a predator/prey environment.

• This capability evolved even more quickly than "simple" wall avoidance

• They postulate that competitive coevolution generates solutions to difficult problems by generating a series of challenges of increasing difficulty.

• Uchibe and Asasa [2006] were able to use coevolution to evolve both competitive and cooperative behaviors in soccer playing bots.

Difficulties with Coevolution

• Mediocre Stable State

• Ficici and Pollack [1998] note additional difficulties.

• Maintaining balance in the arms race.

• The environment should neither be too easy nor too difficult:

• Open-ended coevolution is not necessarily synonymous with a purely competitive framework. Instead, it's necessary to provide a mixture of competitive and cooperative pressures in order to avoid the mediocre stable state.

Previous Work with Robocode

• Eisenstein [2003]

• Used a GA to evolve a subsumption architecture.

• Was successful in developing bots that could fight a specific adversary given a specific starting condition

• Attempted to use coevolution but “after several generations I found the populations rife with catatonics”

• Hong and Cho [2004]

• Used a GA that consisted of 6 chromosomes, representing the bot behavior in the main loop and in 5 of the event handlers.

• Each chromosome consisted of six genes, corresponding to actions such as move or shoot.

• Each action could be one of 2 or more hand-coded implementations.

• Were successful in consistently defeating 3 of the standard bots.

Previous Work With Robocode

• Shichel, Ziserman, and Sipper [2005]

• Used Koza-style GP

• Limited investigation to Haiku Bot (4 lines of code)

• Evolved bots were entered into a Haiku Bot tournament where they placed third out of 27.

• Also attempted coevolution, but could not produce competitive bots.

Hypothesis

• We hypothesize that by using genetic programming and coevolution with an exogenous fitness function we can evolve Robocode agents that can compete successfully against human-coded bots.

• But...

Initial Results

• We did not entirely succeed.

• We did not produce any competitive agents in time to report in the paper.

• We have continued our efforts!

Algorithm Configuration

• Strongly-typed tree-based GP

• Linear-rank selection using stochastic universal sampling

• 80% crossover, 20% mutation (whoa!)

• Elitism (one individual)

• Initial tree depth of 5

Fitness Functions

• simple (shoot and dodge)

• movement, enemy sensing, wall avoidance

• re-proportioned movement value

• final, emphasis on damage with firing/scanning efficiencies

Analysis of Initial Results

• Unearned dodging credit

• overblown scan credit

• Generally a rapid rise, then decay.

• Trees appear non-robust (brittle).

• Behavior arises from the coupling of trees and state. Different event histories lead to different states and thus radically different behavior.

Mistakes and Deficiencies

• Using a Single Population

• Using instantaneous fitness.

• Not monitoring the progress to determine if the population had started to degress.

• Threading issues (Heisenbugs) in Robocode.

Updated Results

Open Questions

•Can coevolution in general succeed in multi-dimensional one-population competitive environments without outside influence?

•Does this environment need the insertion of external human-coded bots to drive fitness adjustments?

•How do we modify the fitness function to improve behavior?

Future Work

• Continue experimenting with the exogenous fitness function, the node set, population size, and the reproduction parameters in an attempt to succeed with the coevolutionary approach.

• Incorporate the results from the progress measurement rounds into the selection process.

• Replace the single population model with a dual population model and investigate the efficacy of the evaluation approaches specified in De Jong [2004] and Bader-Natal [2005]

• Modify the driver program to limit the size of the trees.