Two-Step Evolution Process forPath-Following Virtual Creatures

Yoon-Sik ShimKorea University

necromax@kebi.comhttp://melkzedek.com

Seung-Young ShinKorea University

shinyab@korea.comhttp://kucg.korea.ac.kr/˜syshin

Chang-Hun KimKorea University

chkim@korea.ac.krhttp://kucg.korea.ac.kr/˜chkim

AbstractWe present a two-step evolution system thatproduces controllable virtual creatures which isphysically simulated in 3D environment, andallows them to follow a path trajectory givenby a user. A creature’s body and its low-levelcontrollers for basic swimming motion aregenerated by an evolutionary algorithm. Thena feed-forward neural network is attached tothe low-level controllers, and the connectionweights of the network for a given trajectoryare found by a genetic algorithm. We show thata control system sufficiently effective to allowevolved creatures to follow a complicated pathcan be achieved by two-step evolution process.

Keywords: computer animation, physically-based simulation, artificial life, evolutionaryalgorithm, neural network

1 Introduction

To date, artificial evolution researchers havefound only a few applications for virtual em-bodied agents. Sims [1][2] pioneered an inno-vative approach in generating both the morphol-ogy and the behavior of physics-based articu-lated figures by artificial evolution. His “block-

ies” creatures moved in a way that looked bothorganic and elegant. Other recent work such asSexual Swimers [3], Framsticks [4], and VirtualPets [5], have produced various applications intheir own way. The Golem project [6] even tookthis technique a step further by evolving simu-lations of physically realistic mobile robots andthen building real-world copies.

The artificial evolution of such creatures po-tentially offers some relief from the difficultyand time-consuming task of specifying mor-phologies and behaviors by exploring a hugesearch space of possible creatures automatically.Although the evolutionary algorithm allows togenerate interesting and life-like shapes and mo-tions automatically, it is hard to understand theinternal mechanism of evolved controller of thecreature. This arises from the use of an unnec-essarily complicated structure of the randomlyconnected controller network (which is knownas “augmented neural network”), so it is ex-tremely difficult problem to give a user con-trol to the evolved creatures such as constrainedpath-following behaviors. Although previouswork has evolved point-following creatures us-ing network-based controllers, the size of thesearch space is very huge, and it is hard to makesuccessful combination of various neural func-

1

tions which satisfies both the efficient locomo-tion and the path-following behavior at the sametime.

We now present a two-step evolution processfor evolving path-following creatures. We usea straightforward and intuitive genetic represen-tation of the motor controller using piecewisesinusoidal functions that have more possibilityto produce periodic motion of body parts. Wedescribe an appropriate feed-forward neural net-work model which controls a creature’s motionto follow a given trajectory. We show that ourneural network optimization successfully pro-duces a global solution for a creature to per-form reasonable following behavior for variouspath. In addition, in order to accelerate the sys-tem speed, we model a simplified hydrodynam-ics for a cylindrical body part that the fluid dragis calculated only from the linear and angularvelocities of a cylinder, not by calculating everyforces on each surface.

2 Overview

The entire process is divided into two stages.In the first stage, our creature evolution sys-tem generates the creature’s body and its low-level controllers that support straight swimming.After this basic forward-moving creature has

Figure 1: Overview of the entire system. Theevolution system creates straight-moving creature, and a neural networkcontrols the creature so that it will fol-low a given trajectory.

been generated, a neural network is connected toits controllers, and the connection weights of thenetwork are optimized by a genetic algorithm

to make the creature to follow given trajectory.During the optimization, a set of arrays whichcontains the weight vectors for the neural net-work is used as the population. At each timestep, the network receives the position and ve-locity of the creature’s center of mass, the feed-back of the neural network output, and the targetposition which is calculated in realtime based onthe creature’s velocity and the given trajectory.The signals coming out of the network are fedto the creature’s controllers, and vary their be-havior so that the motion of the body parts ischanged.

The concept can be explained at an intuitivelevel by comparing it to three elements of a reallife form: the body, the motor nerves of the cere-bellum (evolved low-level controllers) which isin charge of motor control, and the cerebrum (aneural network) which gives high-level controlto the cerebellum. In the next sections, we de-scribe each process in more detail.

3 Genotype Representation andCreature Development

The genotype encoding is accomplished usinga directed graph same as Sims’ [1], but with-out any inner neural networks. One more dif-ference between our work and Sims’ is that weused cylinders with hemispherical ends, ratherthan cuboids, as the basic body parts in orderto perform simpler dynamics calculation withits symmetric shape. Each cylinder has a joint

Figure 2: An example of a genotype using di-rected graph.

which connects it to its parent. The joint canhave 1-3 degrees of freedom, and each degreesof freedom has a number of parameters whichrepresent a corresponding low-level controller.The cylinder is aligned along the local z-axis,and each cylinder’s two end points (the centers

2

Figure 3: The joining sequence of a parent-childpair. (a)The child is placed at a po-sition of the parent specified by Lper-cent. (b)The child rotates about theparent’s z-axis by θ. (c)The child ro-tates about its y-axis by ϕ. (d)Thechild rotates about its z-axis by δ.

of each hemi-spherical end) are known as thestart and the end point. The start point is definedas position 0 on the cylinder and the end pointcorresponds to limit parametric distance alongthe cylinder’s local z-axis. A node connectionspecifies how and where to attach each cylinderto the parent. Lpercent indicates the position ona parent cylinder where the child’s 0 point willbe placed, and the value is set to 0-1. The ori-entation of the child cylinder about the parentcylinder is represented by three angles (θ, ϕ, δ)as shown in Figure 3. Initial orientation of thechild cylinder is set as neutral values of eachjoint, and varies over time during the simula-tion. The node connection also has an indicationof scaling values for a child about the cylinder’slength and radius directions.

4 Controller parameters

The controller is defined by a combination ofpiecewise sinusoidal functions similar to [7].The output value of the controller is a desiredvalue of the proportional-derivative controllerwhich is used for calculating the actual motortorque. The controller parameter includes theduration of a cycle and the number of controlpoints. Each control point can have arbitraryvalues between -1 and 1, but all controllersoperate with a cycle of the same duration.Hence, all joint motions can be controlledsynchronously. We can write the controllerfunction U as:

U(t) =pn + pn+1

2+

pn − pn+1

2cos(

Nt

D−n)π

(1)

n = 0, 1, 2, ..., N − 1pN = p0 0 ≤ t ≤ D,

where N is the number of control points, andd is duration of a cycle. The control functionis constructed by smooth interpolation betweeneach pair of neighboring control points. The lastcontrol point is connected back to the first one,which makes the function periodic. These pa-

Figure 4: Low-level controller.

rameters are also coded into each node of thegenotype. Using a periodic sinusoidal functionhas two advantages. First, most locomotion (in-cluding flapping flight) of real animals can beexpressed using a sinusoidal function. Second,the sinusoidal function has an infinite derivativecontinuity so that unexpected fluctuations of ajoint motor can be minimized.

5 Dynamics Simulation

The movement of a physical creature is cal-culated by an articulated rigid body dynam-ics. We used the Open Dynamics Engine byRussell Smith [8] which is an open-source ar-ticulated body simulator. The integrator is afirst order semi-implicit integrator [9], where theconstraint forces are implicit, and the externalforces are explicit. A medium speed (O(n3),where n is the number of links) Lagrange multi-plier method [10] is used to calculate the articu-lated body dynamics.

The forces on each cylinder are calculatedbased on a simplified fluid dynamics by [11].Instead of calculating the forces on every sur-face of the cylinder, the total resistance is di-vided into two terms that act at its center of

3

mass: these are linear resistant force and resis-tant torque [12]. The stream velocity is assumedsimply a negation of the velocity of the centerof mass of the cylinder, each term can be eas-ily obtained as a function of linear and angularvelocity of the cylinder.

Figure 5: The angular resistance of a cappedcylinder.

In case of the linear resistance, the area thatreceives force is idealized as the projection ofthe capped cylinder on to a surface normal to thevelocity vector. Therefore, the linear resistanceof the cylinder can be written as:

F = ρ(2RLsinθ + πR2)vv, (2)

where R and L is the radius and the length notcounting the hemispherical ends. ρ is the densityof the water (1000kg/m3) [13].

For convenience, we approximated a capped-cylinder by a standard cylinder and two hemi-spheres. The resistant torque of a standard cylin-der is calculated by dividing it into slices andintegrating each torque over all cylinder slices.From Equation (2) and Figure 5, the resistanttorques about each local axis of the standardcylinder are derived as

Tcx = −4ρω2

xR

∫ L2

0z3dz · x (3)

Tcy = −4ρω2

yR

∫ L2

0z3dz · y (4)

Tcz = −2ρω2

z(L + 2R)∫ R

0r3dr · z, (5)

where ω denotes the angular velocity about eachlocal axis of the cylinder, and x, y, and z are unitvectors in the cylinder’s local axes directions.

The resistant torques of the cylinder’s each lo-cal axis produced by hemispheres are

Tsx = −ρπR2ω2

x(4R

3π+

L

2)2 · x (6)

Tsy = −ρπR2ω2

y(4R

3π+

L

2)2 · y (7)

Tsz = −ρπR2ω2

z(4R

3π)2 · z. (8)

The term 4R3π represents a length from a hemi-

cycle’s (a cross section of a hemisphere) centralpoint to its center of mass. The angular veloc-ities and torques described above are measuredabout the local axis of each cylinder. Overall re-sistant torque on the capped cylinder is thus:

T = (Tcx + Ts

x) + (Tcy + Ts

y) + (Tcz + Ts

z). (9)

6 Evolution of Basic-MovingCreature

The creature is not necessary to move exactlystraight for a “neutral” movement, because itwill be controlled by a neural network suffi-ciently for a given path. In this stage, the ef-ficiency of locomotion is somewhat more im-portant. The fitness f for a simulation of thestraight-moving creature is defined by the sumof the speed of the creature’s center of mass ateach time step, with the addition of a reward formoving in the direction of the velocity in previ-ous time step (straight movement). At the lastof the simulation, a lineal distance from the startposition is added. Higher values represent betterfitness.

f = kadtot + kb

∫ζ

v(t) · u(t − 1)dt (10)

The symbol ζ denotes the simulation period andk are the weight constants for each term. u is aunit vector in previous velocity direction. Thesimulation speed is improved by prematurelyaborting the simulation of any individual that isnot performing well. A creature is also consid-ered to blow-up if the speed of its body parts ex-ceeds some value (typically 1000m/s), and thatcreature is aborted. Additionally, interim fitnessmeasuring is performed after a quarter of the to-tal evaluation period has elapsed by removing

4

individual whose fitness is less than one fifth ofthat of the least fit creature of the previous gener-ation. After fitness evaluation, a typical geneticoperation (mutation, crossover, grafting) [1] isperformed to produce the next generation.

7 Combining Controllers withNeural Network

The creature control is done by giving variationto the evolved controller of a creature using aneural network. Since we do not have the train-ing samples necessary for path following con-trol, nor the gradient information of numericalphysics simulator needed for the gradient-basedtraining methods such as backpropagation, thenetwork weight is found by a genetic algorithm.

7.1 Neural Network Parameters

Two control values are assigned to a single con-troller which are: scaling s and phase transfor-mation p. The duration of a cycle D is the samefor all the controllers for synchronous motion ofbody parts. The scaling signal is simply multi-plied to the controller output, and a phase trans-formation performs time warping (we use t′ in-stead of t, see figure 8) in a way similar to [14].A bipolar sigmoid function which has a rangeof -1 to 1 is used as the activation function of aneuron.

At the start of each cycle, new values of scal-ing, phase, and duration are obtained by sam-pling the continuous output of the network, andthese values are retained until the cycle ends.The output of the previous state becomes theinput of the network, together with other ex-ternal states of the creature such as the targetposition(T ) and the velocity of the center ofmass (v). These vectors (T ,v) are normalizedand transformed to the local coordinates of aspecific body part of the creature which is de-fined by “central body”. Since we cannot dis-tinguish the body from the legs because of itsirregular shape, the central body is decided onthe body which is closest to the center of massof the initial configuration of the body parts atthe start of a simulation.

The path trajectory is given by a user as apiecewise linear curve. At each simulation time,

Figure 6: Input and output signals of a neuralnetwork. N sets of the scale (s) and thephase transformation (p) are allocatedfor each degree of freedom. For con-venience, input neurons for the targetand swimming direction are describedas a single node.

Figure 7: Obtaining the target position on thetrajectory. The faster creature has amore distant view.

the desired target point is obtained as a functionof the current velocity of the creature’s center ofmass similar to a heading-based approach [15]as shown in Figure 7. The target point is the in-tersection point of the path and the surface nor-mal to the path segment which is closest to thecreature. The surface is determined by this nor-mal and the end point of the vector in the crea-ture’s velocity direction.

7.2 Learning for a Given Trajectory

As shown in Figure 6, the length of achromosome for the genetic algorithm is(H+1)×(4N+9), including connection weightsfrom the bias term, where H is number of hid-den nodes and N is the total number of degreesof freedom in all joints. We use 1 to 2 timesas many hidden nodes as the number of input

5

nodes. The fitness for path following behavioris given by

f =∫

ζ{k1dmin(t) − k2vt(t) − k3aT (t)}dt,

(11)where dmin is the minimum distance from thetrajectory, vt is the speed of the creature in thecurrent path direction, and aT is the accelera-tion towards the target point. k > 0 are weightconstants. Lower values represent better fitness.Each time the fitness evaluation is complete, thepopulation is divided into three groups accord-ing to fitness, giving a strong penalty to last twogroups which show unexpected behaviors. Thefirst such behavior is a blow-up of the simula-tion, and the other occurs when the position ofthe creature’s center of mass deviates seriouslyfrom the path, which we call as heading failure.A creature which does not commit any of thesemisdemeanor (called a survivor), is placed in thefirst group of the population. Heading failuresgo into the second group, and blow-ups go intothe worst group. In those two worse cases, thesimulation is aborted and the fitness is assignedby following metric:

f = P, Q +1ta

∫ ta

0{f(t)}dt. (12)

P and Q are sufficiently huge numbers to dis-tinguish each group, and ta is the time when thesimulation is aborted. The second term indicatesthe mean fitness over the simulated period. Wecheck each criterion only in some time interval(0 ≤ t ≤ tc), which we call the “checktime”.That is because, if we check these criteria in allsimulation periods, few creatures would be sur-vived in the first group (probably most of thecreatures will commit the heading failure). Thechecktime is initially set to some fraction (α) ofthe full duration, and increases gradually as afunction of the number of survivors in the firstgroup

tg+1c = ζ{(1 − α)

Sg

P+ α}, (13)

where g is the generation, S is the number ofsurvivors, and P is the population size.

Figure 8: Time warping for phase transforma-tion.

Figure 9: Neural network output (upper) andresultant controller signal of a joint(lower).

8 Results

In the first stage (evolving simple forward mov-ing creatures), the population size was gener-ally 100, and runs lasted for 100 to 200 gener-ations. The top 20% of genotypes were used aselites, and the remaining 80% of the new gen-eration were created by selecting a couple ofparents by roulette wheel selection, and copy-ing one of them (chosen by tournament selectionwith a 90% probability of selecting the fitter)with probability 30%. The rest of the offspringsare made up of 30% crossovers, 30% grafting,and 10% random individuals. Mutations werethen applied stochastically to the newly gener-ated genotypes. The integration step was gener-ally 0.05 seconds, and the evaluation period wasaround 500 simulation steps.

In the network evolution stage, the popula-tion size was 200-300, and the number of gen-erations was 100-200. The simulation periodwas set to 1000-2000 time steps, depending on

6

the speed of a creature in straight moving. Thesetting for the genetic algorithm were the sameas those for creature evolution, but the numberof crossover points was in the range of 1 to 4.The running time was generally 1-2 hours forbasic-creature evolution and 2-4 hours for op-timizing a neural network, running on a singlePentium-IV 2.53Ghz PC. We can see that thecreature evolution time become very short (inthe Sims’ creatures, it took about 4 hours ona 32-processor connection machine) because ofthe compact search space of possible creaturesreduced by using an efficient controller modelwhich is specific to the periodic motion.

Parameter Value or Range

no. of body parts 3 - 10cylinder length 0.2 - 1.0 m

cylinder Radius 0.1 - 0.4 m

θ 0.0 - π radϕ 0.0 - π radδ 0.0 - π rad

density of cylinder 1000 kg/m3

ρ 1000 kg/m3

ka, kb 0.5, 1.0k1, k2, k3 0.1, 2.0, 1.0

controller duration 0.3 - 1.2 secno. of control points per

controller 2 - 6

Table 1: Some values used in the simulations.

In terms of path-following behaviors, thecreature learned following behavior sufficientlyby optimizing for only a single path, becausea genetic algorithm gave successful global op-timum that satisfies various training samples ofa complicated path during the simulation. How-ever, there were some local optima such as stay-ing near the trajectory with low velocity, or con-tinuing to move straight ahead when the varia-tion of path direction is small.

9 Conclusion and Future Work

We have described a system which generatescontrollable virtual creatures using efficient pe-riodic controller, and showed various path-following behavior trained by a neural network.The creatures are evolved more easily by using

Figure 10: Evolved swimmers. The brighterpart represents the central body of thecreature.

Figure 11: The fitness graphs of the basic crea-ture evolution (left) and the neuralnetwork optimization (right).

simple periodic controllers. By separation of thecreature’s locomotion and its control, we can ac-cess and control evolved creatures more easilystill having creativity of the artificial evolution.

However, in our creature design, we couldhardly see the creature like a watersnake (typ-ically the most life-like behavior as we know)whose motion of a body part is smoothly prop-agated through the following parts. This arisesfrom the restriction that making all body partsto move synchronously for convenience of theneural network training. It seems that this canbe addressed by adding a phase shift term ofthe controller to the genetic description. We ex-pect that this method could also be applied tocreatures in land environments more easily withless running time, because the crawling path hasonly two dimensions. Going a step further, thereis a need for research which investigates theway to evolve other higher-level behaviors, suchas flocking. Another direction for future workmight be to make an entire digital nature with acomplete environment (land, water, sky) includ-ing evolved plants. As computers become morepowerful, we believe that this sort of researchwill be a new frontier, which will give infinitediversity to the virtual world in the near future.

7

Figure 12: Sequential image of creature in basic motion.(sampled at each 30 time steps)

Figure 13: Following motions for three different paths (white lines) and the actual moving trajectoriesof each creature (res dots which are the creature’s centers of mass sampled at each 10 timesteps). First row is a twisted “S” shape which is used for the neural network optimization.Second and third are a spiral and a heart shapes.

8

References

[1] K. Sims. Evolving virtual creatures.In ACM Computer Graphics (SIGGRAPH’94), pages 15–22. 1994.

[2] K. Sims. Evolving 3d morphology andbehavior by competition. In & P. MaesR. Brooks, editor, Artificial Life IV: Pro-ceedings of the Fourth International Work-shop on the Synthesis and Simulation ofLiving Systems, pages 28–39. MIT Press,1994.

[3] J. Ventrella. Attractiveness vs. efficiency:How mate preference affects locomotion inthe evolution of artificial swimming organ-isms. In C. et al. Adami, editor, ArtificialLife VI: Proceedings of the Sixth Interna-tional Conference on Artificial Life, pages178–186. MIT Press, 1998.

[4] M Komosinski and S. Ulatowski. Fram-sticks: Towards a simulation of a nature-like world, creatures and evolution. InD. et al. Floreano, editor, Advances in Ar-tificial Life: Proceedings of the Fifth Euro-pean Conference on Artificial Life, pages261–265. Springer Verlag, 1999.

[5] T. S. Ray. Aesthetically evolved virtualpets. In E. Maley, C.C. & Boudreau, ed-itor, Artificial Life VII Workshop Proceed-ings, pages 158–161. 2000.

[6] H. Lipson and J. B. Pollack. Automatic de-sign and manufacture of robotic lifeforms.Nature, 406:974–978, 2000.

[7] R. Grzeszczuk and D. Terzopoulos. Auto-mated learning of muscleactuated locomo-tion through control abstraction. In Pro-ceedings of SIGGRAPH95, pages 63–70.ACM Press, Los Angeles, Ca, 1995.

[8] R. Smith. Intelligent motion control withan artificial cerebellum. PhD Thesis,Dept of Electrical and Electronic Engi-neering, University of Auckland, Avail-able online including ODE engine athttp://opende.sourceforge.net, 1998.

[9] M. Anitescu and F. A. Potra. Formulat-ing rigid multi-bodydynamics with con-

tact and friction as solvable linear comple-mentarity problems. Nonlinear Dynamics,14:231–247, 1997.

[10] D. E. Stewart and J. C. Trinkle. An im-plicit time-stepping scheme for rigid-bodydynamics with inelastic collisions andcoulomb friction. International J. Numer-ical Methods in Engineering, 39:2673–2691, 1996.

[11] J. Wejchert and D. Haumann. Animationaerodynamics. In number 4 volume 25,editor, SIGGRAPH ’91 Proceedings, pages19–24. 1991.

[12] Y. S. Shim and C. H. Kim. Gen-erating flying creatures using body-brain co-evolution. In ACM SIG-GRAPH/Eurographics Symposium onComputer Animation (SCA 03). ACMPress, 2003.

[13] R. W. Fox and A. T. McDonald. Introduc-tion to fluid mechanics. Wiley, 1976.

[14] X. C. Wu and Z. Popovic. Realistic model-ing of bird flight animations. In Proceed-ing of SIGGRAPH 2003, pages 888–895.ACM Press, San Diego, Ca, 2003.

[15] Personnaz L. Dreyfus G. Rivals, I. andD. Canas. Real-time control of an au-tonomous vehicle : A neural network ap-proach to the path following problem. In5th International Conference on NeuralNetworks and their Applications (NeuroN-imes93).

9