Genetic Algorithm Demonstation System

STATEMENT OF ORIGINALITY

SCHOOL OF COMPUTING

DEGREE SCHEME IN COMPUTING

LEVEL SIX PROJECT

This is to certify that, except where specific reference is made, the work described within this project is the result of the investigation carried out by myself, and that neither this project,

nor any part of it, has been submitted in candidature for any other award other than this being presently studied.

Any material taken from published texts or computerised sources have been fully referenced, and I fully realise the consequences of plagiarising any of these sources.

Student Name (Printed) Benjamin Andrew Murphy

Student Signature ………………………………..

Registered Scheme of Study Computer Games Development (BSc)

Date of Signing 26/11/2015

1

Module Code: SE3U608

Student Name: Benjamin Murphy

Student Number: 12044997

Project Supervisor: Colin Morris

2015 - 2016

Final Year Project (BSc) Computer Games Development: Genetic Algorithm Demonstration System

USING PLAYER MODELLING TECHNIQUES TO CONSTRUCT FITNESS FUNCTIONS IN A GENETIC ALGORITHM

2

Contents

1 Introduction ............................................................................................................................ 4

Objectives ............................................................................................................................... 4

Deliverables ............................................................................................................................ 4

Evaluation ............................................................................................................................... 5

2 Research undertaken and outputs ......................................................................................... 5

2.1 Introduction to Genetic Algorithms ................................................................................. 5

Advanced Crossovers .......................................................................................................... 9

2.2 Player behaviour modelling in games ............................................................................ 11

Using player modelling techniques to evaluate fitness functions.................................... 14

2.3 Research conclusion ....................................................................................................... 15

3 Initial design .......................................................................................................................... 16

3.1 Libraries .......................................................................................................................... 17

3.2 Pseudo code ................................................................................................................... 17

Main Program ................................................................................................................... 17

Board Pieces...................................................................................................................... 18

3.3 Flow Chart ...................................................................................................................... 21

3.4 Game board design ........................................................................................................ 22

3.5 Game Object purposes and functionality ...................................................................... 23

4 Development......................................................................................................................... 24

4.1 Initial Development ........................................................................................................ 24

Game Board ...................................................................................................................... 24

4.2 Adding a genetic algorithm ............................................................................................ 33

Fitness function................................................................................................................. 33

Genetic Algorithm............................................................................................................. 33

Initial Fitness Results ........................................................................................................ 34

Adding all pieces to the genome ...................................................................................... 35

4.3 Player Modelling............................................................................................................. 37

5 Testing ................................................................................................................................... 40

Player Movement ............................................................................................................. 40

6 Evaluation.............................................................................................................................. 41

7 Conclusions ........................................................................................................................... 45

7.1 Aims and objectives ....................................................................................................... 45

3

7.2 Evaluation and results .................................................................................................... 45

8 Future improvements ........................................................................................................... 44

References................................................................................................................................ 45

4

1 Introduction

Aim

To investigate genetic algorithms, and create a genetic algorithm demonstration system in

the form of a game that would demonstrate a genetic algorithm learning how to play

against a human player, player modelling will then be used to improve on fitness functions

based on the actions of a player as they are playing the game.

Objectives Create a simple game to visualise the genetic algorithm organisms while they play

the game to demonstrate them learning to play while giving a visual representation

of a GA.

Write a genetic algorithm that produces a number of generations of organisms to

traverse the game level that will be used to demonstrate it, organisms moves should

be saved and able to load like a playback feature.

Research in and around the subject of Genetic Algorithms where they came from,

what they are used for and their uses in video games and how they may be used as a

form of believable AI in games.

Investigate additional techniques that could enhance behaviour.

Deliverables

A 2D turn based game as the graphical representation of the GA or GAs rendered

using openGL libraries, tiles will have a type each: Empty, Enemy, Trap, and Goal. The

objective of the game will be to get the player from one end of the board to the

other with as little damage taken as possible.

Write a Genetic algorithm to be used in tandem with the 2D game. Its functions to

be based on the moves a player can make. The organisms moves will then be saved

and able to be loaded from file to render specific organisms to demonstrate them on

screen.

A more advanced version of the game could be used where the GA controls the

enemies rather than the player, to demonstrate a more complicated learning

system.

5

Evaluation In order to evaluate the deliverable, the main focus will be on the effectiveness of the

genetic algorithm used and this in combination with the player modelling technique or

techniques used, in order to evaluate the deliverable, it will be given to a number of people

of varying experience with games and of different ages, some may also have a degree of

knowledge in genetic algorithms or player modelling. Each person will be told how the game

should work including a simple explanation as to how the game should learn how a player

plays the game then adjusts its difficulty. They will be asked how well they feel the game

reacts to how they play the game, if any improvements they could think of can be made and

if in their opinion a learning system could work for a varied difficulty based on a player’s

behaviour.

2 Research undertaken and outputs

2.1 Introduction to Genetic Algorithms

A Genetic algorithm is an algorithm modelled on the idea of natural selection where the

elements of an organism are randomised from a set of predetermined traits and then placed

in a population called a generation successive generations are then created using random

elements of the initial organisms these organisms are also selected randomly based on their

fitness to find better solutions to a problem.

A number GAs have been developed by John Holland and his colleges and students at the

University of Michigan, the goals of their research were to explain the adaptive process of

natural systems and to design an artificial system software that would retain the important

mechanisms of natural systems (Goldberg, 1989) (Bäch, 1996)

The goal of using a GA is to find the more optimal solution to a problem what we may

observe when successive generations of organisms are created is that they may have

inherited the best traits of their ancestors and thus this creates an exponential growth to

the fitness of organisms in later generations.

6

A fitness function is a function that determines the value of fitness an organism is worth and

determines the probability that it will be chosen as a parent for the reproduction process,

the method used to determine the fitness value varies for each situation as there are so

many possible situations and possible ways to value fitness. An example of one such

situation could be the use a GA to determine the best times to water a plant where the

fitness value will increase depending on the amount of growth of the plant between the set

time blocks.

Another use of the fitness function could be determine which path is quickest to a

destination by giving a higher fitness value to organisms that reach the destination in the

shortest times. This can be applied to many different situations and in many different ways.

A simple GA can be used on a string of binary digits that’s might denote the on/off position

of a switch the result of the combination of these switches determines the fitness of an

organism without the need for the algorithm to know the inner workings of the program

and only the steps it needs to take in order to get to a solution, Goldberg’s black box is a

good example of this (1989) in (Fig 2.1) we can see that if we were to take a string of binary

digits 0001 this means that the first four switches on the box would be in the off position

and the final switch in the on position, the algorithm does not need to know what this

sequence causes or means inside of the machine but what the payoff of using this

combination is, thus determining the organisms fitness using a fitness function the next

generation is then created using these fitness values to determine the traits of the next

generation of organisms.

7

Fig 2.1 The black box optimization problem with a number of switches to illustrate that

genetic algorithms have no need to know the inner workings of the program or problem.

(Goldberg, 1989)

There are three main stages in the process of a simple genetic algorithm these stages are:

1. Reproduction

2. Crossover

3. Mutation

During the reproduction stage of a genetic algorithm members of the initial generation are

given a fitness level determined by a fitness function this then gives the organisms a

percentage chance of being chosen for the gene pool that will create the next generation,

once they have been assigned a fitness level organisms with higher fitness have higher

probability of being chosen for the gene pool and could even be selected more than once,

once the parent organisms have been selected in the gene pool they are then randomly

assigned a mate to prepare for the crossover step, this can be seen in tables 2.1 and 2.2

where (Tbl 2.1) shows the initial generation of organisms with fitness levels based on the

number they represent (Tbl 2.2) then shows a possible gene pool created from these fitness

values. We can see in table 2.2 that more than one copy of string 2 has been selected

showing that the process can be random.

The fitness in this example is calculated by finding 𝑥 2. (Goldberg, 1989)

8

No. String Value of 𝑥 𝑓(𝑥) Pselect

𝑓𝑖

∑ 𝑓

Expected

Count 𝑓𝑖

𝑓̅

Count

chosen

1 01101 13 169 0.14 0.58 1

2 11000 24 576 0.49 1.97 2

3 01000 8 64 0.06 0.22 0

4 10011 19 361 0.31 1.23 1

Sum 1170 1.00 4.0

Average 293 0.25 1.0

Max 576 0.49 2.0

Tbl 2.1: Shows the initial generation of organisms having performed the fitness function,

and also gives the result of Pselect for each organism. (Goldberg, 1989)

Mating Pool

after

Reproduction

Mate

(Randomly

selected)

Crossover

Site(Randomly

selected)

New

Population

𝑥 𝑣𝑎𝑙𝑢𝑒 𝑓(𝑥)

0110|1 2 4 01100 12 144

1100|0 1 4 11001 25 625

11|000 4 2 11011 27 729

10|011 3 2 10000 16 256

Sum 1754

Average 439

Max 729

Tbl 2.2 Shows the random selection of crossover points for the current gene pool between

mates, it also shows the new population this creates after a single point crossover

operation. (Goldberg, 1989)

The new organisms for the next generation are created in the crossover stage. The parent

organisms are then given a crossover point and everything to the right of the crossover

point is switched between mates creating the new generation of organisms . This is shown in

(Tbl 2.2)

9

The final step is to then mutate organisms this mutation is used sparingly so as not to lose

any important traits organisms may have inherited this is done by randomly selecting a trait

and then changing it to another trait randomly from the available traits (functions) in the

program with the example of binary strings a random bit would be flipped. (Goldberg,

1989)

These stages of a genetic algorithm can be applied to many different problems from

aerospace design to video games, where in aerospace design they can be used for the

design of individual vehicle components or even the propulsion systems, a GA had also been

used to create a guidance navigation control system by David Goldberg at the University of

Alabama. (Anderson, 2003)

But they can also vary in complexity to add more solutions for the system to find. This would

be especially useful in creating an AI in video games as it would make it more believable if

the AI were to make a mistake rather than being able to win the game easily against the

player.

Advanced Crossovers

Advanced crossovers can be used in order to gain very different results as the genomes are

split in different ways for each crossover technique, this could be advantages in a way that

could evolve genomes in such a varying way to avoid getting trapped in local optima in some

cases, a number of advanced crossovers will be mentioned here and what their specific uses

may be in general.

There are many types of crossover to be considered when using a genetic algorithm, varying

in complexity. These different crossover techniques can be used to produce more random

results covering a wider scope than the single point crossover, some of the techniques are

as follows.

(Bäch, 1996)

Multi-point Crossover (Bäch, 1996)

Multi point crossover is when multiple positions are selected at random, the bits of parent

organisms are then exchanged between the odd-numbered and the next even numbered

10

crossover position, the rest of the string is then left unchanged. The multi -point crossover

allows for 𝑧 ≥ 1 crossover points.

Segmented Crossover (Bäch, 1996)

Segmented Crossover is not dissimilar to multi-point crossover, the difference being that the

number of crossover points is not fixed, it can vary around an expectation value. This is done

using segment switch rate, then on average 𝑙 ∙ 𝑝 crossover points are to be expected, where

𝑙 denotes length of the bit string and 𝑝 denotes the switch rate.

Shuffle Crossover (Bäch, 1996)

The shuffle crossover can be used in combination with any form of multi -point crossover.

This is done in order to reduce the positional bias of standard operators.

Shuffle crossover works by randomly shuffling bit positions of both parents in the same way,

it then performs a multi-point crossover operator and lastly returns the unshuffled result.

In a genetic algorithm we can use Schemata, a schema is a similarity template that describes

a subset of data, and can be thought of as a pattern matching device. To use a schema the

current string alphabet can be extended from [0, 1] to [0, 1 , *] where * denotes a “don’t

care” symbol meaning the bit can either be a 0 or a 1. A schema may look as follows *0000

and this would match two possible strings. {10000} and {00000}

Similarly, if the schema were *111* then this would match four possible bit string

combinations. {01110} {11110} {11111} {01111}

Schemata are used as a compact way of communicating similarities between strings of

information.

In general, a particular bit string contains 2𝑙 schemata where 𝑙 = length of the string.

When writing a genetic algorithm, an important part is to consider how the encoding for the

specific application will be performed. GAs usually represent populations of organisms as an

abstract set of string values, these are named Genotypes, before these can be used however

a less abstract form is needed for the application to understand this abstracted form is

11

called a Phenotype. Depending on the application the phenotype can be anything from a

geometric shape, a job schedule or a neural network.

These encoding schemes using a genotype and phenotype are important, as the genetic

operators such as the fitness function and random selection work directly through the

decoding process, if these are not set up correctly for the purposes of the appl ication, then

the GA could just turn into a random search without the benefits of a genetic algorithm.

(Thomas, 2006)

2.2 Player behaviour modelling in games

Player modelling is the act of creating a model of player behaviour based on the past actions

of a player to determine the way they play a game, this information would then be used to

create a behaviour model in which AI can act upon thus knowing how to react to certain

play styles, the modelling of a player does not necessarily have to affect an AIs behaviour

toward players but in a simpler manner could change variables such as a players health or

strength in order to shift the balance of power in games and create a developing challenge

for the player.

In video games in particular genetic algorithms are used to find an optimum solution to a

problem. There are many different problems that may be solved in games AI, but not all of

them are best suited to the task in hand. For example, pathfinding may be best suited by the

use of the A* pathfinding algorithm, a GA could solve a pathfinding problem but could run

through many generations before finding a good solution to the problem. Whereas a GA

would work best when the elements of the problem it has to solve are unpredictable, such

as a player’s movements which cannot be predicted accurately but can be assumed once

they have played the game at least once, by learning how the player plays the game a

fitness function can be applied to get better results from the GA. This would allow for the AI

to adapt to the player’s style to create a challenging and possibly believable opponent.

(Bourg & Seemann, 2004)

In this project, the aim is to create a simple game that will use a Genetic algorithm and

player modelling technique to learn how a player plays the game, making it more of a

challenge the more someone plays it. The game as a result of this will serve as a

12

demonstration system, demonstrating how a GA learns over time keeping the better traits

of ancestor generations. Console outputs will be used to output the fitness values and

identifier values of each generation, while the game will act as the visual representation of

how a genetic algorithm improves results over time.

Player modelling is the act of analysing a players actions based on a current state, and then

using that data to determine what kind of patters or behaviour the player is more likely to

use. This is especially useful in games as it could determine how many objects of each type

should be placed for a specific player, or how particular opponents should act in a given

situation.

One such method of player modelling was based on the psychology of a player, Yannakakis

et al. investigated the cognitive modelling of players and then focused on applying these

models to optimise the player satisfaction and adjust controllable game parameters in real-

time to achieve this. This was done using a game called Bug Smasher on “Playware”, in this

game the objective to smash as many bugs as possible by stepping on a the lit tiles that

indicated a bug, using data including the Presented bugs, Yannakakis. et al. used the

sequential forward selection feature(SFS) technique and applied it to construct a quantative

user preference model from game data, for individual players. The SFS method is a bottom-

up greedy search where one feature is added at a time to the current set of features, The

feature to be added is selected from the set of remaining candidate feature set so that the

new set generates the maximum value of performance over other candidates in their

experiment. (Yannakakis & Hallam, 2009)

There are many methods that could be used, to place enemies and traps randomly in the

game to tailor the difficulty to the player based on the results of analysing how they play

like using self-organizing maps, or quantitative models which have also been built to assist

the learning of basic NPCs in quake 2 by id software to control bot behaviour dependent on

the situation they are faced with, meaning what pickups are near and the current status of

the enemy bot.

(Mahalm, et al., 2010) (Software, id, 1997)

13

Another method of modelling for AI is called the Rubber band technique, this method of AI

changes the difficulty of the game as the player gets better the game becomes more of a

challenge, while as the player does not perform so well the difficulty will become less

challenging. An example of the Rubber band technique was used in the racing game, Mario

Kart: Double Dash, where in this game if the player is further ahead in the position then

enemy AI karts would increase their max speed variable if the inverse where to happen and

the player were lower in position then the max speed of enemy karts would decrease.

(Yasuyuki & Katsuhisa, 2007) (Missura & Gärtner, 2009)

There are a few methods of player modelling each have the same underlying goal, which is

for the challenge of a game to be maintained for less skilled and more skilled players alike

through the adjustment of the difficulty, without putting the higher skilled players at a

disadvantage. The aim is for a balance between challenge and rewards, so that players do

not become frustrated and give up entirely, or feel that the game is too easy therefore

abandoning the game. (Charles, et al., 2005)

An adaptive game design must be specific so that it allows for a wide variety of playstyles,

but it must also be general enough so that it could be applied to other genres of games.

When constructing an adaptive difficulty player models can be used, based on how the

player plays the game e.g. “Aggressive”, “Defensive”, “Avoidance” these traits could be

assigned values based on the player’s movements, and then in tern affect how many

enemies or power ups may appear in the game. (Fig 2.1) shows a potential framework for

an adaptive system while (Fig 2.2) shows the effect of a players actions on behaviour traits.

14

Fig 2.1: A framework for adaptive game system (Charles, et al., 2005)

Action modelling is a straigtforward method of modelling the actions of a player which

consists of a list of the game states and each one combined with one or more player actions,

and the likelyhood that the player will undertake such an action in a state, (Fig 2.2) shows

the process of this. Usually in practice, action models are not implemented as a list of game

states assosciated with actions as the number of game states are very often numerous, and

so they may be implemented as a set of rules for instance that test the current game state

and generate an appropriate action. (Bakkes, et al., 2012)

Using player modelling techniques to evaluate fitness functions

In creating a fitness function for the genetic algorithm it could be based on the way a player

behaves in the game as depicted in (Fig 2.3) Where variables are used to determine the

players behaviour which in turn would tailor the fitness function to that specific players

behaviour e.g. if a player where to be of the “Avoidance” type then the fitness function

might evaluate organisms that spread out further in the game field as being fittest of the

current generation, if “Aggressive” then the function may evaluate organisms that stay

nearest to the player’s goal to be fittest.

There are many ways this might be used to change the variances of a game, if a variable

were added to count how many times the player has failed then this could also be used by

the fitness function to determine that the player may need a little more help thus adding

15

more power ups to the game, of course this being a genetic algorithm it is not guaranteed to

help the player in the first generation based on this information.

Fig 2.3 Figure showing the effect player moves have on behaviour/modelling types,

dependent on the state of object positions or game states.

2.3 Research conclusion

As can be seen from research there are many methods of using genetic algorithms in a

number of different areas, in this project Goldberg’s simple Genetic algorithm will be used

and explored when creating the demonstration system as it is as the name suggests a simple

GA this is included in GA lib which will be used as the deliverables main Genetic algorithm,

this library can also has a number of other genetic algorithms that could be used for

comparing and contrasting in the deliverable if time permits. (Goldberg, 1989) (Wall, 1996)

Given that the library that will be used to create the deliverable includes the crossover

operations mentioned in section 2.1, it will allow the possibility to explore the different

methods of crossover and contrast them when observing its effects on the population of

organisms. In regards to player modelling there are many possible methods of modelling

that can be used to fit different situations such as yannakakis et al and their modelling

16

techniques used to create quantative models for player satisfaction in the game “Bug

smasher”, but in this project the aim is to use a simpler player modelling technique to

construct a fitness function for the genetic algorithm so that it can learn how a player is

playing then apply this to the fitness function. The player modelling technique that will be

used is, the action modelling technique, where variables that depict the player’s behaviour

are changed based on the move a player makes and the current state of the game this

method is explained in section 2.2.1 with Fig 2.3 Which shows a visual representations of a

simple action model.

3 Initial design

When designing the initial demonstration system a few things should be considered like the,

libraries that will be used, the objects to be built in the game and how the program flow will

be. A flow chart will be designed to show how the program should run and when data

should be processed.

The game itself will be a top down board like game where the player navigates tile by tile in

up, down, left, and right directions, the goal of the game is simple the player must reach the

goal tile without losing all health and hitting as little enemies as possible, the player will also

need to avoid trap tiles which cause game over on one hit. The enemies in the game will be

placed randomly on initialization by a genetic algorithm the goal of this project is to

adaptively change the difficulty by placing the enemies using a genetic algorithm and its

fitness function. This will then be incorporated with player modelling techniques in order to

hopefully create a more accurate difficulty that can be seen by the player.

The Genetic algorithm used will likely be the simpleGA class available in GAlib, as the name

suggests it is a simple genetic algorithm that uses the one point crossover method, fitness

will then be determined by a number of factors per game such as the players health and the

number of moves taken to get to the goal tile, later player modelling will be incorporated to

determine how the player behaves and respond in kind by changing the range of tiles

enemies can appear be it closer or further away depending on behaviour.

17

3.1 Libraries

Open GL libs

The Open GL libs that will be used in the program for the rendering of the game board and

their movement around it are:

GLUT – OpenGL utility toolkit that is used to create a windowing application

programming interface, makes it easier to learn and use explore OpenGL

programming. (OpenGL, 1997)

GLEW – an open source library with thread-safe support which provides easy-to-use

methods for checking OpenGl extensions and functionality. This will be used to

render the application.

GAlib

GAlib is a genetic algorithm library which includes GA objects. The library also includes tools

for using a GA to do optimization in any C++ program, using any representation and any

generic operators. This library uses templated classes to achieve this, and so it will be used

to demonstrate a genetic algorithm as it is used for the game by evaluating fitness based on

player modelling.

The use of this library will also give the opportunity to compare and contrast different

methods of crossover and selection that it includes. (Wall, 1996)

3.2 Pseudo code

Here is the pseudo code that outlines the process of how the program should run. In the

pseudocode 𝐺 represents Generation and 𝑡 represents time or generation number.

Main Program Start program

Get moves list (Directions enemies can move)

Initialise 𝐺0 using GAlib

Save Gen

Start GA Loop

Insert Population 𝐺0

18

Generate Grid/ Game Board

Start game loop

Player Move

Render

Enemies Move

Render

End Game Loop

Evaluate player model

Perform fitness function

Create new Generation

Save Gen

Insert Population 𝐺𝑡

End GA loop when finished

Board Pieces

For each of the board pieces these include the player, enemy, trap and goal pieces to be

rendered they will be similar if not the same in some areas due to them being the same

shape and drawn in the same manner, each piece will be a separate class object in order to

have the ability to create as many as is needed easily and change values and positions easily

the following is pseudo code for each of these classes.

Enemy Class

Get available positions

Set enemy position

Load shader

Bind enemy information to buffer

Draw enemy

19

End

Trap Class


Set trap position

Load shader

Bind trap information to buffer

Draw trap

End

Player Class


Set player position

Load shader

Bind player information to buffer

Draw player

End

20

Goal Class


Set goal position

Load shader

Bind goal information to buffer

Draw goal

End

As can be seen from the pseudocode for each of the board pieces they will be very similar in

their structure and implementation, so much so that the only difference for some objects

will be the colour and the shape.

21

3.3 Flow Chart

Here a flow chart will be used as a visual representation of the how the program should look

in terms of its flow, this flow chart is based on the previous pseudocode Fig.3.1 shows this.

Fig 3.1 This figure represents the flow that the program will take and what should happen

at each stage.

22

3.4 Game board design

For the game to be an effective demonstration, it must have a clear representation of the

different objects of the game such as the Goal, Enemies, Player and the game grid itself. The

current pieces for the design are as follows and how they shall be represented via colour.

Normal grid tile – Yellow

Enemy – a smaller square to fit into normal tiles, possibly a different shape to make

clearer. These will be coloured red.

Goal – Green grid tile

Traps – black centre normal tile colour outline.

Player – Blue tile smaller than normal tile, possibly a different shape to make clearer

(Fig.3.2) shows a rough representation of what the game board may look like. While

information about generations if needed will be output in a console window.

23

Fig.3.2 This figure shows a rough design of how the game board may look, this design may

be improved later.

3.5 Game Object purposes and functionality

In the game the objects that are used will have different functionalities, in order for it to be

considered a game with each piece having its own set of constraints or effects on other

objects these are as follows.

Player

The player tile represented by a blue triangle in (Fig 3.2) will be able to move in the

horizontal directions up, down, left or right by means of the user pressing the corresponding

keys on the keyboard. The player’s objective is to reach the green goal tile without touching

an enemy or trap tile.

Enemy

The enemy object represented by a red triangle in (Fig 3.2) will have the ability to move

horizontally one tile at a time in the given four directions up, down, left and right. The

enemy objects objective will be to touch the player tile and in doing so cause the game to

end.

Trap

The trap object represented by a black square in (Fig 3.2) will have the ability to destroy the

player character if they are to move onto that tile, this tile would not have any special

properties or the ability to move as its objective is to only be a hindrance or pitfall for the

player to fall into. In future iterations of the deliverable it is possible that the trap will be

placed based on the results of player modelling, but for now will be static.

Goal

The goal tile represented by a green tile in (Fig 3.2) will be the objective that the player

seeks to get to it will not have any special properties or be able to move other than the

ability to end the game as a win for the player. This tile may be placed randomly on the grid

for each game, making sure not to overlap any other tiles, and may also then be improved

to be based on the results of player modelling.

24

4 Development

In this section the development of the deliverable will be discussed and documented, any

changes that are made along the way if any changes are made they will be explained and

documented here.

4.1 Initial Development

Game Board On creating the game board a 8x8 grid has been set up using openGL to display a set of tiles

in yellow, this can be seen in (Fig 4.1) the next step for the game board is to add the

individual objects that will vary, these include the Enemies, Traps and the player objects as

described in section. (3.5)

Game Grid Setup and Draw PsuedoCode

Set object amounts

Load Positions ready to display on screen

Start Draw Loop

Load Tile Shader and Draw

Load Enemy Shader and Draw

Load Trap Shader and Draw

Load Goal Shader and Draw

Load Player Shader and Draw

Update Enemy Positions if player moves

End Draw Loop when game ends

End Grid Setup

25

Fig 4.1: This Figure shows the grid as setup and drawn with only the tiles drawn.

The Enemy tiles in the game access an array of available positions, defined as an array of

structure objects which contain floats that determine where the enemy will be drawn on

screen, in this case the grid size is 8x8 therefore the tile position array spans from 0-63

these positions then determine where the enemy can be drawn and can be set with the set

position function by passing an integer to it. Figs 4.2 and 4.3 show a number of enemies

being drawn on screen using this method.

The player object shares the same data as the enemies for positioning therefore is set up

using the same positional data but can act independently of them, the player can also move

by pressing the directional keys set which may be changed at a later date, which are currently

I, J, K, and L.

Figs 4.4 and 4.5 show a player moving from one tile to another.

26

Fig 4.2: Shows a number of enemies being drawn using set array positions. These were set with a for loop for simplicity and to quickly display them working.

Fig 4.3: Shows another set of enemies using different array positions to show how they can

be moved around the screen.

27

Fig 4.4: Shows the player being drawn having been placed in a particular position.

Fig 4.5: Shows the player being drawn one square below +8 in array position from its original position in Fig 4.4

The trap tile is setup using the same method as the enemy objects and player object but use

a different structure object array as they need more points to draw therefore contain more

data fig 4.5 shows a number of traps added to the screen, while fig 4.6 shows the last

28

object, the goal position drawn on screen, this object uses the same data to set its position

like with the enemies and player objects.

Fig 4.5: Shows a number of traps placed on screen using a random number generator to

show that they can be placed in a number of different places, this is true for all objects

available.

Fig 4.6: Shows a goal object has been placed on the screen.

29

The next step in the development is to add rules to the objects so that they can interact

with each other, such as if the player were to hit a trap tile then the game is over or if an

enemy is hit then the enemy object will disappear and the player’s health value will

decrease and so on. These interactions will be important as they will also determine a score

later on to be used in the fitness function when a genetic algorithm is added, for the first

iteration of the development player modelling may not be used to make the process better

understood and implemented.

A few rules have been added to the objects of the game in order to make it more like a

game, these include: player health – decreases when an enemy is hit, goal – causes game to

end when player hits it, the player health value will also be used in the fitness function to

calculate the fitness in the GA.

Fig 4.7 and 4.8 show the game ending when the player reaches the goal tile, while fig4.9 and

4.10 show the player health decreasing after hitting an enemy.

Fig 4.7 – Player before hitting the game goal

30

Fig 4.8 - Shows the game ending when the player hits the goal, when this happens input is

requested in the console window asking if the game should continue and evolve the current

GA this should then output a result file.

Fig 4.9 – Shows before the player interacts with an enemy tile.

31

Fig 4.10 – Shows the result of the player interacting with enemy tiles, where the enemies

disappear and the players score is reduced and displayed in the console window, in this

figure it can be seen that the player’s health has reduced by two and two enemy tiles have

been removed from the board.

The trap tiles rules have been made so that if the player were to touch one then the game

will end immediately no matter the player’s health this is demonstrated in fig 4.11 and fig

4.12

32

Fig 4.11 – This shows the game before the player touches the trap tile next to it the next fig

will show what happens afterwards where the game should end and the user prompted to

continue or not.

33

Fig 4.12 – Shows the result of the player hitting the trap tile, then being prompted to

continue the game or quit.

4.2 Adding a genetic algorithm The next step for the demonstration system is to add a genetic algorithm to control the positions of

the enemy pieces, to do this galib will be used to provide a genetic algorithm with population of

individuals that contain a set of numbers between 0 and 64 these will then determine what position

an enemy is placed in. (Wall, 1996) To determine a fitness for the individuals of a generation a

number of things will need to be considered, like the health of the player and what may have

occurred during the game, such as if the player were to hit a trap the game would end this should

give a high fitness value (when this is implemented for traps as well as the enemies) if the player

reaches the goal then the fitness will be determined by the number of moves made by the player

and health remaining.

Fitness function This is an early model of how the fitness will be calculated to determine an individual’s

fitness, Player Health denoted as H, number of moves as M, and Fitness as F, this will be

used in addition to the Objective function found in example 1 in the manual for GAlib. (Wall,

1996)

𝑭 = 𝑴 − 𝑯

This fitness function will need to be improved in order to make a more varied range of

results when the game evolved the Genetic Algorithm

Genetic Algorithm

The type of genetic algorithm that will be used to begin with is the SimpleGA type for

simplicity this could be changed further on in development to conduct tests and compare

the outputs of each type, the GABin2DecPhenotype and GABin2DecGenome will be used in

order to be able to have a string of integer types to be used as the enemy position values.

Different Genetic Algorithms and types may be used throughout the process of producing

results to compare how they differ in variance and accuracy.

34

Initial Fitness Results These are some of the initial results gained from the fitness function using 𝐹 = 𝑀 − 𝐻 these

playthrough’s of the game were simple without any difficult mechanics therefore the results

are also simple the player would simply go toward the goal to try and make the game place

enemies nearer to the goal and better the fitness, these results are shown in Tbl 4.1.

The next table labled Tbl 4.2 shows the same thing as before in Tbl 4.1 With a modified

Fitness function, where the fitness is also divided by the size of the genome this is denoted by

n in Tbl 4.2.

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

Fitn

ess

Generation

F = M - H

11.6

11.8

12

12.2

12.4

12.6

12.8

13

13.2

13.4

0 5 10 15 20 25 30 35

Fitn

ess

Generation

F = M-H / n

35

As can be seen from these two charts the first fitness function is not very effective as it loses

a degree of variance when based solely on two variables by dividing with the size of genome

it reduces the number below zero giving the results a broader range, what can also be seen is

that the second fitness function has a steadier curve when producing results meaning it is

less likely to be trapped in local optima.

Adding all pieces to the genome Then next set of data will show the effect that the second fitness function has on initial

results, after adding all peaceable pieces to the genome meaning that all pieces including the

player, enemies, traps and goal will be determined by the numbers produced in the genetic

algorithm. Tbl 4.3 shows a series of games played, in this series we can see that at some

points the fitness will change significantly and maybe get caught in local optima for a while

before being able to move on due to the way the player plays the game.

Tbl 4.3 This table shows a series of tests with the improved Fitness function, but includes all

pieces available to the game board.

By looking at the initial results of these fitness functions it can be seen that they may not be

accurate enough to make the pieces form a good genome against the player, therefore some

changes will need to be made such as adding constraints to pieces such as traps should not

be in the same place as the goal or starting position therefore should be randomised again if

that happens, these pieces should also be given a radius where traps cannot be placed.

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930

Fitn

ess

Test

F = G-M/n (All Pieces)

36

Now that all pieces have been added to the genome another fitness function that uses a part

of the original fitness function provided in an example from GALib documentation, this

fitness function gives a more gradual increase in fitness the new equation used:

𝐹 =𝐻 + 𝑀

𝑛𝑃ℎ𝑒𝑛𝑜𝑡𝑦𝑝𝑒𝑠𝑖 − 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑖

Where nPhenotypes denotes the number of phenotypes in the current selection and the

sequence of the selection, this makes for a more accurate fitness in terms of gameplay as it

would make sense for the fitness to increase for the game pieces the better it does at

keeping the player from the goal.

In the next test a distance between the player and enemies will be calculated at game end,

this will also be done for traps these values will then be included as an addition to the fitness

function to determine that the closer the pieces are to the player the better.

Tbl 4.4 Shows the resulting fitness values from adding the distance between the closest

enemy and the player, this results in higher fitness values but also makes the increase in

fitness a more gradual increase.

Tbl 4.4 shows the fitness values generated from adding Distance denoted as D in the table

0

20

40

60

80

100

120

140

160

180

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930

Fitn

ess

Generation

F = H + M + D / nPhenotypes - sequence

37

4.3 Player Modelling The next stage of the development is to add the player modelling elements to the game, this

is done by adding variables that will determine the behaviour of the player based on actions

taken and the positions of enemies on the board, these will be evasion and aggression to

begin with as they are a simpler behaviour to observe. The aggression variable will increment

whenever the player makes a move that brings it closer to an enemy, the opposite will

happen if the player moves further away increasing the value of evasion instead.

The values produced by the end of each round are then used to add to the fitness score in

the hopes to gain a more accurate result and have new positions appear closer to the goal

more often. To improve upon this a range value will be implemented to decide on a range of

tiles the enemies will appear in depending on the values of aggression and evasion.

The first set of results shown in Tbl 4.5 shows the result of adding the calculated aggression,

the aggression variable is calculated by taking the current position of the player then

determining weather or not the players new position is closer to an enemy, if it is then

aggression is incremented.

After a number of tests, the values of fitness drastically increase due to the inclusion of the

genome sequence therefore this was removed and replaced with only the genome size, the

new equation is as follows and outputs a more reasonable set of fitness values, A denotes the

aggression variable.

𝐹 =𝐻 + 𝑀 + 𝐷

𝑔𝑆𝑖𝑧𝑒+ 𝐴

0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30 35

Fitn

ess

Test

F = (G+H+D / gSize) + A

38

Tbl 4.5 shows the results of the fitness values over 30 games, as can be seen from these

results there is a far more gradual incline in fitness than in previous tests with a number of

jumps throughout, this could be attributed to the change of removing the number of

phenotypes from the fitness function.

In the next set of results, the evasion variable has been added and works much in the same

way that the aggression variable does, this variable increments whenever the player gets

further away from an enemy and is deducted from the overall fitness when fitness is

calculated as we want the player to move closer to enemies and get caught by them, not to

avoid them. The results of adding this to the fitness function is shown in Tbl 4.6 and the new

equation is as follows, E denotes the evasion variable.

𝐹 = (𝐻 + 𝑀 + 𝐷

𝑔𝑆𝑖𝑧𝑒) + 𝐴 − 𝐸

Tbl 4.6 Shows the result of adding Evasion to the fitness function, as can be seen from these

results they are varied and do not get trapped in local optima often, and act in a similar

manner to which the fitness function with only the added aggression variable behaves.

While manipulating the fitness function by using these variables could be a suitable way for

increasing the difficulty it is not difficult enough to be a playable game, therefore movement

will be added to the game and manipulated through the aggression and evasion variables to

0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30 35

Fitn

ess

Test

F = (H+M+D / gSize) + A - E

39

create a more believable game environment, it was decided to use the player modelling in

this way as due to the limitations of GAlib it is not possible to change the range of values

distributed through the random number generator after initialization, therefore another

method must be used in which the player modelling will affect which direction enemies will

travel.

The direction of travel will be decided by the evasion and aggression variables, if the

aggression is higher than evasion then the enemies will move towards the goal, if the

opposite is true then the enemies will move closer to the player.

To illustrate this console outputs were used to signify which of the statements is true and

how the enemies are moving, this is shown in Fig 4.13.

Fig 4.13 Shows the output of a few moves depending on the evasion or aggression of the

player.

40

5 Testing

During the course of developing the program some elements would need testing, in this

section the tests carried out to ensure that ranges were within the values they should have

been will be documented and any problems that arose and how these were then fixed.

Player Movement One issue that appeared during development is how the player could go off the game board

due to no checks being in placed to stop this from happening, to fix this a check to determine

if the player would go off the board was used if so then the move would be cancelled so that

the player would not disappear, this would also fix any possibility of accessing memory that is

out of bounds of the arrays.

Distribution range

When setting the position values of objects on the game board, each object must be within

the range of 0-63 therefore console output was used when adding the variables to the array

that would hold these positions the result of this is shown in Fig 5.1

Fig 5.1 Shows the positions given to the first generation of objects and stays within the range

that it should be.

41

6 Evaluation

To evaluate the program demonstrating a genetic algorithm and player modelling learning

how a player plays the game, a series of questions have been presented to people of varying

ages and experience with games, the goal of these questionnaires is to determine how well

the program does what it was supposed to and if any improvements could be made to the

program, in order to make it clearer or more understandable.

Age = 16

How often do you play video games?

Very often

What is your understanding of the subject?

Minimal

How well was the intended outcome of the program explained out of 10?

9

How well did the program perform to your understanding out of 10?

10

If anything can be done to improve the intention of the program what would it be?

Screen wrap movements confuses the player.

Age = 9


Very often


Minimal


9


10


42

Age = 33


Often


I understood the objective of the game


8


6


Clear win/lose conditions with an easier reset button

Age = 26


Often


I understood partially


8


7


Make it more aesthetically pleasing.

43

Age = 21


Every day, at least 2 hours a day :) (PC MASTER RACE)


The algorithm should be able to determine what the player is going to do based on previous

results from playing the game. With this in mind you shouldn't be able to finish the game after a

certain amount of attempts.


10


7


I did find it learnt my movement very slowly, perhaps use a faster learning method? Also the

program kept crashing after about 7-8 attempts, might be on my end.

As can be seen from the questions and answers above a large majority of the people who

tested the game believed it to be rather clear what the program should do but in some cases

did not prove to be clear enough whilst playing the game, some improvements suggested

and conclusions will be discussed in the conclusion section.

44

7 Future improvements

To improve the implementation in this project in future a number of things can be done,

some are suggestions made by people who have played the game for evaluation purposes

others are personal improvements due to the way some things in the game were added or

implemented.

The first of the mentioned improvements is that the game should not allow enemies or the

player to wrap around the board when moving so as not to cause confusion when playing,

this could be remedied by adding a better wrap around so that enemies and the player end

up on the opposite side of where they go off the board like in the game pac-man.

Another improvement as suggested by the testers of the program is that the win and loss

conditions should be shown clearly in the main render window as changing from the render

window to the console window is confusing and distracting, this could cause an issue for

some users as the distraction of moving focus from one window to another to continue the

game and the frustration of the game window freezing could cause users not to notice the

changes when starting a new round.

Others found that the game was learning too slowly, therefore some improvements to the

player modelling could be made such as making it more complex than the current system

which only detects if the player gets closer to enemies or further away to determine if they

are being “Aggressive” or “Evasive”.

Other improvements could be made in terms of the code itself such as making classes that

were used for creating objects such as enemies, traps and the player could be made more

ambiguous so as to be easily changed in future to make code neater and easier to read and

understand.

To make the program easier for users to understand a user interface could be designed

which would display stats such as health, and would display a win or loss at the end of each

round, UI could also display information pertaining to the way the player modelling is

interpreting player moves by out putting a message to say if it thought the move an

aggressive one or an evasive one.

45

8 Conclusions

8.1 Aims and objectives The Aim of this project was to create a program that would demonstrate the use of a

genetic algorithm with the added objectives of the incorporation of player modell ing

techniques in order to enhance the behaviour of the learning in some way.

While the other two objectives of the project were met due to a demonstration system

having been built using a simple game as a platform, where most users found that the game

had shown what it was intended to show for the most part.

8.2 Evaluation and results To evaluate if the program met this aim and the objectives in the introduction a number of

questions were given to a number of people with varying ages and experience playing

games, the concept of the game and what it should be doing was explained before each

individual answered the questions, the results of the questionnaires above have shown that

for most individuals they understood the concept of the game and what it was trying to

accomplish, they also found that the game was learning to a degree how they played the

game but not at a satisfactory pace, as some have said that it was too slow to learn the

players behaviour.

Others found that the colours themselves for the game may not have been vibrant enough

or no clear indication in game was given as how to win or lose, and continuing the game was

difficult to understand due to having to switch between the rendered window and the

console output window.

Another issue that some ran into while playing the game is that while the game was waiting

for input in the console window the render window would crash as the user tried to input

without realising the program was waiting for input.

Others found that due to the way the player and enemies move when they reach the edge

of the board it confused them and they were unable to understand how the game would

end sometimes.

In terms of completing aims and objectives it can be said that the project has met its aim to

create a system that demonstrates the use of a genetic algorithm, but has much room for

improvement, while a number of the objectives were met for this project some were not,

46

which in turn changed the development of the overall deliverable such as the ability to

create a move list using the genetic algorithms in GAlib and then storing them to be able to

play back previous games, this was not done unfortunately due to time constraints and

problems that arose but player movement based on the actions of a player was added to

the project in its stead which gave the desired outcome.

While using player modelling techniques in order to model how a player plays the game and

react to each move has been a success it could also be improved in a number of ways as the

implementation of this was simple in this project and could be a more complex algorithm to

determine what to do.

47

References

Anderson, M. B., 2003. Genetic Algorithms In Aerospace Design: Substantial Progress,

Tremendous Potential, Eglin Air Force Base, , FL 32542, USA: s.n.

Bäch, T., 1996. Genetic Algorithms. New York: Oxford University Press.

Bakkes, S. C., Spronck, P. H. & Lankveld, G. v., 2012. Player behavioural modelling for video

games. Entertainment Computing - Games and AI, 3(3), pp. 71-79.

Bourg, D. M. & Seemann, G., 2004. AI For Game Developers. First Edition ed. USA: O'Reilly

Media, Inc..

Charles, D. et al., 2005. Player Centered Game Design: Player Modelling and Adaptive Digital

Games. Digital Research Association 2nd Confrence,Vancoover, p. 14.

Goldberg, D. E., 1989. Genetic Algorithms In Search Optimization And Machine Learning.

Alabama: Addison-Wesley.

Mahalm, T. et al., 2010. Predicting Player Behaviour in Tomb Raider: Underworld. IEEE

Symposium on Computational Intelligence and Games, pp. 178-185.

Missura, O. & Gärtner, T., 2009. Player modeling for intelligent difficulty adjustment. Lecture

Notes in Computer Science, Volume 5808, pp. 197-221.

OpenGL, 1997. opengl. [Online]

Available at: https://www.opengl.org/resources/libraries/

[Accessed 11th November 2015].

Software, id, 1997. Quake 2. s.l.:Activison.

Thomas, D., 2006. Encoding Schemes and Fitness Functions for Genetic Algorithms. In: S.

Rabin, ed. AI Game Programming Wisdom 3. 1st ed. Boston, Massachusetts: Charles River

Media, pp. 677-686.

Wall, M., 1996. GAlib: A C++ Library of Genetic Algorithm Components. Massachusetts: s.n.

Yannakakis, G. N. & Hallam, J., 2009. Real-time Game Adaptation for Optimizing Player

Satisfaction. IEEE Transactions on Computational Intelligence and AI in Games, 1(2), pp. 121-

133.

48

Yasuyuki, O. & Katsuhisa, S., 2007. RACING GAME PROGRAM AND VIDEO. United States,

Patent No. US 7,278,913 B2.

49

Appendix

Degree Scheme in Computing

FINAL Project Assessment and Comment Form 2015-16

Student Benjamin Andrew Murphy

Project Title: Genetic Algorithm Demonstration System

Supervisor: (1 or 2) Colin Morris

Mark Category

Proposed Weighting

Ranges

Agreed Weighting

Mark Allocated

Project Management (Only set by Supervisor 1)

50 – 80

Originality & Self-Direction 40 – 80 50

Technical Complexity 20 – 80 40

Solutions, Evaluation & Conclusions

80 – 120 80

Final & Sub-Report Quality 50

Prototype / System Demo

Or Project Deliverable

50 – 100 60

Sponsor Mark

00 – 60 0

Sub-Total Marks -------

Sub-Total Percentage ------- 60% %

Milestone 1 (initial) Research

Research Applied to Design Prototype Development

-------

20%

%

Final Presentation (including

poster production)

------- 20% %

TOTAL PERCENTAGE ------- 100%

%

The weightings for the various aspects of the project are to be set at the initial objectives

setting stage. The ranges shown are guidelines and the actual weighting set may be outside of

these ranges if deemed appropriate.

50

FINAL YEAR DEGREE PROJECT

Objective Settings Proforma (to be completed and submitted to your supervisor by Friday 16th October,

2015)

Student’s Name: Benjamin Andrew Murphy

First Assessor: ...........................................................

Second Assessor: ...........................................................

Project Title:

Final Year Project (BSc) Computer Games Development: Genetic Algorithm Demonstration System

Using Player Modelling techniques to construct Fitness functions in a Genetic Algorithm

Project Objectives & Deliverables

Objectives

Create a simple game to visualise the genetic algorithm organisms while they play

the game to demonstrate them learning to play while giving a visual representation

of a GA.

Write a genetic algorithm that produces a number of generations of organisms to

traverse the game level that will be used to demonstrate it, organisms moves should

be saved and able to load like a playback feature.

Research in and around the subject of Genetic Algorithms where they came from,

what they are used for and their uses in video games and how they may be used as a

form of believable AI in games.

Investigate additional techniques that could enhance behaviour.

51

Deliverables

A 2D turn based game as the graphical representation of the GA or GAs rendered

using openGL libraries, tiles will have a type each: Empty, Enemy, Trap, and Goal. The

objective of the game will be to get the player from one end of the board to the

other with as little damage taken as possible.

Write a Genetic algorithm to be used in tandem with the 2D game. Its functions to

be based on the moves a player can make. The organisms moves will then be saved

and able to be loaded from file to render specific organisms to demonstrate them on

screen.

A more advanced version of the game could be used where the GA controls the

enemies rather than the player, to demonstrate a more complicated learning

system.

Please tick this box to indicate your awareness of the university’s policy on

ethical issues

The deliverables and objectives can often change due to unforeseen circumstances, or through the student’s

research causing the project to follow a different path. If this is the case, and the project objectives change

significantly, then the first assessor should make a note of the date and fill in a new objectives proforma,

which should also be included as an appendix to the project report. The project organiser is to be consulted at

this stage.

Genetic Algorithm Demonstation System

Documents

Transcript of Genetic Algorithm Demonstation System