Hierarchical Plan Representations for

Encoding Strategic Game

AI

Hai HoangStephen Lee-UrbanHéctor Muñoz-Avila Lehigh University

http://www.cse.lehigh.edu/~munoz/projects/AIGames/

Outline Motivation Related Work

Background

Unreal Tournament© (UT) Hierarchical task Network (HTN) Planning

Encoding Strategic Game AI in HTNs Coordinating teams of UT Bots using HTNs Video

Using Processes-Based Languages to Encode HTNs Final Remarks

Future work

Motivation: FSM vs. Planning

Pla

nn

ing

Op

erat

ors

•PatrolPreconditions: No MonsterEffects: patrolled

•FightPreconditions: Monster in sightEffects: No Monster

Patrol Fight

Monster In Sight

No Monster

FSM:

A resulting plan:

Patrolpatrolled

Fight

No MonsterMonster in sightNeither is more powerful than the other one

But Planning Gives More Flexibility

“Separates implementation from data” --- Orkin

inference knowledge

Pla

nn

ing

Op

erat

ors

•PatrolPreconditions: No MonsterEffects: patrolled

•FightPreconditions: Monster in sightEffects: No Monster…

Many potential plans:

PatrolFightPatrolFightPatrolFightPatrolFight

PatrolFight

…

Motivation: HTNs vs. GOAP

Goal-Oriented Action Planning (Orkin, 2004, 2005) characters can find alternate solutions to situations

encountered in the game characters can handle dependencies that may not have

been thought of at development time HTNs provide distinct advantages over “STRIPS” planning,

on which GOAP is based Key questions:

What kinds of Game AI knowledge to represent in HTNs?

How to reason in dynamic game environments? How to encode the HTNs

Goals & operators

Game developers should Learn LISP?

Related Work: FPS and AI

Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999)

FPS-based system developed and used for actual training of military personnel (Laird & Duchim, 2000)

Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000)

GameBots: A group at USC-ISI and CMU jointly developed the Gamebot project to create an API for other clients to connect to the UT server.

Related Work: HTNs and Game AI Hierarchical planning can be useful in games

(Wallace, 2003)

Bridge Baron game uses HTN ideas (Smith et al., 1998).

Hierarchical FSMs are an extension of FSMs in which states can expand into their own sub-FSMs (Houlette & Fu, 2003)

HTNs used for dialog generation (Cavazza & Charles, 2005)

Unreal Tournament©

Online FPS developed by Epic Games Inc.

Various gameplay modes including team deathmatch and capture the flag

Provides a client-server

architecture for controlling

bots

High-level script

language: UnrealScript

(Sweeney, 2004)

Client-Server Architecture for UT Bots The UT server provides sensory information about

events in the UT world and controls all the gameplay events.

Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server.

Two types of messages: Asynchronous: special events Synchronous: regular intervals

Variants: Enhanced server (USC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan).

Hierarchical Task Network (HTN) Planning Complex tasks are decomposed into simpler tasks.

Seeks to decompose compound tasks into primitive tasks, which define actions changing the world

The knowledge artifacts indicating how to decompose task into subtasks are called methods

task t

task t task t

Why HTN Planning (1)?

HTN planning is provable more expressive than STRIPS planning (Erol et al, 1994)

Even for those domains in which HTN representations can be translated into STRIPS representations, a much larger number of STRIPS operators is required to represent the HTN methods (Lotem & Nau, 2000)

Fast planning through domain-configurable HTN planners (SHOP system)

Why HTN Planning (2)?

Hierarchical planning is natural in many domains Military planning

Tactical

StrategicTheater

CINC

JCS / NCAStrategicNational

JTFOperational

Application: UT Domination Games

A number of fixed domination locations.

When a team member steps into one of these locations, the status of the location changes to be under the control of his/her team.

The team gets a point for every five seconds that each domination location remains under the control of that team.

The game is won by the first team that gets a pre-specified amount of points.

Idea

Use of hierarchical task network (HTN) planning techniques to laid out a grand strategy to accomplish gaming tasks

Use standard event-driven programming allowing the bots to react in highly dynamic environments while contributing to the grand task

UT task: Domination

Strategy: secure most locations

UT action: move Bot1 to location B

Technical Difficulties of using an HTN Planner (1)

UT task: Domination

Strategy: secure most locations

UT action: move Bot1 to location B

1. Declarative language for expressing conditions

2. Dealing with contingencies while Bots are executing actions

Solution to 1: use built-in predicates to represent conditions

Solution to 2: use Java Bots to represent actions (event-driven)

Data Flow to generate HTNs (i.e., HTN Planning)

task

compound? Yes Select applicable method

UT Server

Methodlibrary

Evaluate method’s preconditions

Selected method’s subtasks

No

Execute actionUT Bot

UT Bot …

Video

Empirical Results

0

10

20

30

40

50

60

1 2 3 4 5 avg

HTN

Improved

It is possible to encode strategies that coordinate teams of bots in FPS games using HTNs

Using Processes to Encode Game AI

Think-Act Cycle

Shoot Something Pick up a HealthpackGo Somewhere Else

Call ShootOperator

Ask Decision System:Where Do I Go?

Ask Decision System:Where Do I Go?

Call PickupOperator

TIELT Project: Middleware between Games and AI Decision Systems Decision

SystemLibrary

DecisionSystemLibrary

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

DecisionSystem

DecisionSystem

Learning Module

. . .

Learning Module

TIELTUser

TIELT’sInternal

CommunicationModules

TIELT’s KBEditors

TIELT’s KBEditors

Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases

GamePlayer(s)

TIELT’s User Interface TIELT’s User Interface

EvaluationInterface

TIELTUser

SelectedGame

Engine

SelectedGame

Engine

SelectedDecisionSystem

SelectedDecisionSystem

Raw StateProcessed

State

DecisionAction

Knowledge Base Libraries

Knowledge Base Libraries

Learned Knowledge

(inspectable)

PredictionInterface

CoordinationInterface

AdviceInterface

GameModel

Agent Description

GameInterface

Model

Decision System

InterfaceModel

ExperimentMethodology

GIMGIM

GIM

DSIMDSIM

DSIM

GMGM

GM

ADAD

AD

EMEM

EM

GameEngineLibrary

GameEngineLibrary

Stratagus

Full Spectrum Command

...

EE2

Result: Processes can Be Translated into HTNs

TIELT HTN

Iterations (while) Recursion

Iterations (for) Convert to while, recursion

Tasks: preconditions and effects Methods + operators

If-elseIf-else Method

Functions Parameters of task (or call to external code, or Horn clause)

Set-statement = command

Priorities If-then-else

Final Remarks

HTNs can be used to model effective team strategies for Unreal Tournament (UT) bots.

A grand strategy is laid out by the HTNs and event-driven programming allows the bots to react in this highly dynamic environment while contributing to the grand task.

Processes languages (e.g., TMKLs) can be translated into HTNs

Future Work

Reinforcement learning to learn adequate task decomposition

Case-based reasoning techniques to capture and reuse HTNs

Experiment to measure performance of translated processes