Post on 18-Dec-2015
Hierarchical Plan Representations for
Encoding Strategic Game
AI
Hai HoangStephen Lee-UrbanHéctor Muñoz-Avila Lehigh University
http://www.cse.lehigh.edu/~munoz/projects/AIGames/
Outline Motivation Related Work
Background
Unreal Tournament© (UT) Hierarchical task Network (HTN) Planning
Encoding Strategic Game AI in HTNs Coordinating teams of UT Bots using HTNs Video
Using Processes-Based Languages to Encode HTNs Final Remarks
Future work
Motivation: FSM vs. Planning
Pla
nn
ing
Op
erat
ors
•PatrolPreconditions: No MonsterEffects: patrolled
•FightPreconditions: Monster in sightEffects: No Monster
Patrol Fight
Monster In Sight
No Monster
FSM:
A resulting plan:
Patrolpatrolled
Fight
No MonsterMonster in sightNeither is more powerful than the other one
But Planning Gives More Flexibility
“Separates implementation from data” --- Orkin
inference knowledge
Pla
nn
ing
Op
erat
ors
•PatrolPreconditions: No MonsterEffects: patrolled
•FightPreconditions: Monster in sightEffects: No Monster…
Many potential plans:
PatrolFightPatrolFightPatrolFightPatrolFight
PatrolFight
…
Motivation: HTNs vs. GOAP
Goal-Oriented Action Planning (Orkin, 2004, 2005) characters can find alternate solutions to situations
encountered in the game characters can handle dependencies that may not have
been thought of at development time HTNs provide distinct advantages over “STRIPS” planning,
on which GOAP is based Key questions:
What kinds of Game AI knowledge to represent in HTNs?
How to reason in dynamic game environments? How to encode the HTNs
Goals & operators
Game developers should Learn LISP?
Related Work: FPS and AI
Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999)
FPS-based system developed and used for actual training of military personnel (Laird & Duchim, 2000)
Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000)
GameBots: A group at USC-ISI and CMU jointly developed the Gamebot project to create an API for other clients to connect to the UT server.
Related Work: HTNs and Game AI Hierarchical planning can be useful in games
(Wallace, 2003)
Bridge Baron game uses HTN ideas (Smith et al., 1998).
Hierarchical FSMs are an extension of FSMs in which states can expand into their own sub-FSMs (Houlette & Fu, 2003)
HTNs used for dialog generation (Cavazza & Charles, 2005)
Unreal Tournament©
Online FPS developed by Epic Games Inc.
Various gameplay modes including team deathmatch and capture the flag
Provides a client-server
architecture for controlling
bots
High-level script
language: UnrealScript
(Sweeney, 2004)
Client-Server Architecture for UT Bots The UT server provides sensory information about
events in the UT world and controls all the gameplay events.
Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server.
Two types of messages: Asynchronous: special events Synchronous: regular intervals
Variants: Enhanced server (USC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan).
Hierarchical Task Network (HTN) Planning Complex tasks are decomposed into simpler tasks.
Seeks to decompose compound tasks into primitive tasks, which define actions changing the world
The knowledge artifacts indicating how to decompose task into subtasks are called methods
task t
task t task t
Why HTN Planning (1)?
HTN planning is provable more expressive than STRIPS planning (Erol et al, 1994)
Even for those domains in which HTN representations can be translated into STRIPS representations, a much larger number of STRIPS operators is required to represent the HTN methods (Lotem & Nau, 2000)
Fast planning through domain-configurable HTN planners (SHOP system)
Why HTN Planning (2)?
Hierarchical planning is natural in many domains Military planning
Tactical
StrategicTheater
CINC
JCS / NCAStrategicNational
JTFOperational
Application: UT Domination Games
A number of fixed domination locations.
When a team member steps into one of these locations, the status of the location changes to be under the control of his/her team.
The team gets a point for every five seconds that each domination location remains under the control of that team.
The game is won by the first team that gets a pre-specified amount of points.
Idea
Use of hierarchical task network (HTN) planning techniques to laid out a grand strategy to accomplish gaming tasks
Use standard event-driven programming allowing the bots to react in highly dynamic environments while contributing to the grand task
UT task: Domination
Strategy: secure most locations
UT action: move Bot1 to location B
Technical Difficulties of using an HTN Planner (1)
UT task: Domination
Strategy: secure most locations
UT action: move Bot1 to location B
1. Declarative language for expressing conditions
2. Dealing with contingencies while Bots are executing actions
Solution to 1: use built-in predicates to represent conditions
Solution to 2: use Java Bots to represent actions (event-driven)
Data Flow to generate HTNs (i.e., HTN Planning)
task
compound? Yes Select applicable method
UT Server
Methodlibrary
Evaluate method’s preconditions
Selected method’s subtasks
No
Execute actionUT Bot
UT Bot …
Video
Empirical Results
0
10
20
30
40
50
60
1 2 3 4 5 avg
HTN
Improved
It is possible to encode strategies that coordinate teams of bots in FPS games using HTNs
Using Processes to Encode Game AI
Think-Act Cycle
Shoot Something Pick up a HealthpackGo Somewhere Else
Call ShootOperator
Ask Decision System:Where Do I Go?
Ask Decision System:Where Do I Go?
Call PickupOperator
TIELT Project: Middleware between Games and AI Decision Systems Decision
SystemLibrary
DecisionSystemLibrary
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
DecisionSystem
DecisionSystem
Learning Module
. . .
Learning Module
TIELTUser
TIELT’sInternal
CommunicationModules
TIELT’s KBEditors
TIELT’s KBEditors
Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases
GamePlayer(s)
TIELT’s User Interface TIELT’s User Interface
EvaluationInterface
TIELTUser
SelectedGame
Engine
SelectedGame
Engine
SelectedDecisionSystem
SelectedDecisionSystem
Raw StateProcessed
State
DecisionAction
Knowledge Base Libraries
Knowledge Base Libraries
Learned Knowledge
(inspectable)
PredictionInterface
CoordinationInterface
AdviceInterface
GameModel
Agent Description
GameInterface
Model
Decision System
InterfaceModel
ExperimentMethodology
GIMGIM
GIM
DSIMDSIM
DSIM
GMGM
GM
ADAD
AD
EMEM
EM
GameEngineLibrary
GameEngineLibrary
Stratagus
Full Spectrum Command
...
EE2
Result: Processes can Be Translated into HTNs
TIELT HTN
Iterations (while) Recursion
Iterations (for) Convert to while, recursion
Tasks: preconditions and effects Methods + operators
If-elseIf-else Method
Functions Parameters of task (or call to external code, or Horn clause)
Set-statement = command
Priorities If-then-else
Final Remarks
HTNs can be used to model effective team strategies for Unreal Tournament (UT) bots.
A grand strategy is laid out by the HTNs and event-driven programming allows the bots to react in this highly dynamic environment while contributing to the grand task.
Processes languages (e.g., TMKLs) can be translated into HTNs
Future Work
Reinforcement learning to learn adequate task decomposition
Case-based reasoning techniques to capture and reuse HTNs
Experiment to measure performance of translated processes