FeUdalNetworks for Hierarchical Reinforcement Learning

FeUdal NetworksforHierarchicalReinforcementLearning

byArtem BachysnkyiComputationalNeuroscienceSeminar

UniversityofTartu3May2017

Reinforcementlearning

The basic reinforcement learning model consists of:

• a set of environment and agent states S • a set of actions A of the agent• policies of transitioning from states to actions• rules that determine the scalar immediate reward of a

transition • rules that describe what the agent observes.

ATARIgames

Standart approach

• useanaction-repeatheuristic,whereeachactiontranslatesintoseveralconsecutiveactionsintheenvironment

• notapplicableinnon-Marcovian environmentsthatrequirememory• can’tlearnontheweakrewardsignal

Feudalreinforcementlearningintuition

• levelsofhierarchywithinanagentcommunicateviaexplicitgoals• goalscanbegeneratedinatop-downfashion• goalsettingcanbedecoupledfromgoalachievement

Manager-Workermodel

Manager:• setsgoalsatalowertemporalresolution

Worker:• operatesatahighertemporalresolution

• producesprimitiveactions• followsthegoalsbyanintrinsicreward

Mainproposals

• aconsistent,end-to-enddifferentiablemodel• approximatetransitionpolicygradientupdatefortrainingtheManager• useofgoalsthataredirectionalratherthanabsolute• dilatedLSTMfortheManagerRNNdesign

FuN modeldescription

ℎ",ℎ# – internalstates𝑈% – workersoutput𝜙 – maps𝑔% into𝑤%𝜋 – vectorofprobabilitiesoverprimitiveactions

𝑠% – latentstaterepresentation𝑔% – goalvector𝑥% – observationfromtheenvironment𝑧% – sharedintermediaterepresentation

Learning

Learningsteps:1. receivesanobservationfromtheenvironment2. selectanactionfromafiniteset3. theenvironmentrespondswithanewobservationandascalar

reward4. theprocesscontinuesuntiltheterminalstateisreached

LearningBadidea:

trainfeudalnetworkend-to-endusingapolicygradientalgorithmoperatingontheactionstakenbytheWorker

Goodidea:independentlytrainManagertopredictadvantageousdirectionsinstatespaceandtointrinsicallyrewardtheWorkertofollowthesedirections

Theagentsgoal

Maximizethediscountedreturn

Theagent’sbehaviour isdefinedbyitsaction-selectionpolicyπ.FuN producesadistributionoverpossibleactions.

Managersupdaterule

– valuefunctionestimatefromtheinternalcritic

– cosinesimilarity

– advantagefunction

Workersintrinsicreward

𝑐 – horizon

TheWorkerspolicy

Advantageauthorcritic

Advantagefunction

Architecturedetails

𝑓012310%– ConvolutionalNeuralNetwork:1. 168x8filters,stride42.324x4fil- ters ofstride23.fullyconnectedlayerhas256hiddenunits*eachlayerisfollowedbyarectifiednon-linearity

𝑓"40531 – anotherfullyconnectedlayer𝑓#266 – standardLSTM𝑓"266 – dilatedLSTM

FuN modeldescription

DilatedLSTMStateofthenetworkwith𝑟 separategroupsofsub-states

Attime𝑡 wecanindicatewich groupofcoresisupdated

Ateachtimesteponlythecorrespondingpartofthestateisupdatedandtheoutputispooledacrossthepreviouscoutputs.ThisallowsthergroupsofcoresinsidethedLSTM topreservethememoriesforlongperiods.

*Intheexperimentsr=10.

Experiments:ATARI

Experiments:Montezuma’srevenge

https://www.youtube.com/watch?v=_zbg9rs5QZY

Experiments:Montezuma’srevenge

Experiments:Non-matchandT-maze

Experiments:Watermaze

Experiments:transitionpolicygradient

Experiments:Temporalresolution

Experiments:DilateLSTMagentbaseline

FeUdalNetworks for Hierarchical Reinforcement Learning

Documents

Transcript of FeUdalNetworks for Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning Ersin Basaran 19/03/2005.

FeUdal Networks for Hierarchical Reinforcement Learningproceedings.mlr.press/v70/vezhnevets17a/vezhnevets17a.pdf · FeUdal Networks for Hierarchical Reinforcement Learning ager and

Hierarchical Memory-Based Reinforcement Learning

Hierarchical Reinforcement Learning for Air-to-Air Combat

Introduction to Hierarchical Reinforcement Learning

A Non-Strict Hierarchical Reinforcement Learning for Interactive ...

Hierarchical Deep Reinforcement Learning through Scene Decomposition … · 2020. 8. 19. · Hierarchical Deep Reinforcement Learning through Scene Decomposition for Autonomous Urban

Hierarchical Reinforcement Learning Amir massoud Farahmand Farahmand@SoloGen.net.

Mechanisms of Hierarchical Reinforcement …ski.clps.brown.edu › papers › FrankBadre12.pdfAdvance Access publication June 21, 2011 Mechanisms of Hierarchical Reinforcement Learning

Interpretable Hierarchical Reinforcement Learningdivy.at/UGP_ReinforcementLearning.pdf · Keywords: Reinforcement Learning, Meta Learning, Multi-goal Generalization, Hierarchical

Learning (Part II) Hierarchical Reinforcement

Hierarchical Approaches to Reinforcement Learning Using ... · the decision space (hierarchical reinforcement learning). In this work we ask whether we can use ideas introduced in

Reinforcement Learning-based Hierarchical Seed Scheduling for …csong/ndss21-afl-hier.pdf · 2021. 1. 14. · Reinforcement Learning-based Hierarchical Seed Scheduling for Greybox

Hierarchical Reinforcement Learning - WSUtaylorm/14_580/Beiyu.pdf · 2014. 4. 15. · Reinforcement Learning Hierarchical Reinforcement Learning: To discouver and exploit hierarchical

Hierarchical Reinforcement Learning for Course Recommendation …lfs.aminer.cn/misc/moocdata/publications/AAAI19-zhang-et... · 2019. 4. 3. · Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning in Multi-Agent Environmentprojekter.aau.dk/projekter/files/61065342/1118927942.pdf · area of Hierarchical Reinforcement Learning. First a hierarchical

Reinforcement Learning of Hierarchical Skills on the Sony Aibo …web.eecs.umich.edu/~baveja/Papers/icdl06.pdf · Reinforcement Learning of Hierarchical Skills on the Sony Aibo robot

Hierarchical Reinforcement Learning with …papers.nips.cc/paper/8421-hierarchical-reinforcement...these challenging tasks [6]. In addition, Hierarchical Reinforcement Learning (HRL)

Latent Space Policies for Hierarchical Reinforcement Learning

Visual Tracking via Hierarchical Deep Reinforcement Learning