KI2 - 10

Kunstmatige Intelligentie / RuG

Markov Decision Processes

AIMA, Chapter 17

2

Markov Decision Problem

How to use knowledge about the world to make decision even when the outcomes of an action are uncertain and the payoffs will not be obtained until several (or many) actions have passed.

3

The Solution

Sequential decision problems in uncertain environments can be solved by calculating a policy that associates an optimal decision with every state that the agent might reach

=> Markov Decision Process (MDP)

4

Example

+1

-1

1

2

3

1 2 3 4

start

0.1

0.8

0.1

The world Actions have uncertain consequences

5

6

7

8

9

10

11

Utility of a State Sequence

Additive rewards

Discounted rewards

...)()()(...]),,([ 22

10210 sRsRsRsssU h

...)()()(...]),,([ 210210 sRsRsRsssU h

12

13

The utility of each state is the expected sum of discounted rewards if the agent executes the policy

The true utility of a state corresponds to the optimal policy *

Utility of a State

sssREsUt

tt

00

,)()(

14

15

Algorithms forCalculating the Optimal Policy

Value iteration

Policy iteration

16

Calculate the utility of each state

Then use the state utilities to select an optimal action in each state

Value Iteration

/

)(),,()( //* maxargsa

sUsasTs

17

Value Iteration Algorithm

function value-iteration(MDP) returns a utility function local variables: U, U’ initially identical to R repeat U U’ for each state s do

end until close-enough(U, U’) return U

/

)(),,()()( //maxsa

sUsasTsRsU

Bellman update

18

The utilities of the states by value iteration algorithm

The Utilities of the States ObtainedAfter Value Iteration

+1

-1

1

2

3

1 2 3 4

0.705 0.655 0.611 0.388

0.762 0.660

0.9120.8680.812

19

Policy Iteration

Pick a policy, then calculate the utility of each state given that policy (value determination step)

Update the policy at each state using the utilities of the successor states

Repeat until the policy stabilizes

20

Policy Iteration Algorithm

function policy-iteration(MDP) returns a policy local variables: U, a utility function, , a policy repeat U value-determination(,U,MDP,R) unchanged? true for each state s do

unchanged? false end until unchanged? return

/

)(),,()( //maxargsa

sUsasTs

thensUsssTsUsasTifssa

//

)()),(,()(),,( ////max

21

Value Determination

Simplification of the value iteration algorithm because the policy is fixed

Linear equations because the max() operator has been removed

Solve exactly for the utilities using standard linear algebra

22

+1

-1

1

2

3

1 2 3 4

u(1,1) = 0.8 u(1,2) + 0.1 u(1,2) + 0.1 u(1,1)

u(1,2) = 0.8 u(1,3) + 0.2 u(1,2)

…

Optimal Policy(policy iteration with 11 linear equations)

23

Partially observable MDP (POMDP)

In an inaccessible environment, the percept does not provide enough information to determine the state or the transition probability

POMDP– State transition function: P(st+1 | st, at)– Observation function: P(ot | st, at)– Reward function: E(rt | st, at)

Approach– To calculate a probability distribution over the possible

states given all previous percepts, and to base decision on this distribution

Difficulty– Actions will cause the agent to obtain new percept, which

will cause the agent’s beliefs to change in complex ways