Governor’s School for the Sciences

MathematicsDay 7

MOTD: Andrei A. Markov

• 1856 to 1922• Stochastic

processes• Markov Chains

Markov Process

• State Diagram:

• Represent by System of Linear Difference Equations

• Determine changes in the probabilities for being in a certain state

Key: Transition Matrix

• N states => NxN Transition matrix T• T(i,j) = probability of changing from

state j to state i• X(n) vector of probabilities (at time n)• X(n+1) = T*X(n) (Markov Process)• Properties

1. T never changes 2. At each time, all the probabilities in X(n) sum to 1 3. Columns of T sum to 1

Boardwork

• Keywords: States, Distribution, Transition Matrix

• Disease model (well, sick, dead)• Dice game (roll 1-5 in order before

getting a 6)• Best of 3 Series

Disease Model

Dice Game

Dice Game: Average Position

Best of 3 Series

• Equal prob of winning:

¼ probability of ending at (0,2), (1,2), (2,1), (2,0)

• Team A has prob ¾ of winning game: 9/16 prob of (2,0), 9/32 (2,1) 1/16 prob of (0,2), 3/32 (1,2)

Exercise

• Two states: Not dormed & Dormed• Transition Probabilities (per day)

ND -> ND 0.8 ND -> D 0.2 D -> ND 0.9 D -> D 0.1

• Construct the transition matrix• Determine if you start as Not dormed

what is the prob of you being dormed after 1 day, 2 days, 3 days

Results

• A state is an absorbing state if once you get there you can’t leave

• If a system has only one absorbing state, then the distribution will tend towards that state

• If it has 2 absorbing states, then the distribution will tend towards some split of those two states (depending on the initial distribution)

• If it has no absorbing states then the distribution will tend towards some stable distribution

Break Time

Higher order DEs

• 2nd order: x(n+1) = f(x(n), x(n-1))• Fibonacci sequence:

x(n+1) = x(n) + x(n-1)• Take x(0) = 1, x(1) = 1, then we get

2, 3, 5, 8, 13, 21, 34, 55, 89, …• Take x(0) = 2, x(1) = 4, then we get

6, 10, 16, 26, 42, 68, 110, …

Interesting facts

• For any sequence generated by the Fibonacci rule: x(n+1)/x(n) -> ( = (1+sqrt(5))/2; the Golden ratio)

• Many things in nature seem to follow the Fibonacci rule: seeds in a sunflower, ‘leaves’ on a pine cone (and other plants), etc.

Solving Linear DEs

• Solve x(n+1) = a*x(n)• Solve x(n+1) = a*x(n) + b*x(n-1)• Solve x(n+1) = x(n) + x(n-1)• Solve x(n+1) = a*x(n) + b• Solve x(n+1) = a*x(n) + b*x(n-1)

+ c

System of Difference Equations

• Two sequences: {x(n)} and {y(n)}• Difference equation relating both:

x(n+1) = f(x(n), y(n)) y(n+1) = g(x(n), y(n))

• Example: Predator-Prey x(n+1) = 1.1*x(n) – 0.2*y(n) y(n+1) = 0.2*x(n) + 0.95*y(n) x – rabbits, y - foxes

Lab

• Building transition matrices for a Markov process and studying the results

• T = [0.2, 0.3; 0.8, 0.3]; sum(T)• X = [1;0];• T*X• X(:,i+1) = T*X(:,i)