Markov Chains
J. Alfredo Blakeley-Ruiz
The Markov Chain
• Concept developed by Andrey Andreyevich Markov
• A set of states that can be inhabited at a moment in time.
• These states are connected by transitions
• Each transition represents the probability that a state will transition to another state in discrete time.
• The transition from one state to another state depends only on the current state, not any states that may have existed in the past.– Markov property
Explained Mathematically
Given a series states X1, X2,…, Xn
As long as
Markov chains can be represented as a weighted digraph
http://www.mathcs.emory.edu/~cheung/Courses/558/Syllabus/00/queueing/discrete-Markov.html
Markov chains can also be represented by a transition matrix
1 2 3 4
1 0.6 0.2 0.0 0.2
2 0.6 0.4 0.0 0.0
3 0.0 0.3 0.4 0.3
4 0.3 0.0 0.0 0.7
Peter the Great
• Lived 1672-1725• Tsar: 1682-1725• Ushered Russia into the modern era
– Changed fashions – beard tax– Modernized Russian military– Changed capital to St. Petersburg– Created a meritocracy– Founded Academy of Sciences in
St. Petersburg• Imported many foreign experts (Euler)• Later native Russians began to make their mark• Nikolai Lobachevsky – non-Euclidian geometry• Pafnuty Chebyshev – Markov’s advisor
• Some not so good things– Huge wars of attrition with Ottomans and Swedish– Meritocracy only for the nobles and clergy– New tax laws turned the serfs into slaves
https://en.wikipedia.org/wiki/Peter_the_Great
Jacob Bernoulli
• Born: Basel Switzerland– 1654-1705
• University of Basel• Theologian, astronomer, and
mathematician– Supporter of Leibniz in calculus
controversy – Credited with a huge list of
mathematical contributions• Discovery of the constant e• Bernoulli numbers• Bernoulli’s golden theorem – Law of
large numbers
The law of large numbers
• The relative frequency, hnt, of an event with probability p = r/t, t = s + t, in nt independent trials converges to probability p.
• This also often called the weak law of large numbers.
https://en.wikipedia.org/wiki/Law_of_large_numbers
Adolphe Quetelet
• Born: Ghent, French Republic– 1796-1874
• Alma Mater: University of Ghent• Brussels Observatory
– Astronomer– Statistician– Mathematician – Sociologist
• From a sociological perspective he concluded that while free will was real. The law of large numbers demonstrated that it didn’t matter.
https://en.wikipedia.org/wiki/Adolphe_Quetelet
Pafnuty Chebyshev
• Born in Akatovo, Russian Empire
– 1821-1894
• Alma mater: Moscow University
• Professor: St. Petersburg University
– Andrei Markov
– Aleksandr Lyapunov
• Chebyshev inequality
– Proves weak law of large numbers
https://en.wikipedia.org/wiki/Pafnuty_Chebyshev
Chebyshev inequality
• Let X be a sequence of independent and identically distributed variables with mean 𝜇 and standard deviation 𝜎.
• X = (x1,...,xn)/n
• <X> = 𝜇
• Var(X) = 𝜎2/n
• Chebyachev’s inequality states that for all 𝜀 > 0.P(|X-𝜇|≥ 𝜀) ≤ Var(X)/𝜀 = 𝜎2/(n𝜀)
→ lim𝑛→∞
P(|X−𝜇| ≥ 𝜀) ≤ lim𝑛→∞
𝜎2/(n𝜀) = 0
Pavel Nekrasov
• Lived: 1853-1923
• University of Moscow
• Monarchist and supporter Orthodox Church
• Attempted to use LLN to prove free will
• Came up with a proof for LLN using independent variables.
• His 1902 paper on these two topics inspired Markov to invent Markov chains.
http://bit-player.org/wp-content/extras/markov/#/33
Andrey Andreyevich Markov
• Born: Ryazan, Russian Empire– 1856-1922
• Alma Mater: St. Petersburg University– Advisor Pafnuty Cheyshev
• Professor: St. Petersburg University• Notable achievments
– Published more than 120 papers– Chebyshev inequality – Markov inequality– Invention of Markov Chain
• Proving that the Law of Large numbers could apply to dependent random variables
• Markov had a prickly personality
“I note with astonishment that in the book of A. A. Chuprov, Essays on the Theory of Statistics, on page 195, P. A. Nekrasov, whose work in recent
years represents an abuse of mathematics, is mentioned next to
Chebyshev.”
The Feud
• Nekrosov and Markov where the opposite in every respect
– Moscow vs. St. Petersburg
– Monarchist vs. Anti-Tsarist
– Religious vs Secularist
• These have as much of a role to play in the future perception of these two mathematicians as mathematics
Nekrosov’s argument for free will (1902 paper)• Nekrosov disagreed with Quetelet• Used Chebyech inequality to prove law of large
numbers for specific independent 𝜀• Showed Independent variables -> law of large numbers• Conjectured that independent variables where
necessary for the law of large numbers• Conjectured voluntary acts can be considered like
independent trials in probability theory.• Stated that law of large numbers had been shown to
hold true for social behaviors.• Argued that this was proof of free will
Markov’s counter argument
• Markov did not care about the philosophical arguments and was only interested in the mathematics
• Nekrosov and others assumed that the law of large numbers applied only to independent events
• Markov invented markov chains and used them to prove that the law of large numbers could apply to dependant events
Markov’s 1906-1907 paper
Ground work• In his first Paper on Markov chains, Markov
considered two states and • A simple chain was an infinite series x1,x2,…,xk,
xk+1– Where k is the current time, and xk is the state at time
k.
• For any k, xk+1 was independent of x1,x2,…,xk-1given that xk is known.
• This chain was also time homogeneous in that xk+1 given xk was independent of k
Some Variables
P, = Probability of event xk+1= , Given xk=
𝑃𝛽𝑘+1 = ∑𝛼𝑃𝛼
𝑘𝑃𝛼,𝛽
ai = expected value of independent variable xi
𝐴𝛾𝑖 = 𝐸 𝑥𝑘+𝑖 𝑥𝑘 = 𝛾 = Expected value of xk+I,
Given kk = 𝛾
Markov’s Theorom
Theorem: For a chain with a positive matrix, all
numbers ak+1 and 𝐴𝛾𝑖 have the same limit, which
they differ from by numbers < ∆𝑖 . At the same time ∆𝑖 < Chi, where C and H are constants and 0<H<1.
This theorem shows that the limit of the probability of the next variable converges to zero with both independent variables and dependent variables given a markov chain.
“I am concerned only with ques-tions of pure analysis.... I refer to the question of the applicability of prob-
ability theory with indifference.”
Markov’s 1913 Paper
Markov’s experiment on Pushkin’s Eugene Onegin
Hayes, B 2013
Simple Markov Chains can be used to create a simple weather prediction model
Hayes, B 2013
Simple Markov Chain in Economics
Recession Prediction Social Class Mobility Prediction
ng mr sr
ng 0.97 0.29 0
mr 0.145 0.778 0.77
sr 0 0.508 0.492
http://quant-econ.net/jl/finite_markov.html
poor middle class
Rich
poor 0.9 0.1 0
middleclass
0.4 0.4 0.2
rich 0.1 0.1 0.8
Chemical Kinetics: As a stochastic process
• 𝑎𝑘1↔𝑘2
b
– k1 rate that a transition to b
– k2 rate that b transition to a
• Modeled by a linear chain where each state is a different number of a and b molecules– x1,x2,...,xk,xk+1
– If the state xk there are 50 molecules of a and 0 molecule of b what will be the state at xk+1.
Chemical Kinetics Example
• a0=5
• b0=0
• 20 iterations
• k1= 1
• k2= 1
To emphasize randomness
Chemical Kinetics Example
• a0 = 50
• b0 = 0
• 100 iterations
• k1= 1
• k2= 1
Chemical Kinetics Example
• a0 = 50
• b0 = 0
• 100 iterations
• k1= 5
• k2= 1
Other applications of simple markovchain
• Genetic drift
• Google page rank algorithm
• Social sciences
• Games
– Snakes and ladders
– Monopoly
Hidden Markov Model
• So far we have discussed observable Markov Models
• Sometimes the underlying stochastic process in a system cannot be observed
• Observations resulting from the process can be used to infer the underlying stochastic process
• Developed by Leonard E. Baum and Co.
Hidden Markov Model
• X(t) = state at time t
• X(t) ∈ {x1,…,xn}– n = # unobservable
states
• Y(t) ∈ {y1,…,yk}– k = # possible
observations
• 𝑃 𝑋1:𝑇 , 𝑌1:𝑇 =
P 𝑥1 P 𝑦1 𝑥1 ∏𝑡=2
𝑇
𝑃 𝑥𝑡 𝑥𝑡−1 𝑃(𝑦𝑡|𝑥𝑡)
https://en.wikipedia.org/wiki/Hidden_Markov_model
Simple Example HMM
• Bob and Alice talk on the phone every day
• Alice cannot see the weather.
• Bob only likes to talk about three activities
• The weather’s markovmodel can be predicted using Alice’s observations. https://en.wikipedia.org/wiki/Hidden_Markov_model
Left to right HMM
• State transitions have the property aij = 0, j < i
– No transtions are allowed to states whose indices are lower than the current state
• Transition can be restricted aij = 0, j > i + △.
Single Word Speech Recognition
• v words to be identified– Each word modeled by a
distinct HMM
• k occurrences of each word spoken by 1 or more talkers.
• λv = HMM for each word in the vocabulary
• O = {O1,O2,…,On)
• P(O| λv), 1 < v < V
• v* = argmax[P(O| λv)]
http://www.cs.ubc.ca/~murphyk/Bayes/rabiner.pdf
Identifying unknown proteins
• Homologous proteins are proteins that share a common ancestry (and likely a similar function).
– Orthologs – proteins that originate from a common ancestor
– Paralogs – proteins that originate from copy events in the same ancestor
• The sequence of known homologous proteins can be used to predict the function of unknown homologous proteins.
Identifying unknown proteins
• Global Alignment– CD-Hit
– uClust
• Local Alignment– Blast
• Problem is that sequence does not directly determine function. Structure does.
http://drive5.com/usearch/manual/uclust_algo.html
Protein Homology Identification
• Protein domains represent functional subunits in a protein
• A single protein can be made up of one or more domains– We can use the
domain architecture to predict remote homology
Protein Homology Identification
• Pfam and other domain databases identify domains of highly conserved sequences
• They then create a series of hundreds of representative sequences in order to create an HMM.
• This HMM can then be used to determine the domains found in an unknown protein.
• Function can be inferred from domain architecture
Example HMM’s
Homework
• Given the transition matrix bellow
– Draw the representative digraph
– Assuming we are in State 1 at t0, what is the probability that we will be in state 2 at t2
– Calculate the probability matrix for t2
– At which t do the values converge so that the state of t0 does not matter
1 2 3
1 0.9 0.075 0.025
2 0.15 0.8 0.05
3 0.5 0.25 0.25
References
1. Hayes, B. (2013). First Links in the Markov Chain. American Scientist, 101(2), 92. 2. Schneider, I. (2005). Jakob Bernoulli, Ars Conjectandi (1713). Landmark Writings in Western Mathematics 1640-
1940 (pp. 88-104). Amsterdam: Elsevier.3. Kinchin, A (1929). Sur la loi des grands nombres. Comptes rendus de l’Academie des Sciences, 189, 477-479.4. Vucinich, A. (1960). Mathematics in Russian culture. Journal of the History of Ideas, 21(2): 161-1795. Senata, E. (2003). Statistical Regularity and free will: L. A. J. Quetelet and P. A. Nekrasov. International Statistical
Review, 71, 319-334.6. Grinstead, M. and Snell, J. (1997). Introduction to Probability. American Mathematical Society.
http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/amsbook.mac.pdf7. Basharin, G., Langville, A., Naumov, V. (2004). The life and work of A.A. Markov. Linear Algebra and its
Applications, 386, 3-26.8. Rabiner, L. (1989). A tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.
Proceedings of the IEEEl, 77(2), 1989.9. Baum, L., Petrie, T. (1966). The Statistical Inference of Probabilistic Functions of Finite State Markov Chains. The
Annals of Mathematical Statistics, 37(6), 1554-1563.10. Finn, R., Bateman, A., Clements, J., et al. (2014). Pfam: the protein families database. Nucleic Acids Research,
42(D1), D222-D230.11. Edgar,RC (2010) Search and clustering orders of magnitude faster than BLAST, Bioinformatics 26(19), 2460-2461.12. "Clustering of highly homologous sequences to reduce the size of large protein database", Weizhong Li, Lukasz
Jaroszewski & Adam Godzik Bioinformatics, (2001) 17:282-283.13. Wheele, T., Clements, J., Finn, R. (2014). Skylign: a tool for creating informative, interactive logos representing
sequence alighments and profile hidden Markov models. BMC Bioinformatics, 15(7), 2014
Questions?
Top Related