Editing Markov chain (section)

====Time-homogeneous Markov chain with a finite state space====

If the Markov chain is time-homogeneous, then the transition matrix '''P''' is the same after each step, so the ''k''-step transition probability can be computed as the ''k''-th power of the transition matrix, '''P'''<sup>''k''</sup>.

If the Markov chain is irreducible and aperiodic, then there is a unique stationary distribution {{pi}}.<ref name="auto">{{Cite book |last=Serfozo |first=Richard |date=2009 |title=Basics of Applied Stochastic Processes |series=Probability and Its Applications |doi=10.1007/978-3-540-89332-5 |isbn=978-3-540-89331-8 |place=Berlin |publisher=Springer }}</ref> Additionally, in this case '''P'''<sup>''k''</sup> converges to a rank-one matrix in which each row is the stationary distribution {{pi}}:
:<math>\lim_{k\to\infty}\mathbf{P}^k=\mathbf{1}\pi</math>
where '''1''' is the column vector with all entries equal to 1. This is stated by the [[Perron–Frobenius theorem]]. If, by whatever means, <math display="inline">\lim_{k\to\infty}\mathbf{P}^k</math> is found, then the stationary distribution of the Markov chain in question can be easily determined for any starting distribution, as will be explained below.

For some stochastic matrices '''P''', the limit <math display="inline">\lim_{k\to\infty}\mathbf{P}^k</math> does not exist while the stationary distribution does, as shown by this example:
:<math>\mathbf P=\begin{pmatrix} 0& 1\\ 1& 0 \end{pmatrix} \qquad \mathbf P^{2k}=I \qquad \mathbf P^{2k+1}=\mathbf P</math>
:<math>\begin{pmatrix}\frac{1}{2}&\frac{1}{2}\end{pmatrix}\begin{pmatrix} 0& 1\\ 1& 0 \end{pmatrix}=\begin{pmatrix}\frac{1}{2}&\frac{1}{2}\end{pmatrix}</math>
(This example illustrates a periodic Markov chain.)

Because there are a number of different special cases to consider, the process of finding this limit if it exists can be a lengthy task. However, there are many techniques that can assist in finding this limit. Let '''P''' be an ''n''×''n'' matrix, and define <math display="inline">\mathbf{Q} = \lim_{k\to\infty}\mathbf{P}^k.</math>

It is always true that
:<math>\mathbf{QP} = \mathbf{Q}.</math>

Subtracting '''Q''' from both sides and factoring then yields
:<math>\mathbf{Q}(\mathbf{P} - \mathbf{I}_{n}) = \mathbf{0}_{n,n} ,</math>
where '''I'''<sub>''n''</sub> is the [[identity matrix]] of size ''n'', and '''0'''<sub>''n'',''n''</sub> is the [[zero matrix]] of size ''n''×''n''. Multiplying together stochastic matrices always yields another stochastic matrix, so '''Q''' must be a [[stochastic matrix]] (see the definition above). It is sometimes sufficient to use the matrix equation above and the fact that '''Q''' is a stochastic matrix to solve for '''Q'''. Including the fact that the sum of each the rows in '''P''' is 1, there are ''n+1'' equations for determining ''n'' unknowns, so it is computationally easier if on the one hand one selects one row in '''Q''' and substitutes each of its elements by one, and on the other one substitutes the corresponding element (the one in the same column) in the vector '''0''', and next left-multiplies this latter vector by the inverse of transformed former matrix to find '''Q'''.

Here is one method for doing so: first, define the function ''f''('''A''') to return the matrix '''A''' with its right-most column replaced with all 1's. If [''f''('''P''' − '''I'''<sub>''n''</sub>)]<sup>−1</sup> exists then<ref>{{cite web|url=https://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/Chapter11.pdf|title=Chapter 11 "Markov Chains"|access-date=2017-06-02}}</ref><ref name="auto"/>

:<math>\mathbf{Q}=f(\mathbf{0}_{n,n})[f(\mathbf{P}-\mathbf{I}_n)]^{-1}.</math>
:Explain: The original matrix equation is equivalent to a [[system of linear equations|system of n×n linear equations]] in n×n variables. And there are n more linear equations from the fact that Q is a right [[stochastic matrix]] whose each row sums to 1. So it needs any n×n independent linear equations of the (n×n+n) equations to solve for the n×n variables. In this example, the n equations from "Q multiplied by the right-most column of (P-In)" have been replaced by the n stochastic ones.
One thing to notice is that if '''P''' has an element '''P'''<sub>''i'',''i''</sub> on its main diagonal that is equal to 1 and the ''i''th row or column is otherwise filled with 0's, then that row or column will remain unchanged in all of the subsequent powers '''P'''<sup>''k''</sup>. Hence, the ''i''th row or column of '''Q''' will have the 1 and the 0's in the same positions as in '''P'''.