Discrete-Time, Finite-State Markov Chains
James R. Kirkwood · 2015
In this chapter, our setting is we have a system that consists of a nite numberofstatess1, … , sn and a process that is in exactly one of these states at any giventime.Foradiscrete-timeMarkovchain,onecanvisualizetheprocess runningaccordingtoadigitalclockandtheprocessmakingthedecision ofhowtochangestatesateachchangeintheclock.Inmakingthechange ofstates,theprocessisgovernedbyasetofconditionalprobabilities,called transition probabilities. These are dened by P ni j, ( ) = the probability that the process changes from state si to sj when the time changes from n to n + 1 P ni i, ( ) = the probability that the process remains in state si when the time changes from n to n + 1.