Entropy Rates of a Stochastic Process
Thomas M. Cover, Joy A. Thomas · 2001
The asymptotic equipartition property in Chapter 3 establishes that nH(X) bits suffice on the average to describe n independent and identically distributed random variables. But what if the random variables are dependent? In particular, what if the random variables form a stationary process? We will show, just as in the i.i.d. case, that the entropy H(X1,X2,…,Xn) grows (asymptotically) linearly with n at a rate H(𝒳), which we will call the entropy rate of the process. The interpretation of H(𝒳) as the best achievable data compression given in Chapter 5.