On the finite-length performance of universal coding for k-ary memoryless sources
Ahmad Beirami, Faramarz Fekri · 2010
In this paper, we investigate the performance of universal coding schemes on finite-length memoryless sequences. Rissanen demonstrated that for the universal compression of k-ary memoryless sources, expected redundancy for regular codes is asymptotically lower bounded by (k-1)/2 log n for almost all sources. Xie and Barron derived the minimax expected redundancy for k-ary memoryless sources, which characterizes the maximum redundancy over all possible source parameters. It does not provide much information about different source parameter values. This paper is a finite-length extension to Rissanen's result. Our treatment in this paper is probabilistic. In particular, we derive a lower bound on the probability measure of the sources that are not compressible with a redundancy smaller than a certain fraction of (k-1)/2 log n. In other words, we demonstrate a lower bound on the redundancy for a given percentile of sources. We demonstrate that as the length of the memoryless sequence decreases, the redundancy tends to become significant and comparable to the entropy of the sequence.