Statistical inference using weak chaos and infinite memory
Max Welling, Yutian Chen · Journal of Physics Conference Series · 2010
We describe a class of deterministic weakly chaotic dynamical systems with infinite memory. These "herding systems" combine learning and inference into one algorithm, where moments or data-items are converted directly into an arbitrarily long sequence of pseudo-samples. This sequence has infinite range correlations and as such is highly structured. We show that its information content, as measured by sub-extensive entropy, can grow as fast as K log T , which is faster than the usual ½ K log T for exchangeable sequences generated by random posterior sampling from a Bayesian model. In one dimension we prove that herding sequences are equivalent to Sturmian sequences which have complexity exactly log( T + 1). More generally, we advocate the application of the rich theoretical framework around nonlinear dynamical systems, chaos theory and fractal geometry to statistical learning.