Some Improved Sample Complexity Bounds in the Probabilistic PAC Learning Model
un-ichi Takeuchi · 2005
1 I n t r o d u c t i o n Since Valiant's introduction of the PAC learning model [6] for boolean functions, several extensions of the model to the learning of probability distributions were made. Yamanishi [7] and Kearns and Schapire [3] considered the problem of learning stochastic rules (or probabillstic concepts), which is the problem of learning conditional distributions. Abe and Warmuth [1] looked at the sample as well as computational complexity of learning probability distributions using probabilistic automata as hypotheses. Yamanishi called his learning model the 'stochastic PAC learning model'. Here we use the term the ~probabilistic PAC learning model' to refer to these models collectively. In this paper, we concentrate on the sample complexity aspect of the learning problem for stochastic rules. In particular, we improve upon the best known upper bounds on the sample complexity for learning the important class of stochastic rules called 'the stochastic rules with finite partitioning' (c.f. Yamanishi [7]), with respect to the classic notion of distance among distributions, the KutIback-Leiblsr divergence (KL-divergence, for short). Yamanishi proved an upper bound on the sample complexity of learning stochastic rules with finite partitioning, which was linear in ~ using the property of the MDL estimator. But this result is with respect to a relatively loose notion of distance called the Hellinger distance. 1 With respect to the KullbackLeibler divergence, a similar upper bound had not been obtained. The best known bound to date was of order O(~) due to Abe, Takeuchi, and Warmuth [2]. The quadratic dependence on ~ was due to the fact that Hoeffding's inequality was used to bound the estimation error for lit is well-known that the Kullback-Leibler divergence between two distributions (or stochastic rules) bounds from above the Hellinger distance between the two but the converse does not necessarily hold, even allowing polynomial blow-ups.