Determining Stopping Criteria in the Generation of Web-Derived Langua ge Models
Gary A. Monroe, David R. Mikesell, James C. French · 2000
In this work, we present a small-scale evaluation of two query-based sampling techniques for building language models, using a database comprised of world-wide web documents. We propose a metric by which it is possible to determine when to cease sampling a given web database, and we compare this new metric to other metrics that have been used in previous work to determine the fidelity of sampled language models.