Managing the Google Web 1T 5-gram data set
Aminul Islam, Diana Zaiu Inkpen · 2009
This paper describes how the Google Web 1T 5-gram data set, contributed by Google Inc., can be stored so that it can be used efficiently with respect to time. We present an efficient way of accessing all the 5-grams for a specific word of interest from the stored files. We measure the maximum access and processing efficiency achievable for any word of interest. We also compare results (access time and memory requirements) on the task of accessing all the 5-grams for a list of words, on both the processed and the original organization of the data set.