Choosing LSI Dimensions by Document Linear Association Analysis.
Chenggen Shi, Jie Lü · UTS ePRESS (University of Technology Sydney) · 2003
Latent Semantic Indexing (LSI) has proven to be a valuable analysis tool with a wide range of applications, however the crucial question, choosing an appropriate number of dimensions for LSI, is still unsolved. In this paper, a new method which is to deal with this problem is described. It finds that a sum of total dot products between all document vectors reaches the maximum value at a specific number of dimensions for a given dataset. With this reduced dimensions LSI achieves the best performance. The performance evaluations have demonstrated that this method can choose an appropriate number of dimensions for LSI and effective detect the data structure for a dataset.