An Index Clustering and Mapping Algorithm for Large Scale Astronomical Data Searching
Xiaokang Liu, Zhihui Du, Yinong Chen, Meie Dai · 2009
For large scale unstructured astronomical data documents, the simple index method often results in high communication cost and slow query processing. Based on the characteristics of domain specific astronomical data and the quantitative tracing and analyzing results, a query terms similarity calculation formula is provided. An index clustering algorithm is designed to generate many small clusters with high term association and small real index size which can be stored into different nodes as a whole. To keep high query locality and reasonable load balancing, a practical index mapping algorithm is proposed to map different logical index clusters onto physical nodes. The simulation results show that the algorithms provided in this paper have good scalability for large scale astronomical data index system. Compared with other methods, different queries can be distributed and located onto smaller number of nodes, so communication cost among different nodes can be reduced significantly and the search efficiency could be well improved.