File Replication and Location Dynamic Adjustment Algorithm for Telecom Data Warehouse Based on Cloud Platform
Yang Miao-mia · Journal of Nanjing University of Posts and Telecommunications · 2014
Hadoop uses a uniform data replication scheme to ensure data availability and fault tolerance. For an important data block,the number of replicates can only be set by modifying the configuration file. Through the analysis of file access of telecom data warehouse,a file replicate and location dynamic adjustment algorithm,called the BPDA is proposed to improve the data locality,thus it reduces the I / O and network overhead due to data transmission,and enhances the efficiency of query processing. Using multi-copy consistency Hash algorithm,probability sampling and competitive aging algorithm,the optimized number and the storage location are achieved for each file. Experimental results show that BPDA can improve the efficiency of join query processing.