Design of Big Data Algorithm Based on MapReduce
Liangfu Luo · Proceedings of the 2020 International Conference on Aviation Safety and Information Technology · 2020
With the widespread application of Internet technology, the utilization rate of similar blogs and social networks has been significantly improved. With the intervention of cloud computing and other technologies, a large amount of data generated during browsing the Internet can be effectively accumulated and processed, which indicates that big data has been integrated into people's daily life. MapReduce, as a parallel programming environment, can effectively deal with big data related problems, and has been applied in large Internet companies such as Google and Amazon Maxdiff, an efficient histogram algorithm based on MapReduce, includes accurate algorithm and approximate algorithm, and proposes two-table equijoin algorithm and multistandard equijoin algorithm for data skew And optimize the efficiency of the connection algorithm when one or more data in the data set appear too much. This paper starts with the efficiency optimization of big data connection algorithm based on MapReduce, and studies the efficiency optimization of equijoin algorithm, data skew connection algorithm and connection algorithm in detail, and then puts forward an algorithm that can effectively improve the program execution efficiency, hoping to provide reference for the follow-up research work.