Research of an End-to-End Transfer Mechanism for Big Data in CMAGrid Environment
Limin Liu · Computing Technology and Automation · 2014
In order to improve the using efficiency of existed meteorological resources and the running efficiency of meteorological jobs,we established a Meteorological Grid(CMAGrid)environment with the high performance computing resources of eight regional centers.The input and output of meteorological jobs in the running process need transfer data in the regional centers.The capacity of the meteorological data file is always big,therefore,it is a worthy study topic to improve transmission efficiency of the meteorological data in the CMAGrid environment.This paper focuses on the End-to-End efficient of transmission mechanism of the big meteorological data between the regional centers in the CMAGrid environment. According to the characteristics of meteorological data and he CMAGrid environment,this paper analyzes the key problems that affects the transfer efficiency of the big data and designs an efficient end-to-end multipath parallel data transfer mechanism based on the technology of buffer area,data segmentation and multipath parallel transfer and so on.It aims to fully use the transfer capacity of the entire network and improve the data transmission efficiency in the CMAGrid environment.