Development Method of MapReduce Oriented Data Flow Processing
Dan Ye · Jisuanji kexue yu tansuo · 2011
In the age of information explosion,DataFlow processing widely existed and has shown new features and styles including massive and parallel,meanwhile more and more people choose to use MapReduce to process their data because of its simplicity and higher capability with lower cost,but MapReduce does not directly support complex N-step,N-branch and multiple data sets data flow processing. This paper proposes a model-driven development method for DataFlow processing based on MapReduce. It first defines the logical and physical models of the dataflow as well as the component model,then designs model transfer and code generation algorithms,finally uses the algorithms to generate the MapReduce program code which implements the function defined by the logical model and can run on Hadoop platform. Based on this method,a development tool CloudDataFlow is implemented. As the experiment shows,compared with similar system,it has higher performance,extendibility and usability.