Optimization of Task Scheduling Algorithms in Heterogeneous Environment
Hailan Pan · 2017
Workflow system is the best choice for projects with complex business logic, and cloud workflow technology has emerged because many applications need to join large data processing.This paper first introduces the workflow and MapReduce technology briefly, and then proposes to use agent mode to solve the engine centralization problem of traditional cloud workflow architecture.Finally, this paper describes the workflow scheduling model and algorithm based on MapReduce, which makes full use of time fragmentation by using task segmentation to improve efficiency. Related technical analysisWorkflow technology.Workflow technology is the calculation model of workflow, that is, how to express the appropriate model and implement calculation in the computer for workflow' s the logic and rules.The main problem which workflow can solve is that in order to achieve a business goal, using computers transfer documents, information or tasks in a number of participants according to achieve a predetermined rule automatically.MapReduce model based on Hadoop.Hadoop will divided work into a number of small tasks to perform, these small tasks are divided into two categories: map tasks and reduce tasks.Users only need to customize the map and reduce functions, and not worry about complex processing details [2,3].A brief introduction to the MapReduce model process flow is as follows:A) The related data of Mapreduce tasks will be divided into small blocks of data by Hadoop, the block size tends to data size of HDFS block whose default value is 64MB, and then the task map will be assigned to run in the storage of small input fast node , This is known as the implementation of data localization optimization.The input data of map function is key/value pairs, and intermediate data set generated after processing is key/value pairs too.B) Then sorting the intermediate data set, which is mainly merge sort.Followed by the copy process, that is Reduce requests to read each maps' output of the intermediate data.The sort and International Forum on Mechanical, Control and Automation (IFMCA 2016)