Transparent and Flexible Network Management for Big Data Processing in the Cloud
Anupam Das, Cristian Lumezanu, Yueping Zhang, Vishal Kumar Singh, Guofei Jiang, Curtis Yu · USENIX conference on Hot Topics in Cloud Ccomputing · 2013
We introduce FlowComb, a network management framework that helps Big Data processing applications, such as Hadoop, achieve high utilization and low data processing times. FlowComb predicts application network transfers, sometimes before they start, by using software agents installed on application servers and while remaining completely transparent to the application. A centralized decision engine collects data movement information from agents and schedules upcoming flows on paths such that the network does not become congested. Results on our lab testbed show that FlowComb is able to reduce the time to sort 10GB of randomly generated data by 35% while changing paths for only 6% of the transfers.