Evaluation Study of a Proposed Hadoop for Data Center Networks Incorporating Optical Circuit Switches
Xiaoyu Wang, Malathi Veeraraghavan, Haiying Shen · Journal of Optical Communications and Networking · 2018
Hybrid data center network architectures, which combine electrical packet switches (EPSs) with optical circuit switches (OCSs), have been proposed to handle the inter-rack link oversubscription problem in a cost- and power-efficient manner. However, the high reconfiguration delay of optical switching technologies makes it challenging to effectively leverage OCSs in these hybrid networks. In our prior work, we proposed a modified version of Hadoop, called Hadoop for hybrid networks (HHN), to create network traffic patterns that match the characteristics of OCSs. This paper presents a comprehensive comparative evaluation of HHN on hybrid networks and the original Hadoop on conventional EPS-only networks. Numerical results validate our hypothesis that it is feasible to achieve similar system-level and user-level performance with HHN while simultaneously achieving power and cost savings with the hybrid network. When the percentage of shuffle- heavy (SH) jobs is small, e.g., 5%, HHN performance is the same as that of the original Hadoop on an EPS-only network. When the percentage of SH jobs is large, e.g., 20%, the HHN performance is almost the same even at high loads, and even with a smaller number of input-block replicas when we placed an upper bound on the per-job input-data size. For large SH jobs with input-data sizes larger than the upper bound, the performance of HHN could be improved significantly when storing a larger number of replicas for each input block.