Greener Big Data: Optimizing Data Exchange and Power Procurement for Data Centers
Zakia Asad · TSpace (University of Toronto) · 2017
Big data is revolutionizing modern day life at an environmental cost due to the “ungreen” practices associated with the operation of data centers, a workhorse for big data processing. Turning the big data enterprise into a greener enterprise requires two fundamental steps. The first step is curtailing resource consumption, and making data centers energy efficient. The second step is using greener energy sources to power data centers. Big data processing has challenged the ability of data centers to operate in a greener fashion. In particular, the rise of the cloud and distributed data-intensive (“big data”) applications puts pressure on data center networks due to the movement of massive volumes of data. Reducing the volume of communication is crucial for embracing greener data exchange by efficient utilization of the resources. This thesis proposes the use of a mixing technique, spate coding, as a means of dynamically-controlled reduction in volume of communication. We introduce real-world use-cases, and present a novel spate coding algorithm for the data center networks. We also analyze the computational complexity of minimizing the volume of communication in a distributed data center application, and provide theoretical limits. Moreover, we proceed to bridge the gap between theory and practice by performing a proof-of-concept implementation of the proposed system in a real-world data center. We use MapReduce, the most widely used Big Data processing framework, as our target. Experimental results employing industry standard benchmarks show the advantage of our proposed system compared to the current state-of-the-art. Another important step in promoting green practices for the big data enterprise is reshaping the energy mix required to operate data centers. In this context, we consider power procurement for data centers while incorporating the green initiatives and cyber-physical constraints imposed by evolving power systems. Moreover, we consider implications of our formulation to solution complexity. We use the concept of matroid theory to model the problem as a combinatorial problem and propose a distributed and optimal solution. We account for communications and computational complexity and provide solutions for practical scenarios. In the end, we propose a futuristic cross-plane green orchestrator.