Efficient Geo-distributed Data Processing with Rout
Chamikara Jayalath, Patrick Th. Eugster · 2013
Big data processing undoubtedly represents a major challenge of this era. While several programming models and supporting systems have been proposed to deal with such data in so-called “cloud” infrastructures, they all exhibit the same limitation: all data is assumed to be located in one datacenter. This limitation results from cloud vendors promoting the abstraction of omnipresent computing and storage resources. When dealing with data distributed across datacenters, programmers currently have two options: (1) copying all data to a single datacenter easily becomes tedious if done manually as the original dataset is updated, leads to repetitive copying if performed as part of a program, and is sometimes impossible; (2) writing multiple variants of the same program, with consolidation occurring at different points varying by characteristics of the task (e.g., input sub-dataset sizes) is laborious and does not help determining the most appropriate one for a given run. This paper introduces geo-distributed data structures and operations for expressing data processing tasks taking place across datacenters. We describe the design and implementation of such data structures and operations for the PigLatin language. We illustrate the performance benefits of our geodistributed data structures and operations through several benchmarks, showing up to 2× faster response times.