Ditto: Efficient Serverless Analytics with Elastic Parallelism
Chao Jin, Zili Zhang, Xingyu Xiang, Songyun Zou, Gang L. Huang, Xuanzhe Liu, Xin Jin · 2023
Serverless computing provides fine-grained resource elasticity for data analytics---a job can flexibly scale its resources for each stage, instead of sticking to a fixed pool of resources throughout its lifetime. Due to different data dependencies and different shuffling overheads caused by intra- and inter-server communication, the best degree of parallelism (DoP) for each stage varies based on runtime conditions.