Progress Estimation in Parallel Data Processing Systems
Mareike Höger, Odej Kao · 2016
Parallel data processing frameworks are designed to give the user an easy, convenient way to process his code in parallel. Although giving a feedback of the status of a job is an important part of usability, progress estimation in parallel data processing is an issue that gets little attention. Knowing the estimated overall, remaining runtime will enable the user to monitor the job, thus get indications of failures or problems with the set-up. Furthermore it may be helpful for optimization in the system itself. The job progress or estimated remaining runtime is a key value for improvement of job schedules or resource management. In this paper we introduce progress estimation techniques for parallel data flow engines. We compare a sample run based approach with a new progress propagation technique for pipelined jobs. It is implemented on the parallel data processing framework Nephele, a version of the runtime engine in Apache Flink programming stack. The described progress estimation does not need data or task statistics. We describe the system model, the implementation of the concept. We show that the intuitive sample run based approach brings several issues, that the estimated progress with the progress propagation is is measured quiet accurately for pipelined jobs.