Abstract: PanDA: Next Generation Workload Management and Analysis System for Big Data
Alexei A. Klimentov, A. Vaniachine, K. De, Torre J. Wenaus, S. Panitkin, D. Yu, Gergely V. Zaruba, Mikhail Titov · 2012
In real world any big science project implies to use a sophisticated Workload Management System (WMS) that deals with a huge amount of highly distributed data, which is often accessed by large collaborations. The Production and Distributed Analysis System (PanDA) is a high-performance WMS that is aimed to meet production and analysis requirements for a data-driven workload management system capable of operating at the Large Hadron Collider data processing scale. PanDA provides execution environments for a wide range of experimental applications, automates centralized data production and processing, enables analysis activity of physics groups, supports custom workflow of individual physicists, provides a unified view of distributed worldwide resources, presents status and history of workflow through an integrated monitoring system, archives and curates all workflow. PanDA is now being generalized and packaged, as a WMS already proven at extreme scales, for the wider use of the Big Data community.