Data-oriented Processing in UNICORE
Bernd Schuller, Richard Grunzke, André Giesler · JuSER (Forschungszentrum Jülich) · 2013
In order to support the ongoing shift from traditional batch-job oriented computing towards a stronger focus on efficient processing of "big" scientific data sets, we plan to extend the UNICORE middleware with a new data-oriented processing feature. This feature allows to associate processing rules with a storage, where data processing is triggered automatically without explicit user activity. Compared to the similarly rule-driven data management system iRods, the rules are completely controlled by the end user, and all computations are done on account of the user by virtue of UNICORE’s security system. Furthermore, the system seamlessly integrates with the existing backend storage. This paper describes the initial design, current implementation status and some first results. The implementation is still in progress, with the first release scheduled for late Autumn 2013 as part of the UNICORE 7 major release.