Deriving and Managing Data Products in an Environmental Observation and Forecasting System.
Laura F. Bright, David Maier · 2005
Large-scale scientiflc work∞ows can perform many computationally intensive tasks and generate large volumes of derived data products. These systems pose many challenges to both creating and managing data products, includinge‐cientlyexecutingtasksandtracking data product lineage and metadata. In thispaperwedescribeourexperiencesimplementing an experimental data-product managementsystemtoaddressthesechallengesfor the CORIE Environmental Observation and Forecasting System. We present a novel architecture to store both data products and the tasks that create them. Our system in addition supports tasks to automatically perform system maintenance, and enables dataintensivetaskstoexecuteonmultiplenodesof a Grid. We present several challenges to executingexistingscientiflcwork∞owsonaGrid, and propose several techniques to improve task scheduling in this environment. Preliminary performance results show the potential beneflts of these techniques.