DIRAC data management: consistency, integrity and coherence of data
M. Bargiotti, Andrew Cameron Smith · Journal of Physics Conference Series · 2008
The DIRAC Data Management System (DMS) relies on both WLCG (Worldwide LHC Computing Grid Project) Data Management services (LCG File Catalogues, Storage Resource Managers and File Transfer System (FTS)) and LHCb specific components (Bookkeeping database). The complexity of both the DMS and its interactions with numerous WLCG components as well as the instability of facilities concerned, has frequently turned into unexpected problems in data moving and/or data registration, preventing a coherent picture of datasets. Several developments in LHCb have been made in order to avoid data corruptions, missing data incoherence and inconsistences among Catalogues and physical storage both through safety measures at data management level (failover mechanism, check sums, roll back mechanism) and extensive background checks. In this paper all the tools developed for checking data integrity and consistency will be presented as well as a Storage Usage agent, whose aim is to produce an up-to-date accounting of all LHCb storage usage using the LFC mirror database. The goal of this activity is the development of a generic tool suite able to categorize, analyze and systematically cure the disparate problems affecting the DMS in order to maintain a consistent picture of the main catalogues (Bookkeeping and LFC) and the Storage Elements.