Data loading and mapping using staging DBMS in the grid
Ejaz Ahmed, Nik Bessis, Yong Yue, Des Stephens · Conference proceedings - Canadian Conference on Electrical and Computer Engineering · 2008
With the proliferation of international standards for grid-enabled databases, the need for data loading and data mapping in a large integrated environment of heterogeneous databases highlights issues of consistency and integrity. We discuss methods for providing semi-autonomous data integration by focusing on efficient data loading design and effective mapping strategies. In order to upload and integrate data from various small to large size data repositories, an intermediate staging facility is employed to temporarily store data before it is validated to ensure accurate and useful integration into the target data source. We propose a mechanism that semi-automates the integration process that includes not only new data, but legacy data as well. We expand the notation of a database management system (DBMS) to include the management of the data transfer or data transformation processes. The DBMS now must perform the task of data mapping in the form of value correspondences by using a staging schema or staging DBMS mapping procedure.