Industrializing Data Integration Projects using a Metadata Driven Assembly Line
Albert Maier, Martin Oberhofer, Thomas Schwarz · it - Information Technology · 2012
Abstract Data integration is essential for the success of many enterprise business initiatives, but also a very significant contributor to the costs and risks of the IT projects supporting these initiatives. Highly skilled consultants and data stewards re-design the usage of data in business processes, define the target landscape and its data models, and map the current information landscape into the target landscape. Still, the largest part of a typical data integration effort is dedicated to the implementation of transformation, cleansing, and data validation logic in robust and highly performing commercial systems. This effort is simple and doesn´t demand skills beyond commercial product knowledge, but it is very labour-intensive and error prone. In this paper we describe a new commercial approach to data integration that helps to “industrialize” data integration projects and significantly lowers the amount of simple, but labour-intensive work. The key idea is that the target landscape for a data integration project has pre-defined data models and associated meta data which can be leveraged for building and automating the data integration process. This approach has been implemented in the context of the support of SAP consolidation projects and is used in some of the largest data integration projects world-wide.