Scalable Data Integration System using Representational State Transfer
Pornthip Mensin, Phongphun Kijsanayothin, Worajit Setthapun · Walailak Journal of Science and Technology (WJST) · 2016
Data integration from remote data sources is complicated, due to a large number of heterogeneous information sources that are usually stored in different data schema. This topic is challenging and requires the management of information system enterprises to be able to integrate distributed data and applications effectively. Numerous methodologies and systems have been proposed and developed to address the related issues from different aspects. However, there is still a lack of methodological support and appropriate data integration development in an internet environment that requires good performance and scalability on large-scale data execution. The concept of centralized design for the integration of distributed data repositories as server-based processing solves the traditional problem of tight coupling, but does not address the integrated logic task. To solve this problem, this paper describes a data Integrated Client-Rich approach (ICR) and implementation of Thesis Core System (TCS) data integration system that follows the Representational State Transfer (REST) architectural principle for system implementation. REST is a new principle for system development that has good support for development tools to meet the needs of data integration, and obtain the advantages of simplicity, scalability, and performance. We design a testing scheme to evaluate and analyze the scalability of traditional data integration with server-based integrated processing and TCS data integration. The results show that our approach is more suitable, scalable, and practical for Internet-scale distributed data integration.