Grouping and joining transformations in the data extraction process
Marcin Gorawski, Paweł Marks · Studia Iuridica Lublinensia (Uniwersytet Marii Curie-Skłodowskiej w Lublinie) · 2006
Abstract. In this paper we present a method of describing ETL processes (Extraction, Transformation and Loading) using graphs. We focus on implementation aspects such as division of a whole process into threads, communication and data exchange between threads, deadlock prevention. Methods of processing of large data sets using insufficient memory resources are also presented upon examples of joining and grouping nodes. Our solution is compared to the efficiency of the OS-level virtual memory in a few tests. Their results are presented and discussed. 1