A Simple Approach for Data Cleansing on Hadoop Framework using File Merging Technique
Adnan Ali, Nada Masood Mirza, Rawad Bader, Mohamad Khairi Ishak · 2022
Hadoop framework is known for being top-notch in processing these huge files and providing useful data. Unfortunately, in a scenario with many small files, the framework is inefficient and fails to deliver. These small files cause many issues when the framework's processing criteria and performance levels. Moreover, these small files contain content that is useless or provides no benefit in the key-value decision-making. To overcome this issue of small files and unnecessary content, this paper proposes a simple data cleansing and file merging approach based on specific type and size that will not only be effective but will increase the framework's performance by approx. 68%. This algorithm ensures the output will be a few huge files with essential/important data. The results show that the proposed system not only improves the framework's performance but also reduces deadlocks in the framework processes, which is approximately 68 % improvement over the base Hadoop framework processing.