Leveraging In-Memory Computing for Speeding up Apache Spark and Hadoop Distributed Data Processing
Sarbaree Mishra, Vineela Komandla, Srikanth Bandi · International Journal of Emerging Research in Engineering and Technology · 2022
In-memory computing has been a leading approach to distributed data processing, which in turn has positively affected frameworks like Apache Spark and Hadoop by implementing new features that can overcome limitations of earlier disk-based methods. Most of the traditional disk-based methods, although reliable, have some issues, such as long delays caused by disk I/O bottlenecks, especially when it comes to increasingly large and complex information that needs to be processed. In-memory computing eliminates the inefficiencies by utilizing the computer's random access memory (RAM) for data storage and processing, which results in much lower latency & faster computations. Apache Spark utilizes this idea via its Resilient Distributed Dataset (RDD) model, which stores data temporarily in memory to facilitate repeated tasks and reduce the number of disk operations needed. Likewise, to boost the performance, Hadoop has changed by adding in-memory features like YARN’s memory-based caching. Such an approach is vital in tasks that need input of continuous and quick data, performing analytics in real-time or carrying out repetitive machine learning procedures frequently. Besides quicker execution time, in-memory computing also increases scalability and improves resource utilization by providing more efficient partitioning, caching, and task execution. Furthermore, this also goes hand in hand with the progress of the technology in the field of hardware, like fast memory (RAM) and solid-state drives, which enables even better performance results. Along with optimized data partitioning, compression & fast memory management strategies are the means to alleviate the pressure on resources, allowing systems to operate with low latency/fast response time/high throughput even on bigger datasets. This integration eliminates the overhead involved in the processing, and hence, the organizations become more agile in decision-making because their insights are current and they can respond more quickly