Optimizing Random Forest Algorithms for LargeScale Data Analysis
Shuchi Juyal Bhadula, Myasar Mundher Adnan, Rakesh Kumar, Ajay Rana, Gopal Kaliyaperumal, Bolleddu Devananda Rao, Nandini Shirish Boob · 2024
The exponential growth of data in recent years has necessitated the development of more efficient and scalable machine learning algorithms to handle large-scale data analysis. Random Forest (RF) algorithms, known for their robustness and accuracy, find extensive use in the realms of classification and regression. However, their performance can be significantly hindered when dealing with massive datasets due to increased computational complexity and resource demands. This paper presents a comprehensive approach to optimizing Random Forest algorithms for large-scale data analysis. We explore various strategies, including data partitioning, algorithmic changes and parallel processing to make RF algorithms more efficient and scalable. Additionally, we introduce novel techniques for feature selection and hyperparameter tuning to improve model performance. Extensive experiments conducted on multiple large-scale datasets demonstrate the effectiveness of the proposed optimizations in reducing computation time and memory usage while maintaining or even enhancing predictive accuracy. From genomics to finance, the optimised Random Forest method shows promise as a robust tool for large-scale data processing in many applications, according to the findings.