Efficient Data Processing Techniques for Structured Data Analysis Using Stream Pipeline Parallelism
Sampath Sampath, D. K. Sreekantha · Fusion Practice and Applications · 2024
This research illustrates how dynamic task balancing and data sharing may improve distributed data processing. The technology handles parallel processing system difficulties with huge datasets by minimizing resource utilization, time complexity, and output. We modify the workload on the fly after splitting to ensure that all processing units receive equal work. One last optimization phase optimizes job distribution to maximize system efficiency. We test the solution for latency, speed, scalability, resource utilization, fault tolerance, and synchronization overhead. Results reveal that the new strategy outperforms existing ones in every regard. It features the lowest latency, quickest production, and highest growth potential. The approach handles mistakes well, divides data effectively, and syncs everything at a cheap cost. These properties make it ideal for real-time data processing and fast-growing applications. Future study will concentrate on flexible splitting strategies, fault tolerance mechanisms, and predictive analytics machine learning models. These modifications will improve real-time data handling.