Optimization of Network Interaction on a High-Performance Cluster Using Graph Scheduling
Ivan G. Timokhin, Denis Shaikhislamov, Alexey Teplov · 2024
Data access time become large problem for data processing in distributed environments with a growthof system scale and data size and source of software optimization. In this paper, we research the problemof optimizing I/O and processing operations for a distributed Hadoop cluster that process data in HPCparadigm framework. To increase the efficiency of the framework that processes queries in HDFS, agraph algorithm were developed with respect to optimal use of resources using HDFS data locality ona processing graph. The graph uses data file blocks and replicas, hosts and workers as a nodes, andlinks between them as edges. The results of reading stage optimization phase was implemented andperformance improvements measured comparing baseline and Spark as Industry standard framework.