A Distributed, Asynchronous Algorithm for Large-Scale Internet Network Topology Analysis
Youssef Elmougy, Akihiro Hayashi, Vivek Sarkar · 2024
With the growing complexity of modern internet networks, we introduce a distributed, asynchronous, and scalable algorithm tailored for determining the centrality and importance of various levels of the global internet network topology at a massive scale. By utilizing triangle formations and neighborhood densities in these complex networks, our algorithm provides valuable insights into the structural importance of individual nodes and the intricate relationships among interconnected entities. Focusing on the global internet's prime components — routers, IP addresses, and Autonomous Systems — the algorithm provides a scalable solution extending to the global internet infrastructure, datacenters, cloud networks, and private networks. By taking advantage of the parallelism, distributed processing, efficient communication, and fine-grained asynchronous execution through an Actor-based programming system, our algorithm is capable of efficiently processing and performing rapid computations on large-scale internet networks. We perform scalability studies on the NERSC Perlmutter supercomputer and the Georgia Tech HPC PACE cluster using three large-scale real-world network datasets and one large-scale synthetic network dataset, achieving up to 91.7% parallel efficiency scaling out to 2K cores and reducing execution time from 5.7 hours to 18.3 seconds, while performing 105.4× better on average compared to related approaches. Our research contributes significantly to facilitating enhanced network management, fault tolerance and resilience, and security, but as well to improving overall network stability, reducing latency, and optimizing energy consumption across the vast internet network devices environment.