Software development framework for a distributed storage and GPGPU data processing infrastructure
I. S. Kamenskikh, Dmitry M. Sinelnikov, D. S. Kalintsev, A. A. Kozlov, Mikhail M. Rovnyagin, D. A. Shulga · 2016
The problem of choosing the cluster or a cluster node for task execution is important for the overall performance of a distributed system. This paper presents a complex approach to the planning of computations on heterogeneous distributed systems - a set of clusters and NoSQL storage systems. Dynamic scheduling algorithm depends on: the inter-cluster network parameters, characteristics of cluster interconnect, compute nodes utilization, co-processors computing capabilities, etc. In this work Hadoop YARN, CUDA technology and NoSQL-system Apache Cassandra has been used as the experimental platform.