HPC & Co strike back: Where are distributed paradigms heading toward?
Sébastien Limet, Alessio Merlo, Luca Spalazzi · Concurrency and Computation Practice and Experience · 2018
From the dawn of parallelization, the diffusion of High-Performance Computing (HPC) has grown exponentially until nowadays, thereby leading to the birth and development of a huge research community in this field.During the last decades, the important research results have led to extend the domain of HPC into three different dimensions, namely Architectures, Research Topics, and Application Domains. ArchitecturesTraditional HPC architectures have been sided with several wide-area distributed paradigms such as Peer-to-Peer (P2P), Grid, Cloud, Fog and Pervasive Computing.[1][2][3] Each of them aims new specific objectives along with the original goal of improving performance and reliability.In this paper, the different declinations of the traditional HPC are grouped into three macro-categories, namely SIMD, Multi- * , and Cluster.• SIMD.It stands for "single instruction, multiple data" according to the Flynn taxonomy.4 This refers to computing systems where a single instruction stream operates on multiple data streams in order to perform operations that may be naturally parallelized (eg, array processors, GPUs, GPGPUs, etc).This category also includes many specialized processing units, as vector extensions of modern processors like Streaming SIMD Extensions (SSE) 5 or Advanced Vector Extensions (AVX). 6• Multi- * .It refers to many-core, multi-core, and multi-processor architectures.They are all architectures characterized by the presence of multiple instruction streams operating on multiple data streams.What distinguishes this type of architectures from other MIMD architectures ("multiple instructions, multiple data" always according to the Flynn taxonomy 4 ) is the sharing of the central memory.• Cluster.It refers to Cluster Computing, namely a group of cost-effective linked commercial off-the-shelf computers working together so closely that they act as a single machine.What distinguishes this type of architecture from other MIMD-like architectures is the absence of central memory sharing (each processing unit has its own private memory) but the presence of mass memory sharing.As a matter of fact, the current Top500 supercomputers 7 are mostly clusters.Regarding wide-area distributed paradigms, all of them are included in the MIMD category (according to the Flynn's taxonomy).What distinguishes this type of architecture from other MIMD architectures is the lack of sharing of both the central memory and the mass memory.This paper takes into consideration the main four distributed paradigms that arose from traditional HPC, namely Grid Computing, Ubiquitous Computing, Peer-to-Peer Computing, Cloud, and the emerging Fog Computing.In brief:• Grid Computing 8-10 denotes a distributed architecture that enables coordinated resource sharing for problem-solving within dynamic organizations consisting of individuals, institutions, and resources.• Ubiquitous Computing (also known as Pervasive Computing or UbiComp) 11,12 refers to making available to users computational resources, which provide information and services, anytime and everywhere throughout the physical environment.• Peer-to-peer (P2P) 13 is a distributed paradigm made by a network of nodes, where each node has the same capabilities of other nodes; the computation is carried out by the interactions between a node and its neighbors.P2P networks have been originally used for sharing contents.• Cloud and Fog Computing 14,15 refers to a computing model that enables ubiquitous and on-demand access to a shared set of resources spread over the Internet and delivered as services.Such resources are quickly provisioned and released at an infrastructure, platform, or application level.Cloud Computing 15 is often seen as an evolution of Grid Computing where the best-effort access to resources is substituted by a rapid provision of customized resources requiring a minimal management effort.Indeed, both Grid and Cloud computing trace their roots back to the notion of Utility Computing, ie, computing being delivered as a public utility.This notion was introduced by John McCarthy in a speech at the MIT Centennial in 1961.16 As in the Cloud, the computation is executed on remote machines; there could be non-negligible latency and security issues.This