Structuring Overlays for Distributed Computing: from Edge-Devices to the Clouds
Erwan Le Merrer · HAL (Le Centre pour la Communication Scientifique Directe) · 2016
Distributed systems are now powering most of successful andworld scale computing applications, ranging from storage backends forsocial networks, to machine learning tools for contentpersonalization, and to global consensus for new crypto-currencies.The key challenge for those systems is to structure the collaborationof the potentially millions of dedicated computers, in order toefficiently and reliably implement services.In this document, I report on some contributions to the design ofoverlays for large scale specific applications. Overlays formthe logical interconnections between computers, and thus encode thecollaborative aspect of the application implementation. I firstdescribe two very different types of overlays in the context ofapplications implemented on top of user-devices: one overlay leveragesthe symmetry of a distributed hash table, to ensure the lack of asingle point of trust (for a secure group communicationapplication). The second overlay is voluntarily skewed in itsstructure, for representing the variety of health statuses of theparticipating nodes. I then move to a cloud environment - where the core logicis also distributed in nature - to present the design of an overlaytailored for the aggregation of incoming data flows. I finallypresent a graph algorithm, called the second order centrality, that isexecuted on overlays in order to provide criticality scores toparticipating computers. This criticality can constitute an indicator ofthe overlay connectivity drift.Along this report, I comment on the general shift of the applicationenvironment of overlays from wide-area networks to clouds, andfinish by re-opening to trending topics back in those public networks.