Characterizing Networking as Experienced by Users (White Paper)
I. Sfiligoi · TigerPrints (Clemson University) · 2020
The progress in wide area network throughput has been a boon for data intensive applications, giving them the option of bursting past the confines of local institutions and allowing them to make full use of capabilities of distributed compute resources.All larger scientific endeavors these days make heavy use of distributed computing, often reading a large fraction of their data directly over WAN link, with notable examples being the LHC experiments, LIGO and IceCube.This was not always the case, and some early distributed computing pioneers, e.g. the LCG community, had to carefully plan and execute data placement to be close to available resources, which in turn required careful planning and scheduling of network data transfers, which was both labor intensive and error prone.This benefit to research is by now widely recognized and has led to several activities that are aimed at maximizing the benefits to the affected communities.One of the most important concepts in this area was the wide adoption of the Science DMZ concept and the proliferation of data transfer nodes (DTN), which allow for both high throughputs in and out of those zones, and for reliable characterization of single network links.The Pacific Research Platform (PRP), and the follow-up TNRP grant, have extended this concept by adding the ability to easily deploy and schedule services on top of nodes in Science DMZs, thus making network measurement both simple to execute and easy to reliably schedule.These activities have thus led to network links between DTNs to be highly performant most of the time, and drastically reduced the time to resolution in case of failures.While the above activities are absolutely needed, they are however not sufficient on their own.Users typically do not compute nor fetch data from DTNs, at least not exclusively.Examples of additional resources currently not covered by the above activities are compute clusters inside the various University departments and the commercial Cloud providers.The problem of characterizing network throughput of non-DTN resources is twofold.On one side, there are too many nodes that would need to be characterized, counting in the tens of thousands; creating a full mesh would be impossible, due to the combinatorics nature of the problem.Moreover, single node performance is often not all that important; large users typically run concurrently on thousands of such nodes, and the bottleneck is likely somewhere in the middle.And public Cloud providers have the additional property of requiring at least partial payment for the use of their resources, networking included, which makes extensive generic testing cost prohibitive.The proposed solution is to perform targeted tests that approximately mimic the user behavior, i.e. many clients reading from a few server nodes.This allows for characterizing the network experience by final users and helps detect problems that can be addressed using the remainder of the network exercising ecosystem.The major element of such a setup is the ability to easily provision and schedule a large number of endpoint that can concurrently run the desired data transfer tests.We ran a few such tests in the past few months, ranging from proving that