Analysis and measurement of latency in group communication systems
E. Thomopoulos, Louise E. Moser · 1998
Group communication systems provide a foundation upon which fault-tolerant distributed systems can be built by supporting the application programmers in maintaining the consistency of replicated information. To determine whether these distributed systems can meet their real-time requirements, it is necessary to determine the message delivery latency of the underlying group communication systems. This dissertation presents an analysis, and measurements, of the probability density functions (pdfs) for the latency from origination to delivery of a message by the Totem group communication system in the presence of message loss and token loss. The Totem system provides reliable totally ordered delivery of messages to processes in process groups over single or multiple local-area networks (LANs) connected by gateway processors. Totem imposes a logical token-passing ring on each of the LANs to achieve these message delivery guarantees. The main idea of the analysis of the pdfs for the latency for the Totem protocols involves decomposing the latency into independent components and convolving the pdfs of those components. The analysis shows that the difference in performance between Poisson and deterministic arrival processes, though significant for a single ring, is negligible for multiple-ring networks. The analysis also shows that multiple-ring networks can achieve lower mean latency, less variability, and shorter tails of the latency distribution than can be achieved with a single ring containing the same number of processors. Experimental measurements of the message delivery latency for the Totem system are presented and compared with the analytical results. The experiments involved a single LAN (Ethernet) and two LANs (Ethernets) of Sun workstations connected by a gateway workstation. The clock skew between the workstations was eliminated from the measured latencies by equating the sample means of the actual latencies. The measurements of the pdfs for the latency are similar to those obtained from the analytical formulas, particularly for values of the latency with high probabilities. Limitations of the analytical model, and insight about communication and scheduling of the Solaris operating system, are also discussed.