No time for asynchrony
Marcos K. Aguilera, Michael Walfish · 2009
In their early days, distributed systems were designed under a synchronous (message-passing) model, which assumes bounds on processing and communication delays. These assumptions allowed processes to handle unresponsive processes easily with end-to-end timeouts. However, current distributed systems cannot meet endto-end timing properties because they are composed of a diverse stack of layers, each with complex timing behavior. At best, these systems have predictable timing in the common case, but even slight deviations from normal load or operating conditions can produce long delays that violate the model’s assumptions (§2.1). Since the model fails to represent reality, even systems that are correct under the model can produce errors in practice. Because of this mismatch, many algorithms and systems have been designed under the asynchronous model, in which there are no assumptions on responsiveness and no way to distinguish slow from crashed processes. While this generality is appealing, we argue in this paper that the asynchronous model is poorly suited to building real systems (§2.2): the case people make for this model is problematic, the model can actually be detrimental because it hides useful timing information within the layers of the system, and the model leads to complex systems because of the inability to distinguish slow from crashed processes. Our arguments are supported by the fact that systems that are safe under asynchrony are rare, even when they incorporate components, like Paxos [9], that are specifically designed to tolerate asynchrony (§2.3). We advocate a different model, namely the asynchronous model augmented with a perfect failure detector (PFD) [4], which enables a process to tell whether another process has crashed or is merely slow. This model allows for arbitrary delays, thereby accommodating the timing complexity of current layered systems, yet it avoids a main source of complexity in the asynchronous model: the uncertainty caused by slow processes (§3.1). Failure detection in systems is often implemented using end-to-end timeouts on heartbeat messages. Such an approach fails to yield a PFD if messages have arbitrary delays (§3.1). We propose a scheme that replaces end-to-end timeouts with a more informed, accurate, and powerful mechanism: spy modules or spies (§3.2). A spy uses specialized information—including timing as-