Message level fault injection in distributed systems.
Scott Dawson, Farnam Jahanian · Deep Blue (University of Michigan) · 1997
Ensuring that a system meets its prescribed specification is a growing challenge that confronts software developers, especially for distributed applications with strict dependability and timeliness constraints. This dissertation presents scORCHESTRA, in architecture for building fault injection layers that are inserted into protocol stacks. These fault injectors are used for identifying design and implementation errors in distributed protocols and applications. scORCHESTRA is highly portable. It can be inserted into different protocol stacks and can be placed at different layers within a single stack. This dissertation makes several key contributions to the state of the art in fault injection of distributed communication protocols. First, the scORCHESTRA framework supports fault injection with the aim of discovering and removing faults from protocol implementations. The state of a participant in a distributed protocol depends largely on the messages exchanged between participants. scORCHESTRA employs a technique called message-level fault injection, which allows system designers and test engineers to manipulate messages as they are exchanged, effectively manipulating the state of a participant. Second, scORCHESTRA addresses the intrusiveness of fault injection on target protocols, taking advantage of real-time operating system features, when possible, to quantify and to compensate for intrusiveness. Third, scORCHESTRA has been applied to testing protocols that have many participants. Coordination between scORCHESTRA fault injectors, and the semantics of data collected at different places in the system are addressed. Finally, this research introduces techniques that allow scORCHESTRA to interpret and manipulate the headers of multiple protocols, resulting in an scORCHESTRA-based approach to testing services comprised of multiple protocols. scORCHESTRA tools have been implemented on a variety of platforms. These tools have demonstrated scORCHESTRA's effectiveness through experimentation on several commercial and prototype protocol implementations. The classes of protocols studied include reliable data transmission on the Internet (e.g., TCP), multiparticipant and multilayered services such as group membership and multicast protocols, and services providing timeliness guarantees to real-time applications such as audio-conferencing. These experiments have led to the discovery of numerous specification violations and design errors in these systems. Insights gained through this experimentation have resulted in significant improvements to scORCHESTRA and a public release of the tool.