Chaos engineering experiments in middleware systems using targeted network degradation and automatic fault injection
Anthony Pierce, Jason Schanck, Alexander Groeger, Raed Mahdi Salih, Michael R. Clark · 2021
Chaos Engineering is a way to break things in a controlled manner to assess system resilience. One of the pioneers of this concept is Netflix with its Chaos Monkey tool. Their tool was designed to automatically take down virtual machines that host their services to understand how their system reacts. This paper examines Chaos Engineering in middleware network services, as they are becoming an important technology of modular open systems to improve their resiliency. We introduce the Chaos Engineering concept through fault injection and network manipulation. We run experiments where we apply these techniques individually against a target application running on the middleware system and collect data to get an understanding of how the system responds. The collected data leads to a greater understanding of system operation, which provides actionable insight into increasing the resilience of the system and its applications. Our fault injection experiments demonstrate the ability to find specific faults that, when injected into specific application points, cause the entire application to crash into an unrecoverable state. Our network manipulation experiments can pinpoint specific network conditions that cause failures to our target application. The goal of our research is to develop methods for applying Chaos Engineering to open systems architecture as a way to improve the resiliency of such systems against natural and adversarial failure conditions, which will, in turn, lead to the development of more resilient military mission systems.