When Reliable Programs Fail: Designing for Timely, Efficient, Push-Button Recovery
Troy Martin Hughes · 2015
Software quality comprises a combination of both functional and performance requirements that together specify not only what software should accomplish, but also how well it should accomplish it. Recoverability—a common performance requirement—represents the timeliness and efficiency with which software or a system can resume functioning following a catastrophic failure. Thus, requirements for high availability software often specify the recovery time objective (RTO), or the maximum amount of time that software may be down following an unplanned failure or a planned outage. While systems demanding high or near perfect availability will require redundant hardware, network, and additional infrastructure, software too must facilitate rapid recovery. And, in environments in which system or hardware redundancy is infeasible, recoverability only can be improved through effective software development practices. Because even the most robust code can fail under duress or due to unavoidable or unpredictable circumstances, software reliability must incorporate recoverability principles and methods. This text introduces the TEACH mnemonic that describes guiding principles that software recovery should be timely, efficient, autonomous, constant, and harmless. Moreover, the SPICIER mnemonic is introduced, which describes discrete phases in the recovery period, each of which can benefit from and be optimized by TEACH principles. Software failure is inevitable but its negative impacts can be minimized through savvy SAS® development practices that facilitate recovery.