CrashMonkey and ACE
Jayashree Mohan, Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, Vijay Chidambaram · ACM Transactions on Storage · 2019
We present C rash M onkey and A ce , a set of tools to systematically find crash-consistency bugs in Linux file systems. C rash M onkey is a record-and-replay framework which tests a given workload on the target file system by simulating power-loss crashes while the workload is being executed, and checking if the file system recovers to a correct state after each crash. A ce automatically generates all the workloads to be run on the target file system. We build C rash M onkey and A ce based on a new approach to test file-system crash consistency: bounded black-box crash testing ( B 3 ). B 3 tests the file system in a black-box manner using workloads of file-system operations. Since the space of possible workloads is infinite, B 3 bounds this space based on parameters such as the number of file-system operations or which operations to include, and exhaustively generates workloads within this bounded space. B 3 builds upon insights derived from our study of crash-consistency bugs reported in Linux file systems in the last 5 years. We observed that most reported bugs can be reproduced using small workloads of three or fewer file-system operations on a newly created file system, and that all reported bugs result from crashes after fsync()-related system calls. C rash M onkey and A ce are able to find 24 out of the 26 crash-consistency bugs reported in the last 5 years. Our tools also revealed 10 new crash-consistency bugs in widely used, mature Linux file systems, 7 of which existed in the kernel since 2014. Additionally, our tools found a crash-consistency bug in a verified file system, FSCQ. The new bugs result in severe consequences like broken rename atomicity, loss of persisted files and directories, and data loss.