RAMS a nd BlackSheep: Inferring White-box Application Behavior Using Black-box Techniques
Jiaqi Tan, Priya Lakshmi Narasimhan · 2008
A significant challenge in developing automated problem-diagnosis tools for distributed systems is the ability of these tools to differentiate between changes in system behavior due to workload changes from those due to faults. To address this challenge, current, typically white-box, techniques extract semantically-rich knowledge about the target application through fairly invasive, high-overhead instrumentation. We propose and explore two scalable, low-overhead, non-invasive techniques to infer semantics about target distributed systems, in a black-box manner, to facilitate problem diagnosis. RAMS applies statistical analysis on hardware performance counters to predict whether a given node in a distributed system is faulty, while BlackSheep corroborates multiple system metrics with application-level logs to determine whether a given node is faulty. In addition, we have developed and demonstrated a novel technique to extract, from existing application-level logs, semantically-rich behavior that is immediately amenable to analysis and synthesis with other numerical, black-box metrics. We have evaluated the efficacy of RAMS and BlackSheep in diagnosing real-world problems in the Hadoop distributed parallel programming system. Submitted in partial fulfillment of the requirements for the Senior Honors Thesis program in the School of Computer Science at Carnegie Mellon University Acknowledgements: I would like to thank Priya Narasimhan for her advice, constant encouragement and inspiring thoughts, and Xinghao Pan for enduring many animated descriptions of this work, without which this project would not have been possible. This work is also dedicated to my parents and sister, for their unwavering, whole-hearted support for me to pursue my dreams.