Automated Test Results Processing The Art of Automated Crash Dump & Log Analysis
Edward Guy Smith, Mangosoft Incorporated · 2001
QA automation often leverages success on the ability to perform continuous, non-stop testing. When one test completes, its results are collected and a new test is begun. The collected results may include test logs, product logs, Dr. Watson® dumps, user mode process dumps, and kernel mode memory dumps. QA engineers then process the results, distilling them down into problem reports. As a manual process, it is difficult to achieve any level of efficiency in processing results and provide any consistency in problem reporting. Consistency in problem reporting is key to being able to distinguish new problems from ones that have already been reported, and to provide the details development engineering needs to isolate and identify a defect. Automating the results analysis process provides an efficient method for producing problem reports that uniquely identify product failures by standardizing the format and content of failure summaries and their supporting details. This paper introduces techniques used to automate the results analysis process. It examines the analysis of crash dump files and log files to extract consistent failure summaries and details, showing how these are used in problem reporting. It then studies the practical application of Automated Test Results Processing at Mangosoft Incorporated and presents data showing the impact this has had in product testing.