A case study of gray-box fuzzing with byte- and tree-level mutation strategies in XML-based applications for exposing security vulnerabilities
Şerafettin Şentürk, Vahid Garousi, Nejat Yumuşak · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · 2025
Fuzzing is an automated process for detecting crashes and vulnerabilities in software system and it is classified as grammar- or mutation-based in terms of input generation. While the grammar-based fuzzing generates inputs from a specification and takes highly-structured inputs, mutation-based fuzzing generates inputs by modifying input files and abstract syntax trees randomly. There are not many case studies comparing the crash detection capabilities in the scope of mutation-based fuzzing. To add to the body of empirical evidence in this area, this case study compares fuzzing with different mutation strategies to evaluate their effectiveness in three aspects: fault detection effectiveness, fault detection performance and types of faults detected. Additionally we evaluate the effects of seed generation techniques to fuzzing effectiveness. We perform the fuzzing on three XML parsers, libxml2, Apache Xerces and Expat. To perform fuzzing, we use well-known mutation-based fuzzers that have different level of mutation strategies. We carry out the investigations to find the effects of seed generation to fuzzing by utilizing publicly-available seeds and the PCSG (Probabilistic Context Sensitive Grammar) based seeds. In terms of fault detection effectiveness and performance, we evaluate the number of crashes, and the number of test cases generated. With respect to mutation strategies, our results demonstrate that the bit/byte-level mutation strategy detects more crashes than tree-level mutation strategy. According to the fuzzing results, PCSG-based seeds can help detect higher number of crashes than publicly-available ones. In terms of generated test cases, whereas there is less test cases generated for PCSG based seeds compared to publicly-selected ones, bit/byte level mutations result in more test cases when compared to tree level mutation. Empirical results show that crash detection capabilities of fuzzing differ importantly based on the mutation strategy used.