Revolutionizing the Field of Grey-box Attack Surface Testing with Evolutionary Fuzzing
Jared D. DeMott, Richard J. Enbody · 2007
Runtime code coverage analysis is feasible and useful when application source code is not available. An evolutionary test tool receiving such statistics can use that information as fitness for pools of sessions to actively learn the interface protocol. We call this activity grey-box fuzzing. We intend to show that, when applicable, grey-box fuzzing is more effective at finding bugs than RFC compliant or capture-replay mutation black-box tools. This research is focused on building a better/new breed of fuzzer. The impact of which is the discovery of difficult to find bugs in real world applications which are accessible (not theoretical). We have successfully combined an evolutionary approach with a debugged target to get real-time grey-box code coverage (CC) fitness data. We build upon existing test tool General Purpose Fuzzer (GPF) [8], and existing reverse engineering and debugging framework PaiMei [10] to accomplish this. We call our new tool the Evolutionary Fuzzing System (EFS), which is the initial realization of my PhD thesis. We have shown that it is possible for our system to learn the targets language (protocol) as target communication sessions become more fit over time. We have also shown that this technique works to find bugs in a real world application. Initial results are promising though further testing is still underway. This paper will explain EFS, describing its unique features, and present preliminary results for one test case. We will also discuss ongoing research efforts. First we begin with some background and related works. Previous Evolutionary Testing Work “Evolutionary Testing uses evolutionary algorithms to search for software test data. For white-box testing criteria, each uncovered structure-for example a program statement or branch-is taken as the individual target of a test data search. With certain types of programs, however, the approach degenerates into a random search, due to a lack of guidance to the required test data. Often this is because the fitness function does not take into account data dependencies within the program under test, and the fact that certain program statements need to have been executed prior to the target structure in order for it to be feasible. For instance, the outcome of a target branching condition may be dependent on a variable having a special value that is only set in a special circumstancefor example a special flag or enumeration value denoting an unusual condition; a unique return value from a function call indicating that an error has occurred, or a counter variable only incremented under certain conditions. Without specific knowledge of such dependencies, the fitness landscape may contain coarse, flat, or even deceptive areas, causing the evolutionary search to stagnate and fail. The problem of flag variables in particular has received much interest from researchers (Baresel et aL, 2004; Baresel and Sthamer, 2003; Bottaci, 2002; Harman et aL, 2002), but there has been little attention with regards to the broader problem as described. [1]” The above quote is from a McMinn paper that is pushing forward the field of traditional evolutionary testing. However, in this paper we propose a method for performing evolutionary testing (ET) that does not require source code. This is useful for third-party testing, verification, and security audits when the source code of the test target will not be provided. Our approach is to track the portions of code executed (“hits”) during runtime via a debugger. Previous static analysis of the compile code, allows the debugger to set break points on functions (funcs) or basic blocks (BBs). We partially overcome the traditional problems of evolutionary testing by the use of a seed file, which gives the evolutionary algorithm hints about the nature of the protocol to learn. Our approach works differently from traditional ET in two important ways: 1. We use a grey-box style of testing that allows us to proceed without source code 2. We search for sequences of test data, known as sessions, which fully define the documented and undocumented features of the interface under test (protocol discovery). This is very similar to finding test data to cover every source code branch via ET. However, the administration, of discovered test data is happening during the search. Thus, test results, are discovered as our algorithm runs. Robustness issues are recorded in the form of crash files and Mysql data, and can be further explored for exploitable conditions while the algorithm continues to run.