Static detection of C++ vtable escape vulnerabilities in binary code.
David Dewey, Jonathon T. Giffin · 2012
Static binary code analysis is a longstanding technique used to find security defects in deployed proprietary software. The complexities of binary code compiled from object-oriented source languages (e.g. C++) has limited the utility of binary analysis to basic applications using simpler coding constructs, so vulnerabilities in object-oriented code remain undetected. In this paper, we present vtable escape bugs—a class of type confusion errors specific to C++ code present in real, deployed software including Adobe Reader, Microsoft Office, and the Windows subsystem DLLs. We developed automated binary code analyses able to statically detect vtable escape bugs by reconstructing high-level objects and analyzing the safety of their use. We implemented our analysis in our own general object code decompilation framework to demonstrate that classes of object-oriented vulnerabilities can be uncovered from compiled binaries. We successfully found vtable escape bugs in a collection of test samples that mimic publicly disclosed vulnerabilities in Adobe Reader and Microsoft Excel. With these new analyses, security analysts gain the ability to find common flaws introduced by applications compiled from C++. 1.