Accurate Recovery of Functions in COTS Binaries
Rui Qiao · Academic Commons (Stony Brook University) · 2017
Binary analysis and instrumentation play a central role in COTS software security. They can be used to detect and prevent vulnerabilities, mitigate exploits, enforce security policies, and so on. Many security instrumentations work at the granularity of functions. However, unlike high-level languages, functions in binaries are not clearly demarcated. To complicate matters further, functions in binaries may have multiple entry points and/or exit points. Some of these entries or exits may not be determined simply by instruction syntax or code patterns. Moreover, many functions are reachable only through indirect control transfers, while some may be altogether unreachable. In this dissertation, we present an approach that overcomes these challenges to accurately identify function boundaries, as well as calls and returns. Our approach is based on fine-grained static analysis, relying on precise models of instruction set semantics derived in part from our previous work. In the later part of the work, we expand our investigation to recover the next crucial piece of information that is lost in high-level language to binary translation: the types and numbers of function parameters. We propose an approach that uses fine-grained binary analysis to address this problem. We evaluate this technique by applying it to enforce fine-grained control-flow integrity policies. While our approach is widely applicable to all binaries, when combined with recovered C++ semantics, it provides significantly improved protection. | 110 pages