Rewriting x86 binaries without code producer cooperation
Kevin W. Hamlen, Richard Wartell · 2012
Binary code from untrusted sources remains one of the primary vehicles for software propagation and malicious software attacks. All previous work to mitigate such attacks requires code-producer cooperation, has signicant deployment issues, or incurs a high performance penalty. The problem of accurate static x86 disassembly without metadata is provably undecidable, and is regarded by many as uncircumventable. This dissertation presents a framework for x86 binary rewriting that requires no cooperation from code-producers in the form of source code or debugging symbols, requires no client-side support infrastructure (e.g., a virtual machine or hypervisor), and preserves the behavior of even complex, event-driven, x86 native COTS binaries generated by aggressively optimizing compilers. This makes it exceptionally easy to deploy. The framework is instantiated as two software security systems: Stir, a runtime basic block randomization rewriter for Return-oriented programming (ROP) attack mitigation, and Reins, a machine veriable Software Fault Isolation (SFI) and security policy specication rewriter. Both systems exhibit extremely low performance overheads in experiments on real-world COTS software| 1.6% and 2.4% respectively. The foundation of the system includes three novel approaches to static x86 disassembly, along with a method of statically proving transparency for rewriting systems.