Reining In Windows API Abuses with In-lined Reference Monitors
Kevin W. Hamlen, Vishwath Mohan, Richard Wartell · 2010
Malware attacks typically effect damage by abusing operating system resources (e.g., the file system) that are exposed via system API calls. A method of using automated binary code-rewriting to monitor API calls and their arguments is presented and evaluated. Unlike traditional monitoring approaches, the framework requires no modification of the operating system, has no effect upon trusted processes, and preserves the behavior of most complex x86 native code binaries generated by mainstream compilers, including binaries that are object-oriented, graphical, contain callbacks, and use a mixture of static and dynamic linking. A separate verifier certifies that rewritten binaries cannot circumvent the monitor at runtime, allowing the binary-rewriter to remain untrusted. An implementation for Microsoft Windows demonstrates that the technique is effective and practical for real-world systems and architectures.