Cloud Security via Virtualized Out-of-Band Execution and Obfuscation
Dean C. Mumme, Brooke Wallace, Robert M. McGraw · 2017
We describe a multi-layer security system called "Application Protected Execution" (APEx) that has an "In-VM monitoring" functionality protected by out-of-band memory created within a Virtual Machine (VM) on cloud-based nodes. The In-VM monitor functionality protects execution of security related software and is triggered by hooked system events to avoid context switched overhead. The APEx system is a robust defense layer that can protect a wide variety of security modules, such as security monitoring and policy enforcement. The paper also describes an application that employs APEx to protect user space software from reverse engineering and Return Oriented Programming (ROP) attacks. The application, "Code Obfuscation Engine" (CObE) performs code stirring and utilizes system calls and out-of-band memory to obfuscate program flow and protect the return stack. Utilizing APEx, it is able to jump into out-of-band memory for execution of sensitive code areas, calculation of jump points and return addresses. This protects program flow against hijacking-particularly buffer overflows and ROP attacks. CObE transforms a binary file for use with APEx protected execution directly and does not require source code.