Custom virtual machine implementation and its influence on executable static properties
Jozef Kostelansky, Ľubomír Dedera · 2019
There are many ways how to change static properties of an executable code. Packing and encryption are the most common ways. All these techniques suffer from von Neumann architecture model, which means that the code to be executed must be loaded into the primary memory. So usually the analyst is able to dump the unpacked code from the memory. Virtual machine-based obfuscators try to solve this issue by implementing custom RISC-based interpreters. So even if the analyst is able to dump the shell code from the memory, only proprietary and unknown instruction set based shell code is found, which decreases the value of current tools. The goal of this paper is to design and implement custom metamorphic reconfigurable virtual machine using instruction substitution method and measure the influence of this technique on static properties. Measurement are made on Fibonacci order implementation in custom shell code.