Virtual machine based self modifying code detection and monitoring
He Yeping · Computer Engineering and Applications Journal · 2011
Self Modifying Code(SMC) is the common form of malware to disable the static analysis of reverse engineering techniques.It is difficult for traditional operating system to handle SMC efficiently.This paper presents a Virtual Machine (VM) based framework called CASMonitor,which can monitor the given process out of guest operating system dynamically and transparently,analyze the new code generated by SMC during runtime,for entry point detection or virus-scanning.Experiments on x86/Win32 show that it can handle various popular binary packers.