Hook Point Estimation for System Call Detection by Virtual Machine Monitor
Masaya Sato, Taku Omori, Toshihiro Yamauchi, Hideo Taniguchi · 2022
Virtual machine monitors (VMMs) are used to monitor the behavior of virtual machine (VM) programs. System call detection is frequently used to monitor behavior. To monitor the system calls, the VMM inserts a breakpoint, called a hook point, into the memory of the monitored VM. Researchers have determined hook points based on experimental knowledge or reading the source code of operating systems (OSes). However, reading the source codes of OS requires specialized knowledge. In addition, the appropriate hook point differs among OS versions. Therefore, analyzing the source code in each OS update is impractical. We propose a method for estimating the hook point using a memory analysis technique. The proposed method acquires the memory of the monitored VM and then searches for an appropriate instruction appropriate to hook. The search instructions depend on the processor architecture. Therefore, the proposed method does not require knowledge of OS versions or reading the source code of OSes. The experimental results showed that the proposed method precisely estimates the hook point. In addition, the overhead of the proposed method is small, considering the boot time of the monitored VM.