On the Detection of Exploitation of Vulnerabilities That Leads to the Execution of a Malicious Code
Yury V. Kosolapov · Automatic Control and Computer Sciences · 2021
Abstract Software protection from exploitation of possible unknown vulnerabilities can be ensured both by searching for (for example, using symbolic execution) and subsequently eliminating vulnerabilities and by means of detection and/or intrusion prevention systems. In the latter case, this problem is usually solved by forming a profile of normal program execution and an unacceptable deviation from the normal behavior is regarded as an anomaly or attack. The task considered in this study is to protect a given executable file (program) P from exploiting unknown vulnerabilities. The method proposed for this purpose is to construct the normal execution profile of program P, in which not only a set of legal chains of system and library functions of length l is taken into consideration, but so is the distance between adjacent function calls. This distance is calculated as the difference in the call addresses of respective functions. Taking into account distances between function calls allows detecting the execution of a malicious shell code using system and/or library function calls when the distance between at least one of the calls used in the shell code and the preceding call is untypical for program P. This study constructs an algorithm and a system for detecting abnormal code execution and describes several tests conducted when P is the Firefox browser.