Innovative Method for Detecting Malware by Analysing API Request Sequences Based on a Hybrid Recurrent Neural Network for Applied Forensic Auditing
Serhii Vladov, Victoria Vysotska, Vitalii Varlahov, Mariya Nazarkevych, Serhii Bolvinov, V. H. Piadyshev · Applied System Innovation · 2025
This article develops a method for detecting malware based on the multi-scale recurrent architecture (time-aware multi-scale LSTM) with salience gating, multi-headed attention, and a sequential statistical change detector (CUSUM) integration. The research aim is to create an algorithm capable of effectively detecting malicious activities in behavioural data streams of executable files with minimal delay and ensuring interpretability of the results for subsequent use in forensic audit and cyber defence systems. To implement the task, deep learning methods (training LSTM models with dynamic consideration of time intervals and adaptive attention mechanisms) and sequence statistical analysis (CUSUM, Kulback–Leibler divergence, and Wasserstein distances), as well as regularisation approaches to improve the model stability and explainability, were used. Experimental evaluation demonstrates the proposed approaches’ high efficiency, with the neural network model achieving competitive indicators of accuracy, recall, and classification balance with a low level of false positives and an acceptable detection delay. Attention and salience profile analysis confirmed the possibility of interpreting signals and early detection of abnormal events, which reduces the experts’ workload and reduces the number of false positives. This study introduces the new hybrid architecture development that combines the advantages of recurrent and statistical methods, the theoretical properties formalisation of gated cells for long-term memory, and the proposal of a practical approach to the model solutions’ explainability. The developed method implementation, implemented in the specialised software product form, is shown in a forensic audit.