Resilient Dynamic Analysis for Windows Malware Technique Discovery against Behavior Obfuscation
Yi-Ting Huang, Lisa Liu, Ying-Ren Guo, Guo-Wei Wong, Timothy Lynar, Meng Chang Chen · ACM Transactions on Privacy and Security · 2026
In this article, we focus on the robustness of behavior-based malware analysis models, justified by the need to address the high mutation rates of malware executables that debilitate conventional signature-based approaches and even behavior-based AI solutions. In response to these challenges, we propose MAMBA + , an obfuscation-resistant dynamic analysis approach tailored for uncovering malware behavior. We have assembled a comprehensive collection of behavioral obfuscation attacks designed to undermine behavior-based models. The central concept behind MAMBA + involves treating obfuscated calls as perturbed data and introducing a novel loss function to effectively balance ground-truth predictions and the handling of these perturbations. To facilitate this approach, MAMBA + designs adapted embedding mechanisms to transform traces of API calls into high-dimensional vectors for attention calculations. Through a comprehensive empirical study with seven obfuscations and three unseen attacks, we reveal important qualitative properties of MAMBA + , and quantitatively demonstrate its superiority in performance and robustness to all compared methods.