HAP-Analyzer: A Hybrid Static, Dynamic, and Memory Analysis Approach for Fileless Malware
Ilker Kara, Kubra Yılmaz · IEEE Access · 2026
Fileless malware has become an increasingly prominent threat in contemporary computing environments, largely due to its reliance on in-memory execution and its ability to circumvent traditional file centric security mechanisms. Addressing these challenges, this study introduces HAP-Analyzer, a hybrid forensic analysis framework that combines static, dynamic, and memory-based techniques to enable a systematic and traceable examination of fileless malware activities. Instead of introducing a novel detection algorithm, the proposed framework concentrates on the structured integration and correlation of multiple analysis phases to enhance artifact reliability and forensic traceability. Within this framework, static analysis is applied to inspect binary properties and embedded components, while dynamic analysis focuses on monitoring runtime behavior, execution paths, and interaction patterns. These stages are further augmented by memory-based analysis, which provides access to transient artifacts residing exclusively in volatile memory and therefore inaccessible through conventional inspection methods. The findings generated at each phase are explicitly linked, allowing the results of one stage to inform and refine subsequent analyses, thereby establishing a cohesive and iterative investigation workflow. The practical applicability of the proposed approach is demonstrated through a real-world fileless malware case study, in which artifacts obtained from static, dynamic, and memory analyses are examined collectively to provide a unified forensic perspective. In addition, the framework is evaluated quantitatively using artifact-oriented metrics, including analysis duration, artifact recovery effectiveness, and consistency across analysis phases. A comparative assessment against existing hybrid malware analysis frameworks further positions HAP-Analyzer as an analyst-driven and forensics-focused solution, prioritizing interpretability, methodological transparency, and reproducibility rather than full automation. The results suggest that the proposed framework is particularly well suited for incident response operations, post-compromise investigations, and advanced malware forensic scenarios where evidential integrity and analytical clarity are of paramount importance.