Advancements in File Similarity Techniques: Traditional and Modern Approaches for Malware Detection
Udbhav Prasad · International Journal of Computer Trends and Technology · 2024
Threat hunting, malware analysis and digital forensic techniques often use signatures to identify malicious executables. While cryptographic hashes are helpful for identifying a particular file uniquely, attackers often tailor their malware to particular systems, releasing variants that target different platforms, operating systems, and even specific organizations or governments. As attacks become more sophisticated, security researchers have proposed “similarity” digests that attempt to overcome the limitations of cryptographic hashes and other traditional signatures by detecting variants of an executable. Modern enterprises manage tens of thousands of endpoints with billions of files, making the scalability of the proposed techniques more important than ever. This survey reviews traditional file similarity digests, such as ssdeep, sdhash, and TLSH, alongside emerging technologies like embeddings and vector databases. By classifying and comparing these techniques, the paper highlights their strengths, weaknesses, and practical applications in malware detection. Key contributions include a structured taxonomy of methods and insights into integrating traditional digests with modern vector database solutions for scalable, efficient detection. This work provides a roadmap for future research and development in this critical domain.