Detecting and Modeling Polymorphic Shellcode
Omar Nbou · Spectrum Research Repository (Concordia University) · 2010
In this thesis, we address the problem of modeling and detecting polymorphic engines shellcode.By polymorphic engines, we mean programs having the ability to transform any piece of malware into many instances consisting of different code but having the same functionality as the original malware.Typically, polymorphic engines work by encrypting the target malware using various encryption techniques and providing a decryption module in order to execute the newly encrypted instance.Moreover, those engines have the ability to mutate their decryption routine making them unique from one instance to another and hard to detect.Our analysis focuses on polymorphic shellcode, which is shellcode that uses a polymorphic engine to mutate while keeping the original function of the code the same.We propose a new concept of signatures, shape signatures, which cope with the highly mutated nature of those engines.Those signatures try to identify the constant part as well as the mutated part of the deciphering routines.This combination is able to cope with the highly mutated nature of those engines in a much more efficient way compared to traditional signatures used in most intrusion detection systems.The second part of the thesis aims at modeling those polymorphic engines by showing that they exhibit common characteristics.The analysis of bit positions and byte composition of decoders shows us that polymorphic decoders exhibit a specific byte composition and can be mapped.iiiList of Tables 4.1 Window Extraction for a Gene . . . . . . . . . . . . .