Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics Testing
Chenke Luo, Jiang Ming, Jianming Fu, Guojun Peng, Zhetao Li · IEEE Transactions on Information Forensics and Security · 2023
As an efficient and multi-platform scripting language, Lua is gaining increasing popularity in the industry. Unfortunately, Lua’s unique advantages also catch cybercriminals’ attention. A growing number of IoT malware authors switch to Lua for malicious payload development and then distribute malware in bytecode form. To impede malware code analysis, malware authors obfuscate standard Lua bytecode into a customized bytecode specification. Only the attached interpreter can execute that particular bytecode file. Rapid recovery of Lua obfuscated bytecode is essential for a swift response to new malware threats. However, existing generic code deobfuscation approaches cannot keep up with the pace of emerging threats. In this paper, we present a novel reverse engineering technique, calledinterpreter semantics testing. Given a customized interpreter used to execute obfuscated Lua bytecode, we construct a set ofLuaGadgetsthat can adapt to the customized interpreter. Each LuaGadget contains a carefully chosen opcode sequence to fulfill an observable calculation—it is designed to test one or two particular opcodes at a time. Next, we mutate unknown opcode values to generate a bunch of test cases and run them using the customized interpreter; we can observe the expected result only when the mutation hits the opcode’s right value. We perform test case prioritization to cost-effectively recover the semantics of all obfuscated opcodes. Our approach makes no assumptions about the interpreter’s structure and is free from analyzing the numerous execution traces of opcode handlers. We have evaluated our tool,LuaHunt, with Lua malware variants and real-world applications. LuaHunt is able to recover the obfuscated bytecode’s semantics within 90 seconds for each test case, and all of our deobfuscation results can pass the correctness testing. The encouraging results demonstrate that LuaHunt is a promising tool to lighten the burden of security analysts.