Automatic Reverse Engineering of Unknown Security Protocols from Network Traces
Yudan Fan, Yuna Zhu, Lin Yuan · 2018
Previous network-based protocol reverse engineering methods have only considered plaintext format of payload, and have not been suitable for security protocols which include many ciphertext data. We propose a novel approach to reverse security protocols from network traces-named SPREA (security protocols reverse engineering approach). SPREA extracts protocol keywords sequences hierarchically using sequential pattern mining for the first time, which would provide a new idea for plaintext format parsing. On this basis, SPREA utilizes the randomness feature of ciphertext data to locate ciphertext fields based on entropy estimation. Then SPREA infers the state machine using sequential pattern mining and Prospex method. Finally we evaluate SPREA on four classical security protocols. The experimental results show that without using dynamic binary analysis, SPREA can parse true protocol format and infer state machine purely from network traces with high accuracy.