A Study of Applied Passive TLS Analysis
Anders Sefjord Torbjørnsen · NORA - Norwegian Open Research Archives · 2018
While the Internet is moving towards more and more encryption of the network traffic, it is also a trend that is picked up by authors of malware. The traditional way of detecting malicious traffic or malicious behaviour on the network is to use a signature-based network intrusion detection system. A signature-based system relies on reading the network traffic in plaintext in order to detect patterns, which it is not able to do when the traffic is encrypted. One work-around for this problem is to use SSL/TLS inspection. Instead of breaking end users privacy by inspecting what they believe is encrypted communication, this thesis investigates the possibility of detecting malicious TLS encrypted network traffic passively. By taking a look at properties exchanged when the encrypted network communication channel is established, as well as the behaviour of the network traffic, the thesis uses these properties in a machine learning algorithm to classify network traffic as either benign or malicious. While the machine learning algorithm is easy to implement in a proof-of-concept, the lack of publicly, up-to-date datasets of benign, encrypted network traffic with TLS is almost non-existent. This leads to the creation of a TLS encrypted network traffic generator that creates a baseline for what is considered as benign TLS traffic. Malicious TLS traffic is collected from open sources, and run through a multilayer perceptron with backpropagation. Two experiments were carried out during this thesis; one leading to a correct classification rate of 83% using network behaviour. The other experiment looked at the ciphersuites found in the TLS handshake of the traffic, and had a correct classification rate of 80%.