Server based DoS vulnerabilities in SSL/TLS Protocols Master Thesis
Sukalp Bhople · 2012
When availability is an important security requirement for the organizations then (D)DOS ((Distributed) Denial of Service) attack is one of the important attacks that needs to be addressed. The SSL protocols are widely used to secure e-commerce and other sensitive online transactions. The SSL protocol imposes a significant overhead on the web server. However, it does not create a significant overhead on the client-side. This creates an opportunity for an attacker to execute the computational DoS attacks on the SSL servers. In addition, the SSL protocols are complex in nature. In this thesis, we perform a number of experiments to analyse the DoS attack possibilities on the SSL protocol. To do so, we study the SSL protocols to come up with a number of SSL functionalities those are likely to be the weak-link and can be exploited to execute the DoS attacks. We also review the Openssl implementation to investigate the presence of DoS attack vulnerabilities in the implementation. Our experimental results show that the client authentication can create a significant computational overhead on the server side. The compression utility does not introduce any buffer overflow condition. The cryptographic operations involved in the SSL protocols are expensive and these operations are unbalanced. The server has to perform more expensive operations than the client especially when RSA key exchange method is used. We also perform the comparative experiments in which, we measure the impact of the three different DoS attack strategies on the SSL server performance. We found that the SSL renegotiation based DoS attack outperforms the other two DoS attack strategies. We describe the SSL renegotiation feature and working of the thc-ssl-dos tool [1] which is based on the SSL renegotiation feature. The results of these tests also indicate that the SSL protocol is processor intensive. Therefore, adding more CPUs can help in alleviating the impact of the DoS attack. Since, SSL renegotiation based DoS attack only works when the renegotiation is supported, it is easy to mitigate such attack just by disabling the renegotiation. Our results suggest that there is no simple fix to completely mitigate the DoS attacks. However, the likelihood of the DoS attacks can be reduced by employing enough CPU power, hardware accelerators, memory and network bandwidth. The SSL server needs be configured carefully according to the business requirements.