Detecting Hidden Communications Protocols
Richard R. Brooks · 2013
Abstract : The work funded by the grant is structured in three parts: We analyzed the vulnerability of the current generation anonymity tools to traffic analysis attacks. We specifically concentrate on SSH security and The Onion Router (Tor) anonymity tools. Our analysis used deterministic hidden Markov models (HMMs). We used traffic timing data to analyze one of the most sophisticated and popular types of cybercrime tools -- botnet. We presented two botnet detection methods: centralized botnet traffic detection using HMMs and probabilistic context-free grammars (PCFGs) for centralized and P2P botnet traffic detection. Finally, a hybrid network security scheme that combines the advantages of widely deployed security technologies (intrusion detection systems (IDS) and honeypots) was proposed. The scheduling problem of the security system was modeled as an average decentralized partially observable Markov decision process (DEC-POMDP) and solved using our nonlinear programming (NLP)-based solution method.