Suspicious URL and Device Detection by Log Mining
Yu Tao · Summit (Simon Fraser University) · 2014
Malicious URL detection is a very important task in Internet security intelligence.Existing works rely on inspecting web page content and URL text to determine whether a URL is malicious or not.There are a lot of new malicious URLs emerging on the web every day, which make it inefficient and not scalable to scan URL one by one using traditional methods.In this thesis, we harness the power of big data to detect unknown malicious URLs based on known ones with the help of Internet access logs.Using our method, we can find out not only related malicious URLs, but also URLs of new updates and CC(command and control) servers for existing malware, botnets and viruses.In addition, we can also detect possibly infected devices.We also discuss how to scale up our method on huge data sets, up to hundreds of gigabytes in our experiment.Our extensive empirical study using the real data sets from Fortinet, a leader in Internet security industry, shows the effectiveness and efficiency of our method.I would like to express my sincerest gratitude