Leveraging cloud computing to speedup user access log mining
Yun Li, Yongyao Jiang, Fei Hu, Chaowei Phil Yang, Armstrong, Thomas Huang, D. F. Moroni, Chris Fench · 2016
It is very challenging for scientists to find the right oceanographic data in a fast manner. A novel approach was proposed to analyze user access logs to explore the implicit relationship between oceanographic datasets. This paper reports a cloud-based data analytics framework to speed up the process for dealing with problems, such as (1) user access logs keep growing as users keep interact with data center websites; (2) the data analysis process involves several computing- intensive steps such as session reconstruction and latent semantic analysis (LSA); (3) the dynamic data volume requires on-demand computing resources to deliver time-sensitive computing services. To meet the requirement of dynamic computing-resources, cloud computing is leveraged to facilitate setting up cluster and speed up log mining process. In addition, Spark-based log partition strategies are integrated into our cloud-based framework to conduct log processing tasks in parallel. This experimental system is deployed on the NASA AIST cloud platform.