File Transfer Log Analysis: A meteorological data 2 center case study
I. Koga, Eugênio Sper de Almeida · Journal of Computational Interdisciplinary Sciences · 2016
The Center for Weather Forecast and Climate Studies (CPTEC) from the Brazilian National Institute for Space research (INPE) provides valuable meteorological information through file transfer services.Logs usually provide information related to services usage, furthermore they can show meaningful insights from data, user's location and predict data infrastructure growth.Due to the high demand of these services, it is very difficult to keep track of the information stored in the file transfer service logs.In this work, we captured file transfer logs and used the open source Elasticsearch, Logstash and Kibana (ELK) stack for data analysis.Logstash serves for data collection and transport pipeline, Elasticsearch allows data index and Kibana provides a user-friendly interface for queries, results, and visualizations.Our preliminary results with File Transfer log data from INPE/CPTEC show interesting details from data: high rate of access due to a misconfigured script, activity of real and anonymous users and more.We would like to emphasize that this work can be extended and used with other kinds of data, providing useful insights to researchers from many other scientific areas.