Log detector: enhancing efficiency through automated issue detection and reporting at ALMA
VIcente Coopman, Johnny W. Reveco, Camila Martinez, Enzo Barbaguelatta · 2024
Operating a cutting-edge radio telescope like ALMA demands optimal utilization of every minute available in the sky. With an increasing allocation of observation hours to researchers each year, the imperative for continuous, seamless operations grows. ALMA relies on an array of computer systems functioning on a full-time basis, with numerous concurrent users, generating approximately 50,000 logs per minute and a staggering 70 million logs per day. Addressing the challenge of managing this voluminous data flow, log detector emerges as an in-house solution designed to automate the detection and reporting of known issues. By scrutinizing logs, this tool empowers users to define Finite State Machine (FSM) states and transitions. Subsequently, users can feed logs into this machine, inducing state transitions that signal potential problems or facilitate system monitoring tasks. This article aims to spotlight the capabilities of Log Detector and its impact on operational efficiency. Additionally, it offers insights into the lessons learned while developing an in-house operational tool and outlines future development plans.