Anomaly Detection as a Data Reduction Approach for Test Event Analysis at the Edge
Mariya Occorso, Michael An, Robert Olsen, Vincent P. Perry · 2023
Traditionally, big data generated by the Army Test and Evaluation (T&E) community must be collected, processed, and stored before data analysis can occur. These phases of the big data life cycle cause a delay between when data were generated and when actionable insights are available to users. As a potential solution, we present a machine learning based approach for immediate analysis of big data collected during testing events. We utilized historical instrumentation datasets to train an anomaly detection model. This model was then used to label anomalies in the historical data records. Once the data was labelled, we trained a random forest model to classify based on the anomaly labels. This was then used to find feature importance scores in the datasets. We were able to successfully detect anomalies and determined which features were optimal for visualizing anomalous data points.