Impacts of Shared Filesystem Performance on Real-Time Data Acquisition and Analysis
Justin M. Wozniak, Sushil Regmi, Tong Shu, Ian T Foster · 2025
OverviewLarge scientific instruments are shared resources that are often coupled with advanced computing resources, such as HPC clusters, powerful storage systems, and, increasingly, machine learningoriented hardware.Each of these resources may be a multi-user system shared among disparate teams.In the context of real-time data acquisition and analysis, sharing can have a negative impact on the ability of the computing system to satisfy scientific objectives.In this study, we focus on the impact of a shared filesystem (FS), such as NFS or GPFS, on the scientific workflows that move and analysis variable-sized data sets.We then present a preliminary study on the use of predictive methods to capture and potentially feed back information about filesystem usage to the scientific goal level. MotivationThe overarching goal of the work presented here is to support automated, near-real-time scientific data acquisition and analysis by enhancing reliability through performance prediction and anomaly detection.This will be delivered as a service to workflow-level services that are capable of responding to dynamic resource availabilities and capabilities, automatically steering data collection in support of scientific goals.Communication within this workflow, including among monitoring and predictive components, will be performed over a reliable Kafka-like service [2,3].The first step in such an effort is to select or develop a predictive model that can make reasonable predictions.To do this, we need to collect representative data.In this paper, we present a representative