Deep reinforcement learning for long-horizon reservoir operation: Temporal horizon, state representation, and hydrological data synthesis
Zaichao Xie, Minglei Ren, Wei Xu, Te Zhang, Bing Zhu, Dong Li, Shuncai Zhang, Junbo Wang · Journal of Hydrology · 2026
Deep reinforcement learning offers a flexible paradigm for reservoir operation. However, its deployment in long-horizon settings remains difficult when rewards must be propagated across years under strong seasonal regularities and limited multi-year inflow trajectories. Principled designs for episode horizons, seasonality-aware states, and inflow data augmentation are still lacking. This study aims to develop a DRL environment tailored to long-horizon reservoir operation and systematically evaluate its performance. First, five episode-length configurations are compared within a unified Actor-Critic framework to determine the case-specific optimal training horizon for the Three Gorges Reservoir (TGR). Second, a state-space ablation study is conducted to quantify the independent contribution of each state dimension. Third, a synthetic inflow generation scheme is proposed to mitigate data scarcity. The scheme integrates STL (seasonality-trend-remainder) decomposition, autoregressive modeling, and Markov year-type transitions to synthesize hydrologically consistent inflow data. The results from the TGR case study show that a 4-year episode length achieves an optimal balance between training efficiency and value-estimation stability. Two-dimensional periodic date encoding yields an 85.5% performance improvement over the baseline without date encoding by endowing the agent with explicit seasonality awareness. The proposed synthetic inflow scheme generates large-scale training data while closely preserving key multi-scale hydrological statistics and enables robust policy performance under consecutive extreme inflow scenarios. The DRL environment, state representation, and data-generation designs developed in this study provide reusable methodological templates for DRL development and deployment in long-horizon reservoir operation.