What is Wrong with Continual Learning in Medical Image Segmentation?
Camila González, Nick Lemke, Amin Ranem, Georgios Sakas, Anirban Mukhopadhyay · 2025
Continual learning (CL) is crucial for advancing medical image segmentation. In real-world clinical scenarios, datasets arrive sequentially, are often incomplete, and are subject to strict privacy constraints, making traditional static training approaches impractical. While many CL methods focus on mitigating catastrophic forgetting, they frequently overlook the broader requirements of scalability and compatibility with real-world workflows. Moreover, most existing methods are not robust to distribution shifts and require domain labels. We present UNEG (U-Net Expert Gate), a simple yet highly effective multi-model benchmark for continual learning in medical image segmentation. UNEG avoids catastrophic forgetting by maintaining separate task-specific segmentation models and using an autoencoder-based oracle to dynamically identify the most appropriate model during inference. Unlike complex methods, UNEG adheres to key constraints necessary for deployment in clinical environments: privacy compliance, robustness across domains, scalability, and generalizability. Through a systematic comparison, we show that UNEG satisfies all these constraints, making it an ideal baseline for evaluating the effectiveness of other CL methods. This work not only highlights the limitations of existing methods but also redefines the priorities for future research to ensure clinical applicability and deployment readiness.