Reliability assessment framework based on feature separability for pathological cell image classification under prior bias
Takaaki Tachibana, Toru Nagasaka, Yukari Adachi, Hiroki Kagiyama, Ryota Ito, Mitsugu Fujita, Kimihiro Yamashita, Yoshihiro Kakeji · Intelligence-Based Medicine · 2026
Background and objective: Prior probability shift between training and deployment datasets challenges deep learning-based medical image classification. Standard correction methods reweight posterior probabilities to adjust prior bias, yet their benefit is inconsistent. We developed a reliability framework identifying when prior correction helps or harms performance in pathological cell image analysis. Methods: We analyzed 303 colorectal cancer specimens with CD103/CD8 immunostaining, yielding 185,432 annotated cell images across 16 cell types. ResNet models were trained under varying bias ratios (1.1-20$\times$). Feature separability was quantified using cosine similarity-based likelihood quality scores, reflecting intra- versus inter-class distinctions in learned feature spaces. Multiple linear regression, ANOVA, and generalized additive models (GAMs) evaluated associations among feature separability, prior bias, sample adequacy, and F1 performance. Results: Feature separability dominated performance ($β= 1.650$, $p 0.5$ were robust without correction, whereas those $<0.3$ consistently required adjustment. Conclusion: Feature extraction quality, not bias magnitude, governs correction benefit. The proposed framework provides quantitative guidance for selective correction, enabling efficient deployment and reliable diagnostic AI.