Automated Nuclei Segmentation in PR-IHC Breast Cancer Images Using the Cellpose Deep Learning Model
Hasanul Bannah, Mohammad Faizal Ahmad Fauzi, Sarina Binti Mansor, Md. Shoukhin Khan, Wan Siti Halimatul, Munirah Wan Ahmad, Md Serajun Nabi, Seow-Fan Chiew, Phaik‐Leng Cheah, Lai‐Meng Looi · 2025
In digital pathology, precise nuclei segmentation in immunohistochemical-stained tissue sections is essential for clinical decision-making and subsequent quantification of biomarkers. This task is particularly important for the analysis of hormone receptors in breast cancer, where the status of the progesterone receptor (PR) plays a key role in determining the response to treatment. However, because of differences in nuclear morphology, staining intensity, and overlapping structures, nucleus segmentation in PR-IHC images is still difficult. In order to separate PR-expressing nuclei from high-resolution breast cancer histopathology images, we present an automated instance segmentation frame-work in this work that is based on the Cellpose deep learning model, supported by an entirely novel ground truth (GT) dataset produced by a hybrid pipeline. In order to create the GT, 250 high-resolution PR-IHC images with trustworthy binary nuclei masks were combined with automated segmentation (StarDist), extensive manual corrections, and multiround pathological validation. On the test set, our Cellpose-based approach consistently performs properly, achieving an average F 1 score of 0.8535, precision of 0.8882, recall of 0.8215, and IoU of 0.7445. Strong segmentation of extracted and overlapping nuclei is confirmed by visual results. This study offers a useful resource for future research in hormone receptor quantification and computational pathology, as well as the first automated segmentation benchmark for the PR-IHC dataset.