Anomaly detection applied to the classification of cytology images
Carlo Bruno Marta, M. Doblaré, Jónathan Heras, Gadea Mata, Teresa Ramírez · Biomedical Signal Processing and Control · 2025
Cytology is a branch of pathology that diagnoses diseases and identifies tumours by looking at single cells, or small clusters of cells, using images observed under microscopes. Traditionally, pathologists manually analyse cytology images, a time-consuming and subjective task that could be significantly accelerated and improved through the application of computer vision and deep learning techniques. However, existing deep learning methods for cytology images need annotation at the cell level, a laborious and cumbersome process that requires the ability of pathologists with high experience. In this paper, we tackle this issue by using an anomaly detection approach that requires only annotation at the level of cytology images. Our approach splits cytology images into patches, and then uses an anomaly detection model to highlight anomalous cells. For the anomaly detection model, several reconstruction-based and embedding-based methods have been studied, the latter showing a better performance than the former. In particular, the best reconstruction-based method, based on a GAN model, achieved a perfect recall, a precision of 73.61%, and an AUROC of 69.8%; whereas, the best embedding-based method, being the PatchCore algorithm with a ResNet 50 backbone, obtained a perfect recall, a precision of 98.39%, and an AUROC of 99.98%. Finally, in order to facilitate the usage of our approach by pathologists, an ImageJ macro has been implemented. Thanks to this work, the analysis of cytology images and the diagnosis of associated diseases will be faster and more reliable. • An anomaly detection approach for classifying cytology images is proposed. • The approach requires annotation not at the cell level but at the image level. • Several anomaly detection methods are studied. • The PatchCore algorithm produced the best results. • A simple-to-use ImageJ macro was developed to facilitate adoption by pathologists.