Efficient Style Transfer for Computational Pathology with Cross-modality Local Manipulation
Shichang Sun, Yang Zhihao, Lin Hongfei, Meng Jiana, Zhang Jincheng, Rui Wang · 2024
Style transfer has been proven to be effective in mitigating domain shift in clinical settings, enhancing the adaptability of pathology image models. However, existing methods assume that the texture of the entire image is domain-specific and irrelevant to class-specific representations. These methods enhance all regions with a single style, which can result in information loss. In this work, we propose CLAP (Cross-modality Local Augmentation for histoPathology), a data augmentation approach that enables cross-modality local manipulation for pathology images. Specifically, the combination of a text extractor network and a feature mapping network enables the integration of CLIP embeddings for style descriptions into editable latent spaces at a fine-grained level. This approach prevents the loss of regional information in whole-slide images, eliminates the need to painstakingly select directions in latent space, and enhances creativity style selection. Experimental results demonstrate that CLAP improves the regional accuracy of cross-modality editing and achieves state-of-the-art performance by enhancing generalization in histopathology classification tasks.