A 3D PRINTED EMBEDDED AI-BASED MICROSCOPE FOR PATHOLOGY DIAGNOSIS
Bowen Chen, Drew F. K. Williamson, Faisal Mahmood · Journal of Pathology Informatics · 2022
Background: Management of aggressive malignancies, such as glioma, is complicated by a lack of predictive biomarkers that could reliably stratify patients based on the treatment outcome.The central hypothesis is that complex processes driving glioma recurrence and treatment resistance cannot be fully understood without integration of multiscale factors such as cellular morphology, tissue microenvironment, and macroscopic features of the tumor and the host tissue.At the same time, AI provides a wonderful tool to examine and integrate complex features from diverse data and enhance patient outcome prediction.Methods: We present a weakly-supervised, multimodal deep learningbased model fusing histopathological and radiology features for glioma survival predictions.We deploy a multiple-instance learning approach, which allows to train the model using patient outcome as the only label, effectively surpassing the need for manual annotations of predictive regions, which are often unknown.The model is trained on glioma data from The Cancer Genome Atlas and The Cancer Imaging Archive, which contain paired whole-slide H&E-stained images, multimodal radiology scans (including preoperative T1Gd, FLAIR, T2-w, and T1-w MRI brain scans) and ground truth survival labels.First, unimodal networks for the respective WSIs and radiology scans are trained individually for the survival task.Consequently, the unimodal models are used as feature extractors for the multimodal framework.The multimodal model fuses the information from the respective modalities by applying a gating-based attention mechanism to first control the expressiveness of each modality, followed by the Kronecker product to model pairwise feature interactions across modalities.In this way, the multimodal survival model can incorporate large and diverse data and explore new potential biomarkers across modalities.Results: The models were trained on a set of 205 patients, randomly divided into train (90%) and validation set (10%).Performing 10-fold cross validation, the unimodal algorithms trained on radiology or pathology data obtained an average concordance index (c-index) of 0.704 and 0.712 respectively.By incorporating information from different modalities, the multimodal algorithm resulted in higher performance, with an average c-index of 0.733. Conclusion:The presented framework demonstrates feasibility of weakly-supervised multimodal integration of radiology and histology data for improved survival prediction in glioma patients.Due to the absence of manual annotations, the framework can be easily extended to accommodate other modalities such as IHC stains, molecular data, or patient clinical records, to further enhance patient risk stratification.