Time Series Classification for Modality-Converted Videos: A Case Study on Predicting Human Embryo Implantation from Time-Lapse Images
Mehryar Abbasi, Parvaneh Saeedi, Jason Au, Jon C. Havelock · 2023
Video analysis requires both spatial and temporal data analysis, unlike image analysis, which is limited to processing only spatial information. The added complexity for analyzing videos translates into the requirement for more sophisticated algorithms with diverse data to optimize a model. Creating large video datasets for analysis tasks is challenging, especially in the medical field, where data accessibility is limited. As a result, many computationally complex methods, such as 3D-CNN models, are not well-suited for medical applications. Therefore, innovative strategies are required to train deep learning (DL)-based models for limited video data. This paper proposes a system to predict human embryo implantation outcome in In Vitro Fertilization (IVF) process by analyzing image sequences captured during incubation. However, data availability is restricted for this task as acquiring and annotating data involves several complex steps. The proposed approach focuses on utilizing morphological changes of embryos over time and linking them to the outcome. We convert embryonic microscopic videos into multivariate time series arrays and apply state-of-the-art time series classifiers to predict growth patterns and outcomes. However, these classifiers fail to utilize the temporal patterns in the data and result in poor performance. Therefore, we propose to modify the time series classifiers with attention mechanisms that can capture both short- and long-term dependencies and improve the accuracy of predicting the success of the IVF procedure. The proposed method11https://githuh.com/mehryar72/Emhryo-TSC demonstrates promising results, improving the prediction accuracy by 3.3% for Day 3 and 3.1 % for Day 5 embryo time-lapse videos. Our ensemble classifier achieved a prediction accuracy of 77.5%, a 5.2% improvement over the state-of-the-art.