Multiview Integration Network for Multitask Robotic Surgical Scene Analysis
Wenting Shen, Yaonan Wang, Min Liu, Jiazheng Wang, Renjie Ding, Zhe Zhang, Erik H. W. Meijering · IEEE Transactions on Instrumentation and Measurement · 2025
Surgical scene analysis holds a pivotal role in robot-assisted surgery. However, existing methods often suffer from single or little views, leading to erroneous scene analysis conclusions. To address these issues, a novel Multiview Integration Network (MVINet) is proposed, which comprehensively analyzes the surgical scene by integrating effective information from multiple views, including global, local, dynamic, and static views. As a multitask scene analysis network, MVINet could simultaneously perform semantic segmentation of surgical instruments and detection of instrument-tissue interaction. By designing a unique Global-Local Spatial Feature Memory module (G-LSFM), the joint information from globally and locally analyzed views following graph analysis simultaneously enhances the accuracy of multitask scene analysis. The Dynamic-Static Visual Feature Memory module (D-SVFM) introduces an innovative approach by simultaneously incorporating temporal feature from consecutive frames and static feature from single frame into the node features of the interaction reasoning network. The multiscale perspective of both features further enhances the ability of the module to analyze the complex scenes. Experimental results on a public and a private dataset demonstrate that our method achieves superior performance compared to other state-of-the-art methods for two crucial tasks in surgical scene analysis. In the instrument segmentation task, MVINet outperforms the second-best method by 1.73% and 0.55% in terms of mIoU scores. For the interaction detection task, MVINet surpasses the second-best method by 10.14% and 5.47% in mAP scores.