Development of an Object Recognition-Based Assistance System for Surgical Instrumentation Using an UR5e Robotic Arm
Guillermo Michel González, Salvador Castro Reynoso, José Pablo Hernández Alonso · 2024
Given the critical shortage of surgical instrument nurses, especially in Mexico, with a deficit of 68.9%, this study presents the development of a prototype assisted by artificial intelligence to enhance efficiency and safety in low-risk surgical procedures. The prototype utilizes object recognition to assist in surgical instrumentation. The methodology includes creating a dataset of images representing surgical instruments, using YOLOv8 for object identification, MediaPipe for detecting key points in the hands, and setting up a visual space using an Intel RealSense camera. These data are transformed into Cartesian coordinates to communicate with a UR5 collaborative robot using Python. The surgical instruments identified with YOLOv8 were classified into five categories: scalpel, forceps, curved Mayo scissors, straight Mayo scissors, and a discriminative class. The use of the pre-trained MediaPipe model allowed the system to identify handoff points for the instruments on the surgeon's hand. Additionally, the Intel RealSense depth camera facilitates the calculation of the viewing area size, enabling the determination of specific coordinates for the UR5 robot, which uses the key points generated by YOLOv8 and MediaPipe. This prototype provides a comprehensive solution to address the shortage of surgical instrument nurses in surgical settings, promoting closer and more effective collaboration between medical professionals and advanced technology.