3Describe-Creating Tangible AR (Augmented Reality) Objects Using Depth Camera
Kevin Zhang, Mike Tianci Ye, Chris Cheng Zhang, Rongdi Ni, Yitong Liu, Anqi Xing · 2023
The transition from in-person to online classes, accelerated by the global impact of Covid-19, has brought both accessibility and disengagement challenges. While online platforms facilitate learning for distant and international students, the loss of interactive elements diminishes the overall educational experience. This article proposes a novel solution inspired by MIT Professor Dr. Patrick Winston's concept of using “props” to enhance learning. Leveraging augmented reality (AR) technology, an application can be developed to introduce tangible AR elements into the online learning environment. The design and methodology outline the use of Python libraries, including OpenCV and Mediapipe, along with the Intel RealSense D435 depth camera. By employing hand-tracking techniques, real-world coordinates are deduced, allowing the creation of interactive AR objects. Trigonometry is utilized to convert 3D coordinates into 2D projections on the video screen, ensuring accurate representation. The visual perception of depth is achieved by subdividing lines, allowing for the dynamic interaction of virtual objects and real hands. The results and analysis section showcases the functionality of the developed application. A 3D cube or prism appears on-screen, responding to touch and rotation gestures. The collision detection algorithm, assuming a spherical bounding box, determines whether the cube is touched, altering its color and position accordingly. Limitations, such as the imprecise collision area for elongated shapes and potential aliasing issues, are discussed as sources of error. Looking forward, the discussion section explores future enhancements and applications. Incorporating advanced modeling tools like OpenGL or Wavefront could introduce more complex 3D models. Interactive features such as hand gestures for rotation or grabbing could further enrich the online learning experience. This project serves as a foundation for the development of interactive and engaging online learning methods, bridging the gap between physical and virtual educational environments.