Small Marker Tracking with Low-Cost, Unsynchronized, Movable Consumer Cameras For Augmented Reality Surgical Training

Surgeons improve their skills through repetition of training tasks in order to operate on living patients, ideally receiving timely, useful, and objective performance feedback. However, objective performance measurement is currently difficult without 3D visualization, with effective surgical training apparatus being extremely expensive or limited in accessibility. This is problematic for medical students, especially in situations such as the COVID-19 pandemic in which they are needed by the community but have few ways of practicing without lab access. In this work, we propose and prototype a system for augmented reality (AR) visualization of laparoscopic training tasks using cheap and widely-compatible borescopes, which can track small objects typical of surgical training. We use forward kinematics for calibration and multi-threading to attempt synchronization in order to increase compatibility with consumer applications, resulting in an effective AR simulation with low-cost devices and consumer software, while also providing dynamic camera and marker tracking. We test the system with a typical peg transfer task on the HoloLens 1 and MagicLeap One.