Virtual Reality for Synergistic Surgical Training and Data Generation

We developed a cost-effective and synergistic framework, called Asynchronous Multibody Framework Plus (AMBF+), to facilitate surgical training while simultaneously generating data for algorithm development. AMBF+ provides a virtual reality (VR) environment with stereoscopic display and haptic feedback, enabling users to practice surgical skills. It also allows the generation of diverse data, including object poses and segmentation maps, that are essential for downstream computer vision algorithm training. We designed AMBF+ to be highly flexible, utilizing a modular plugin setup to accommodate different surgical procedures without modifying the core simulation logic. As a use case, we created a virtual drilling simulator for skull-base surgery, where users can modify patient anatomy using a virtual surgical drill. Additionally, we demonstrated how the data generated by the simulator can be applied to computer vision tasks, such as anatomy tracking and depth estimation, highlighting its potential for both training surgeons and developing image-guided interventions.

  • I conceived and established the initial prototype for a virtual drilling simulator, allowing users to interactively practice drilling volumetric data, specifically in anatomy
  • I conceptualized and implemented the algorithm for drill-shaft collisions, delivering haptic feedback to users regarding the virtual surgical drill shaft's interaction with anatomical structures
  • I implemented safety features to display warning messages when drilling near critical anatomy regions, enabled real-time saving and loading of simulation states, and facilitated recording and replaying of drill movements within the simulator.

Publication

Munawar, A., Li, Z., Kunjam, P., Nagururu, N., Ding, A.S., Kazanzides, P., Looi, T., Creighton, F.X., Taylor, R.H. and Unberath, M., “Virtual reality for synergistic surgical training and data generation.” Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization (2021): 1-9. Outstanding Paper Award
[GitHub]   [PDF]   [DOI]


All images © 2021 Taylor & Francis