Holograms for Minimally Invasive Surgery Training and Planning

Abstract: The medical field has rapidly adopted Minimally Invasive Surgery (MIS) as a powerful alternative to traditional surgery as it reduces infection, pain, hospital stays, and costs. Despite these benefits, the in-depth knowledge of procedures coupled with the variety of tools used requires extensive training and technical planning for successful surgery. Prevailing Virtual Reality (VR) and Augmented Reality (AR) applications enable a broad spectrum of MIS training and pre-surgery planning. However, these currently existing systems are bulky, complex, and expensive and have had poor adoption rates in the field attributed to a lack of knowledge and financial barriers2. In this paper, we present a novel low-cost approach to develop a mobile hologram platform for MIS surgery training and potentially pre-surgery planning. Our developmental process consists of converting a patient's CT DICOM data to a 3D model using region of interest identification and volumetric or surface rendering. We prioritize realistic model texturing as the texture has a significant impact on visual perception and potential for use in medical diagnosis. The realistic 3D model can then be displayed on a variety of accessible devices such as mobile phones, 2D tablets, 3D tablets, and phone-based Google Cardboards or HUD headsets, all with their tradeoffs. The integration of models with devices can rely on different software for development, in our instances exploring the capabilities of Unity, Unreal, React, and XCode. Using this software, we then developed an overlay functionality that tracks aspects of the physical world and places the virtual 3D model on physical models of organs, mannequins, and/or live humans, allowing for real-world replications of medically relevant augmented scenes. An integral part of surgical practices is incisions that can cause blood flow. This is a feature we are mimicking with fluid dynamics and implementing using particle systems to yield a high-quality real-world medical training platform. To use these realistic models for MIS training, we developed specific controls based on ergonomics research that were suited for each platform. We experimented with approaches such as hand-tracking and touch-based screen gesture controls to tailor usability to device specifications. Finally, to improve perceptual adaptation, we focus on the disparate distance as it plays a critical role in user experience. Further development in user interactions and surgical design is required to allow for pre-surgical planning functionality on our applications.

Keywords: Mis, Surgery Training, Hologram, Ar, Vr, Augmented Reality, Gesture Control

DOI: 10.54941/ahfe1004848

Cite this paper:

​