Mixed Reality as a tool for enhancing precision in surgery planning

Abstract: Mixed reality (MR) is an emerging technology that combines features of augmented reality (AR) and virtual reality (VR) by overlaying virtual elements onto a natural environment. This fusion of the real with the digital allows users to interact naturally and intuitively with the various aspects, making MR a valuable tool for its application in different fields, including the clinical field. This work aims to present a working methodology for the application of Mixed Reality in different surgical specialities, showcasing scenarios generated for its use in the surgical fields of trauma and vascular surgery. By superimposing images and using 3D anatomical models over the surgeon's field of vision, the aim is to support the surgeon's movement guidance in complex procedures or areas that are difficult to visualise. To achieve this, the development process of the MR scenarios is detailed: firstly, the work of the medical image and extraction of the anatomical models using Materialise’s Mimics software, followed by the importation into the Unity engine for the design and positioning of virtual elements to be displayed, and finally the visualisation and design of the interactions with the digital environment through the use of different devices (tablets, smartphones, headsets). In addition to combining 3D anatomical models with information from the DICOM file of the medical image, the working methodology presented also details the work carried out for the positioning of guiding elements, such as vectors, angles, trajectories or other geometric elements that aid in guiding the surgeon’s movements, using devices that allow professionals to keep their hands free. All of this aims to show a possible use of Mixed Reality by offering greater immersion through anatomical models that faithfully represent the patient's anatomy and the ability to interact with them in real-time, making it technological support for the clinician's training for diagnosis and surgical planning, improving anatomical understanding in complex cases. The potential and application of MR in various surgical fields, especially in surgical planning, could significantly transform medical practice by allowing greater personalization of interventions, optimising precision for better clinical outcomes, and saving time in the operating room by increasing the efficiency and safety of the surgical procedure.

Keywords: Mixed Reality (MR), Surgical Planning, 3D Anatomical Model, EVAR, Arthrodesis, Transpedicular Screw, Mimics, Unity, MRTK, HoloLens2.

DOI: 10.54941/ahfe1005936

Cite this paper:

​