Mixed Reality-supported Training of First Responder Skills in International Crisis Situations: Evaluation of the SmartSkills Pilot Study

Abstract

Military first responders and civilian experts in international peace missions must make rapid, safety-critical decisions while managing conflict escalation under acute stress. Live field exercises such as the biannual “Native Challenge” provide high realism but are costly, resource-intensive, and difficult to repeat in a standardized way. The Austrian SmartSkills project addresses this gap by implementing Mixed Reality (MR) training that combines repeatable crisis scenarios with an instructor-facing decision support concept and human factors monitoring to strengthen learning and debriefing quality.A central contribution of SmartSkills is the meaningful integration of digital twins into MR training—not as a static background, but as a functional component that anchors perception, navigation, cover/line-of-sight, and object interaction in a spatially correct replica of real mission environments. High-fidelity digital twins are generated via mobile mapping (e.g., drone/backpack/vehicle laser scanning) and processed into point clouds and textured 3D models. These assets are then optimized for real-time MR (balancing level-of-detail and data volume) and enriched with annotations to support scenario authoring and the consistent placement of interactive objects, hazards, and “checkpoint” events across repetitions.We report the design and evaluation plan of the SmartSkills pilot study in a simulation-center setting using the Illegal Checkpoint scenario, including a direct comparison between MR-supported training and a conventional real-world simulated training setup. Trainees must apply de-escalation, compliance, and team coordination procedures while being confronted with escalating threats. To enable comparable measurements across runs and between modalities, the scenario uses stress probes—standardized, time-locked stimuli embedded at unavoidable key scenes—implemented equivalently in both the MR and real simulated conditions.Participants are instrumented with unobtrusive sensing to capture complementary psychophysiological channels relevant to stress, emotion regulation, and situation awareness. Signals are synchronized with scenario events to relate these markers to behavioral and subjective outcomes. The pilot study thereby establishes an empirical basis for repeatable MR training that couples digital-twin realism with measurable human-factors outcomes, while also quantifying how MR compares to real simulated training in terms of perceived workload, stress response patterns, and training acceptance.At the conference presentation, we will demonstrate representative datasets and comparative results across MR and real simulated training, including questionnaire outcomes, qualitative feedback from trainees and instructors, and an overview of bio-signal sensor analytics (stress-analytical measures) derived from synchronized scenario events and stress probes.

Keywords: Skill Training, Mixed Reality, Conflict Scenarios, Bio-signal Sensors, pilot study

DOI: 10.54941/ahfe1008069