Wearable Biosignal and Pupillometry Analytics for Stress-Probe Evaluation in Mixed Reality Illegal Checkpoint Training

Abstract

Military first responders and civilian experts in international peace missions must make rapid, safety-critical decisions while managing conflict escalation under acute stress. Live field exercises such as the “Native Challenge” provide high realism but are costly and difficult to repeat systematically. The SmartSkills project addresses this gap by implementing Mixed Reality (MR) training with high-fidelity digital twins, standardized crisis scenarios, and a decision support concept that links human factors monitoring to instructor dashboards and debriefing.This work will demonstrate an extended SmartSkills analytics pipeline that focuses on physiological stress and emotion regulation during the Illegal Checkpoint MR scenario. Building on the SmartSkills concept of stress probes—standardized, time-locked stimuli placed at unavoidable key scenes to enable comparable measurements—we will analyze probe-evoked responses for stressors that are (a) artificially applied (e.g., scripted audiovisual cues, time pressure, unexpected instructions) and (b) executed by human agents (role players performing armed threat, separation, searching, and escalating communication).Participants of the pilot study were instrumented with unobtrusive sensing using smart biosignal shirts to capture complementary stress channels: (i) cardiovascular measures: ECG/PPG to estimate heart rate (HR) and heart rate variability (HRV), (ii) respiratory measures for breathing rate, and (iii) eye tracking–based pupillometry (MR headset embedding). Signals were synchronized with scenario events to quantify baseline-corrected, probe-evoked dynamics (peak reactivity, recovery slope, and habituation across repetitions). We will relate these markers to behavioral outcomes relevant to checkpoint management (e.g., de-escalation compliance, timing of key actions, and team coordination).Analytically, we will (1) compare stress signatures between artificial vs. human-enacted probes, (2) estimate individual differences in stress reactivity and emotion regulation (e.g., faster pupillary recovery and HRV rebound as putative indicators of effective regulation), and (3) provide the conceptual basis for developing interpretable multimodal models that map features to categorical stress/risk levels suitable for real-time visualization and after-action review within the SmartSkills decision support framework.At the conference presentation, we will demonstrate representative datasets and results, including event-related physiological traces aligned to stress probes, to illustrate how probe-based MR designs can support repeatable, data-driven evaluation and adaptive training for crisis operations.

Keywords: Wearables, Physiological Biomarkers, Stress, Mixed Reality, First Responder Training

DOI: 10.54941/ahfe1007377