Mobile Platform for Integrated Data Capture in Immersive First Responder Training and Decision-Making

Abstract

Immersive VR training is increasingly adopted by first-responder academies, yet objectively assessing decision-making under operational stress remains challenging. This is largely due to bio-signals, gaze data, and contextual video being captured with heterogeneous tools that produce inconsistent timestamps and offer limited robustness for field use.We present a mobile, deployment-ready platform that integrates psychophysiology, eye tracking, and synchronized video capture into a single workflow to support reproducible studies in real training environments, as piloted at the State Firefighting Academy Carinthia.Methods. The platform combines (i) immersive VR via Meta Quest 3 (2064×2208 per eye, up to 120 Hz) connected by cable for stable streaming, (ii) high-frequency eye tracking using a mounted Pupil Labs Neon module (two IR eye cameras, 192×192 @200 Hz; ~150 Hz when extracting gaze features), and (iii) multi-channel bio-signals captured through a BIOPAC MP160 with BioNomadix transmitters for ECG and EDA, configured in AcqKnowledge. In addition, a Polar H10 chest strap provides a redundant ECG source for cross-checking physiological recordings. All streams are consolidated in a custom “Study Recording Software/Study Controller” that supports connection setup and status monitoring, live signal visualization, anonymous user ID entry, standardized baseline recordings, manual start/stop of scenario recordings, and automatic folder structures for traceable data storage.To capture the context and enable behavioral annotation, two complementary video channels are recorded: (a) an external camera overview and (b) a first-person VR perspective recorded with Open Broadcaster Software, including an on-screen system timestamp for post-hoc alignment. Temporal synchronization is achieved using system timestamps embedded in received sensor streams, by continuously measuring device clock deviations relative to a common time server and complemented by event anchors in the ego videos.Scenario segmentation is automated by detecting these anchors and generating standardized clips; additional expert-defined anchors enable narrower excerpts (e.g., “arrival at incident scene”) for structured expert ratings.Results and discussion. In the pilot field deployment, the platform demonstrated stable multi-stream recording and consistent data packaging across bio-signals, gaze, and dual video. The synchronization and segmentation pipeline produced analysis-ready datasets and expert-review clips without disrupting the training operations. The approach increases robustness, reliability and trustworthiness through redundancy (dual ECG), explicit time-referencing, and standardized recording protocols. The next steps focus on scalable feature extraction from the synchronized exports (blink rate, saccade metrics, pupil diameter) and linking multimodal biomarkers to decision-quality indicators to support evidence-based debriefing and future adaptive assistance in immersive first-responder training.

Keywords: VR training, first responders, decision-making, bio-signals, eye tracking, temporal data synchronization, distributed systems

DOI: 10.54941/ahfe1007928