AHFE Open AccessCognitive Computing and Internet of Things
Editors: Lucas Paletta
Topics: Cognitive Computing and Internet of Things
ISBN: 978-1-964867-81-6
DOI: 10.54941/ahfe1007224
Table of Contents
View2Decide: A Wearable Traffic-Light Display for Real-Time Physiological Decision Support in Military First Response
Emerging trends in soldier systems increasingly emphasize wearable sensing and real-time physiological status monitoring to enhance operational decision-making and situational awareness. Within the “Soldier of the Future” paradigm, individual physiological data has become a key enabler for both medical support and tactical assessment. While the Austrian Armed Forces’ VitalMonitor infrastructure allows continuous monitoring of vital parameters, existing visualization tools – such as smartphones, tablets, or laptops – are often impractical for first responders under high-stress or time-critical conditions. View2Decide addresses this gap by providing a compact, wearable display module that conveys critical physiological status through an intuitive traffic-light scheme, enabling rapid assessment of multiple individuals in the field. The system integrates modular hardware, BLE-based communication, and simplified status derivation to maintain robustness, low cognitive load, and operational flexibility. The proof-of-concept prototype demonstrates that an ESP32-based display, combined with a smartphone or future embedded communication nodes, can reliably present real-time physiological status in an easily interpretable format, including sequential assessment of multiple casualties. This paper presents the system architecture, user requirements, visualization logic, and prototype implementation, highlighting how device-agnostic, low-barrier status visualization can support frontline decision-making. View2Decide represents a promising step toward scalable, sensor-supported triage solutions for future soldier systems.
Florian Haid, Thomas Schnabel, Markus Bergen, Thomas Hölzl, Gerald Bauer
Open Access
Article
Conference Proceedings
Early Prediction of Physiological Strain Using Multivariate Time-Series Data
The increasing availability of wearable physiological sensors has enabled continuous monitoring of human performance in safety-critical and high-demand environments. Predicting physiological strain in advance provides valuable support for proactive decision-making and risk mitigation in sociotechnical systems. This study investigates short-term forecasting of physiological strain using multivariate wearable time-series data. The objective is to predict a bounded strain index several minutes in advance using heart rate (HR), respiratory rate (RR), and core body temperature (CBT) measurements. The dataset comprises continuous physiological recordings from 86 participants with approximately 734,000 time points, standardized to a sampling rate of 1 Hz. This work builds upon data collected in the RTVitalMonitor project, which develops predictive models for monitoring psychophysiological strain and performance in military task simulations. A multi-horizon forecasting framework predicts strain levels at 5, 10, and 15 minute intervals using sequential deep learning models. The methodological approach extends prior work conducted on a public graded exercise testing dataset, where exhaustion prediction was formulated as a binary classification task. In contrast, the present study reframes the problem as a regression-based forecasting task using a Gated Recurrent Unit (GRU) network, enabling continuous early warning of physiological strain. Models are evaluated using mean absolute error (MAE), root mean squared error (RMSE), and coefficient of determination (R2). Results demonstrate that the GRU model achieves robust predictive performance across all forecast horizons and remains stable under parameter-reduction scenarios. The findings highlight the potential of cognitive computing approaches for privacy-aware, real-time physiological monitoring in sociotechnical systems. The proposed framework contributes to the development of predictive human performance monitoring solutions suitable for wearable and edge-based deployment.
Mohammad Yusuf Mohammad Yunus Quadri, Julia Tschuden, Florian Haid, Anna Weber, Markus Bergen, Lucas Paletta, Michael Schneeberger, Gerald Bauer, Thomas Hölzl
Open Access
Article
Conference Proceedings
Real-time detection and machine learning classification of physical fatigue in construction workers using multi-modal digital biomarkers
This study introduces a multimodal wearable sensing framework that combines physiological (heart rate, breathing rate), thermoregulatory (skin temperature, electrodermal activity), and biomechanical (plantar pressure and acceleration) digital biomarkers for objective, real-time fatigue detection in construction workers. Fifteen healthy construction workers, aged 18 years or older, executed a standardized one-hour bar-bending tasks while utilizing wearable devices. Subjective fatigue levels were assessed using the Borg-20. Five supervised machine learning classifiers were assessed utilizing 10-fold cross-validation. In single-biomarker models, thermoregulatory features (Classification Accuracy [CA], 88.9%) surpassed physiological (CA, 72.3%) and biomechanical features (CA, 71.1%). The integration of physiological, thermoregulatory, and biomechanical parameters markedly enhanced classification accuracy (CA, 92.3%). The random forest is the most effective machine learning classifier for both single and multimodal biomarker data. This study constitutes the first direct comparison and integration of these three biomarkers specifically among construction workers, illustrating the evident superiority of multimodal methodologies. The suggested human-centred, AI-driven architecture effectively integrates medical physiology, ergonomics engineering, and wearable technology, providing a scalable, real-time early-warning system for fatigue-related hazards.
Shahnawaz Anwer, Maxwell Fordjour Antwi-afari, Imran Mehmood, Arnold Yu Lok Wong, Siu Ngor Fu, Heng Li
Open Access
Article
Conference Proceedings
Ergonomic Assessment of Lower-Limb Exoskeleton on Physiological Responses in Wildland Firefighters
Wildland firefighters are exposed to sustained physical demands that impose substantial physiological and perceptual strain. Lower-limb exoskeletons have been proposed as assistive devices to reduce internal workload. However, their effectiveness in this occupational context remains unclear. This study evaluated the effects of a hip-based lower-limb exoskeleton on physiological responses during a treadmill protocol simulating locomotor demands in wildland firefighting. Six wildland firefighters completed a 60-min walking protocol including level, uphill, and downhill stages while wearing full personal protective equipment and a 20 kg backpack. Two experimental conditions were compared in randomized crossover design: exoskeleton assistance activated (EXO) and exoskeleton worn without assistance (NO EXO). Oxygen uptake (VO2), heart rate (HR), and rating of perceived exertion (RPE) were recorded throughout the protocol. All variables changed significantly over time (p<0.001), confirming the physiological and perceptual demands of the task. However, no significant main effect of condition was found for VO2, HR or RPE, and no condition × time interactions were observed (p>0.05). Descriptive analyses suggested slightly lower VO2 (−5.6%) and HR (−2.5%) with exoskeleton assistance, alongside a marginally higher RPE (+4.7%). These findings indicate that, under the present experimental conditions, hip-based exoskeleton assistance did not produce clear reductions in physiological strain during simulated wildland firefighting locomotion. Further studies with larger samples, longer familiarization periods, and field-based protocols are needed to determine whether specific tasks or device configurations may improve effectiveness.
Belén Carballo-Leyenda, Sheila Vázquez-Vegas, Álvaro Regadío, Jose A Rodríguez-marroyo
Open Access
Article
Conference Proceedings
Integrating firefighters’ individual physical state in enhanced automated respiratory protection monitoring as decision-support: Influence on cognitive load in complex incident operations in a VR-Study
We developed an extended automated respiratory protection monitoring system integrating firefighters’ physical states to reduce cognitive load on incident commanders in complex firefighting operations. Background: During critical firefighting operations, decisions must be made in the presence of potentially stressful factors such as threats, lack of assessment criteria, or interruptions and disruptions. These are related to limited cognitive processing and memory capacities influencing the quality of information processing in decision-making. Method: In total, 63 incident commanders participated in an experimental VR-study with a 2x2 design, which varied the use of the extended automated respiratory protection monitoring system and incident complexity. Results: The results reveal a significant interaction between the extended automated respiratory protection monitoring system and the incident complexity of the operation on cognitive load (F (1, 56) = 5.69, p = .02, η2 = .09). While the monitoring system reduces cognitive load in operations with medium incident complexity, it increases cognitive load in operations with high incident complexity resulting from the accident of a team member. Conclusion: This study highlights the relevance of extended automated respiratory protection monitoring with its potential positive impacts on incident commanders’ cognitive load, while also emphasising a human factors orientated design of the information interface.
Clemens Koenczoel, Deborah Huber, Manfred Pollheimer-Stadlober, Alexander Elser, Jochen A Mosbacher, Martin Pszeida, Wolfgang Weiss, Sandra Pichler, Jasmina Schmidt, Klaus Tschabuschnig, Josef Rampitsch, Georg Schwarzott, Hannes Kern, Marie Ottilie Frenkel, Lucas Paletta, Dietrich Albert, Bettina Kubicek
Open Access
Article
Conference Proceedings
Conversational Co-Design with Machine Agency
Once Artificial Intelligence evolves into a truly intelligent system, capable of pursuing its own goals, learning through self-observation, and recursively adapting to context, the paradigm of interaction design must fundamentally shift. This paper explores a future where machines move beyond being simple tools to become active partners in co-creation. Rooted in Gordon Pask’s Conversation Theory, the proposed framework envisions a design process inclusive of machine-to-human conversations, where both entities negotiate goals and share systemic insights. By elevating the machine to a creative partner, the design process evolves from a human-centric exercise into a multilateral collaborative exchange. The paper argues that this inclusive approach to co-design is essential for ethical innovation and ensuring that future autonomous systems are built on a foundation of sustainable, equitable, and accessible design.
Rutuja Jog
Open Access
Article
Conference Proceedings
Investigating Mindfulness and Decision-Making under Stress Using Immersive Virtual Reality Firefighting Scenarios
First responders, including firefighters, must maintain focused attention and make high-quality decisions under time pressure, with incomplete information, and rapidly evolving hazards. Their actions directly impact the safety and survival of civilians and fellow emergency responders. The aim of this study was to examine how trait mindfulness moderates the relationship between decision-making style (deliberative versus intuitive) and state mindfulness among firefighting incident commanders in immersive virtual reality emergency scenarios. Based on Klein's Recognition-Primed Decision model of naturalistic decision-making and the dual-process theory, as well as the concept of mindfulness, we hypothesize that higher trait mindfulness is associated with higher state mindfulness across scenarios, independent of decision-making style (H1). Additionally, trait mindfulness is expected to moderate the relationship between decision-making style and state mindfulness during these scenarios (H2). Sixty firefighters completed two immersive virtual reality scenarios while their decision-making style, trait mindfulness, and state mindfulness were assessed. Linear moderation analysis revealed a significant main effect of trait mindfulness on state mindfulness, as well as a significant interaction between decision-making style in the scenarios and trait mindfulness. The results suggest that higher trait mindfulness is associated with greater state mindfulness, especially among deliberative decision makers. Additionally, they support the concept that mindfulness serves as a protective resource in firefighters. The results also extend naturalistic decision-making theories by demonstrating how trait mindfulness can reduce effects of distractions and enhance present-moment awareness. Further research is needed to investigate causal mechanisms, generalize the findings to other groups of first responders, and develop interventions, such as real-time biofeedback systems.
Alexander Elser, Clemens Koenczoel, Deborah Huber, Manfred Pollheimer-Stadlober, Hannes Kern, KATJA HUTTENBRENNER, Jochen A Mosbacher, Martin Pszeida, Wolfgang Weiss, Sandra Pichler, Jasmina Schmidt, Klaus Tschabuschnig, Josef Rampitsch, Georg Schwarzott, Lucas Paletta, Dietrich Albert, Bettina Kubicek, Marie Ottilie Frenkel
Open Access
Article
Conference Proceedings
Decision-Making in Emergency Response Organisations: Human Factors Challenges and Implications for Digital Support Systems
Emergency response organisations operate under conditions of extreme time pressure, uncertainty, and high stakes, where the quality of operational decisions directly affects the safety of responders and the public. Increasing technological complexity, novel hazard profiles such as alternative energy carriers and lithium-ion battery systems, and dynamic multi-actor environments significantly increase cognitive load and expose limitations of existing procedural and technological support mechanisms.Although digital Decision Support Systems (DSS) and advanced information and communication technologies are widely promoted as tools to improve operational decision-making, their practical adoption and effectiveness at the tactical level remain limited. One important reason for this gap may lie in the insufficient alignment of such systems with the Individual and contextual dynamics of real emergency response operations.This paper examines decision-making challenges in emergency response organisations from a Human Factors perspective and discusses implications for the design of digital decision support systems. The study builds on insights from the EMERDEC project, which investigates how tactical decisions are formed during the early and most critical phases of emergency response.Methodologically, EMERDEC combines participatory approaches, ethnographic field research, and controlled simulation studies in virtual and real-world environments. Advanced wearable sensor technologies and immersive simulations are used to capture psychophysiological indicators of stress, workload, and situational awareness.Based on these empirical insights, the paper identifies key Human Factors challenges for digital decision support and argues for a shift from technology-driven system development toward human-centred DSS design that aligns with the cognitive realities of emergency response operations.
Hannes Kern, KATJA HUTTENBRENNER, Clemens Koenczoel, Deborah Huber, Manfred Pollheimer-Stadlober, Alexander Elser, Jochen A Mosbacher, Martin Pszeida, Wolfgang Weiss, Sandra Pichler, Jasmina Schmidt, Klaus Tschabuschnig, Josef Rampitsch, Georg Schwarzott, Marie Ottilie Frenkel, Lucas Paletta, Dietrich Albert, Bettina Kubicek
Open Access
Article
Conference Proceedings
Mobile Platform for Integrated Data Capture in Immersive First Responder Training and Decision-Making
Immersive VR training is increasingly adopted by first-responder academies, yet objectively assessing decision-making under operational stress remains challenging. This is largely due to bio-signals, gaze data, and contextual video being captured with heterogeneous tools that produce inconsistent timestamps and offer limited robustness for field use.We present a mobile, deployment-ready platform that integrates psychophysiology, eye tracking, and synchronized video capture into a single workflow to support reproducible studies in real training environments, as piloted at the State Firefighting Academy Carinthia.Methods. The platform combines (i) immersive VR via Meta Quest 3 (2064×2208 per eye, up to 120 Hz) connected by cable for stable streaming, (ii) high-frequency eye tracking using a mounted Pupil Labs Neon module (two IR eye cameras, 192×192 @200 Hz; ~150 Hz when extracting gaze features), and (iii) multi-channel bio-signals captured through a BIOPAC MP160 with BioNomadix transmitters for ECG and EDA, configured in AcqKnowledge. In addition, a Polar H10 chest strap provides a redundant ECG source for cross-checking physiological recordings. All streams are consolidated in a custom “Study Recording Software/Study Controller” that supports connection setup and status monitoring, live signal visualization, anonymous user ID entry, standardized baseline recordings, manual start/stop of scenario recordings, and automatic folder structures for traceable data storage.To capture the context and enable behavioral annotation, two complementary video channels are recorded: (a) an external camera overview and (b) a first-person VR perspective recorded with Open Broadcaster Software, including an on-screen system timestamp for post-hoc alignment. Temporal synchronization is achieved using system timestamps embedded in received sensor streams, by continuously measuring device clock deviations relative to a common time server and complemented by event anchors in the ego videos.Scenario segmentation is automated by detecting these anchors and generating standardized clips; additional expert-defined anchors enable narrower excerpts (e.g., “arrival at incident scene”) for structured expert ratings.Results and discussion. In the pilot field deployment, the platform demonstrated stable multi-stream recording and consistent data packaging across bio-signals, gaze, and dual video. The synchronization and segmentation pipeline produced analysis-ready datasets and expert-review clips without disrupting the training operations. The approach increases robustness, reliability and trustworthiness through redundancy (dual ECG), explicit time-referencing, and standardized recording protocols. The next steps focus on scalable feature extraction from the synchronized exports (blink rate, saccade metrics, pupil diameter) and linking multimodal biomarkers to decision-quality indicators to support evidence-based debriefing and future adaptive assistance in immersive first-responder training.
Martin Pszeida, Michael Schneeberger, Wolfgang Weiss, Clemens Koenczoel, Alexander Elser, Klaus Tschabuschnig, Deborah Huber, Manfred Pollheimer-Stadlober, Sandra Pichler, Jasmina Schmidt, Josef Rampitsch, Georg Schwarzott, Dietrich Albert, Bettina Kubicek, Marie Ottilie Frenkel, Jochen A Mosbacher, Hannes Kern, Lucas Paletta
Open Access
Article
Conference Proceedings
Towards Fair Representation in AI-Mediated Decision-Making: A Conceptual Model for Socio-Technical Contexts
AI-mediated decision-making enhances human performance and efficiency in small and medium-sized enterprises (SMEs). However, compared with human decision-making, it raises concerns about human-centred principles such as transparency, fairness, and representation. In particular, formal worker representatives, such as works councils, who have relied on conventional oversight, often lack the technical entry points required to influence, oversee, or negotiate black-box AI-mediated decisions. This mismatch creates a representation gap that challenges the legitimacy of algorithmic outcomes and raises doubts about whether workers’ voices are adequately considered. Consequently, despite ongoing efforts to balance biased AI, it remains unclear how fair representation can be conceptualized and practically realized within digitally mediated decision-making. Building on this foundational challenge, the study asks how workers’ voices can shape AI-mediated decisions concerning the shop floor; how fair representation can be conceptually defined and embedded as a human-centred principle. To achieve this, the study first evaluates existing socio-technical and institutional attempts to embed human-centred principles into algorithmic decision-making, from social forms of representation, such as collective bargaining, to technical forms, such as Multi-Agent Systems (MAS). Secondly, it addresses the limitations of these existing approaches and comprehensively bridges the representation gap by developing a novel five-layered model, grounded in a case study and shop-floor insights. The model spans Level 0 (problem recognition) to Level 4 (AI-mediated decisions via MAS), integrating worker interests into AI-driven processes to reduce centralized, unilateral decision-making and ensure a substantive role for workers and their representatives. The study shows that bridging the representation gap requires more than algorithmic systems; the proposed framework highlights a deeper, trust-building form of worker involvement at the shop-floor level. Since AI is limited in fulfilling the social and legitimate representational functions of works councils, their participation at the proposed layers becomes essential. This interdependence forms a “nested representation”, integrating workers’ needs into AI tools to strengthen the human-centred foundations of AI-supported decisions in SMEs.
Noushin Qeybi, Khadija Sabiri, Gustavo Vieira
Open Access
Article
Conference Proceedings
Creating a Framework for the Collection of Biometric and Environmental Data During Collegiate Flight Training
The aviation industry has long recognized fatigue as a critical safety hazard, yet fatigue management strategies have predominantly focused on long-haul commercial operations (Olaganathan et al., 2021). This focus leaves significant gaps in General Aviation (GA) specifically the collegiate flight training sectors (Mendonca et al., 2021). The purpose of this ongoing research project is to investigate the psychophysiological experiences of flight students and certified flight instructors who operate within a rigorous academic environment that also includes high-stakes flight training. To advance our understanding of pilot performance, the researchers are collecting and analyzing high fidelity multimodal data. The current study employs a comprehensive sensor suite to capture the interaction between a pilot’s internal state and the physical environment of the training aircraft. The research utilizes biometric data such as heart rate (HR), electrocardiography (ECG), electrodermal activity (EDA), heart rate variability (HRV), and other data to monitor physiological responses. Concurrently, the team measures environmental factors such as noise, vibration, and temperature. This objective data is paired with subjective pre-flight and post-flight surveys to provide a more complete context for each training event. By integrating these distinct data streams, the research team can identify trends and patterns. The analysis further explores how physiological metrics fluctuate during various flight training activities. This paper will discuss how multimodal monitoring can be incorporated into flight training activities, lessons learned, and future research opportunities. Ultimately, the progression of this type of research will support the development of next-generation flight risk assessment, including Fatigue Risk Management Systems (FRMS) that are specifically tailored to the unique flying of collegiate aviation pilots.
Debra Henneberry, Mark Wilson, Sudip Vhaduri, Julius Keller, Dimitrios Ziakkas
Open Access
Article
Conference Proceedings
Augmented Memory and Attention in UI Interaction: NTDC as an Information-Theoretic Framework for Learning and Multitasking
Modern user interfaces require continuous learning, rapid attentional shifts, and the sustained appearance of multitasking, yet attention, memory, and learning are often modeled separately in HCI and cognitive theory. This paper presents the Networked Two-Dimensional Communication Channels (NTDC) framework as an information-theoretic model of UI-mediated interaction under uncertainty. Within NTDC, Shannon’s six entropy relations are instantiated as a directional Shannon-system and paired into bidirectional TDC nodes within a networked interaction model. Attention is modeled as decoding through mutual information, while memory is modeled as stabilized actionable interface entropy through the CAIO-UAIO partition or, under an alternative boundary specification, as part of the user’s cognitive state. Learning is represented as the progressive stabilization of previously uncertain actionable options across interaction states, and multitasking is interpreted as rapid sequential relocation of attention rather than parallel cognition. NTDC therefore offers a compact, boundary-relative analytical framework for examining informational alignment, interface scaffolding, and learning dynamics in complex UI systems.
Lance Chong
Open Access
Article
Conference Proceedings
Perceived Light Environment in Closed Space Based on EEG Analysis
In order to study the impact of enclosed space lighting environment on human psychological perception and work efficiency, an experimental environment was constructed using VR (virtual reality) technology, and subjective perception and EEG signal acquisition experiments were conducted on the lighting environment in the virtual environment. Four working conditions were established by combining different color temperatures (3000 K, 5000 K) and different illuminance levels (300 lx, 1000 lx). The results show that: (1) The comprehensive evaluation score of low color temperature scenes in a narrow-confined sleeping space was 90% higher than that of high color temperature scenes. (2) Illumination and color temperature in a narrow-confined sleeping space had a certain impact on EEG, and showed a significant negative correlation with relaxation coefficient.
Xiaohui Du, Shaowen Ding, Andi Wei, Xiaoyu Cao
Open Access
Article
Conference Proceedings
Distributed Cognition in Phygital Design Interventions: Community–System Communication in Marginalized Urban Contexts
Design interventions in marginalized urban communities involve complex interactions among people, material artifacts, and digital systems. In such settings, cognition and decision-making are not confined to individuals but are distributed across social, physical, and computational elements. This paper examines how communication between community members and a supporting system shapes collective sense-making during community-based design interventions. Drawing on comparative fieldwork in Campana-Altamira (Mexico) and Kampung Gedong Pompa (Indonesia), the study analyzes participatory workshops in which physical artifacts (e.g., maps, models, and visual notations) were combined with a lightweight digital system that recorded, organized, and re-presented community-generated knowledge. The paper introduces the concept of a Situated Knowledge Ecosystem (SKE) to describe this arrangement—not as a standalone technology, but as an emergent cognitive ecosystem situated between physical and digital interventions. Empirical observations show that communication between the community and the system played a critical role in distributing cognitive labor. Physical artifacts supported immediate discussion and negotiation, while the digital layer functioned as a persistent external memory that stabilized interpretations across time and participants. Rather than producing decisions directly, the system mediated how information was revisited, compared, and reflected upon, thereby influencing how collective judgments evolved. This paper highlights how cognitive ecosystems can support distributed cognition in socially complex environments. The findings suggest that the effectiveness of such systems lies not in computational intelligence, but in how they are embedded within ongoing human practices and material interactions.
Savira Aristi, Kenta Ono, Hisa Martinez Nimi
Open Access
Article
Conference Proceedings
Driver Cognitive, Emotional, and Behavioral Responses to Single-Day Highway Work Zones with Unexpected Lane Hazards
Single-day highway work zones introduce temporary lane closures, altered traffic flow, and constrained maneuvering space that may affect drivers’ cognitive, emotional, and behavioral states. Despite their prevalence, the human-factors implications of short-term work zones, especially when combined with unexpected hazards, remain underexplored. This study examines how typical single-day work-zone layouts and minor unexpected objects influence driver workload, emotional state, and driving behavior. A high-fidelity VR-based driving simulator was developed to replicate a three-lane highway with a temporary daytime maintenance work zone, including realistic signage, cone delineation, and ambient traffic. Thirty-five licensed drivers completed a familiarization drive followed by three conditions: Baseline highway driving, a Work-Zone condition with right-lane closure, and an Events condition in which unexpected objects (e.g., displaced cones) appeared in the travel lane. Measures included subjective workload (DALI), emotional state (SAM), and driving performance metrics. Results showed significant increases in mental and temporal demand, self-reported stress, and emotional arousal in the Events condition compared to Baseline and Work-Zone driving. Speed variability was also higher in both Work-Zone and Events conditions than in Baseline, indicating sustained changes in longitudinal speed control. These findings indicate that even brief single-day work zones can substantially elevate driver demand, and that minor unexpected hazards can markedly amplify cognitive and emotional strain, with implications for temporary traffic-control design and safety.
Paolo Pretto, Mateo Jukic, Manuela Prior, Inge Gsellmann, Thorsten Muik, Manfred Rosenberger
Open Access
Article
Conference Proceedings
Human–AI Co-Navigation for Indoor Object Search under Uncertainty
Assistive technologies for people with visual impairments increasingly use artificial intelligence to support object-finding and navigation in indoor environments. Yet fully autonomous perception remains unreliable in such settings, as indoor spaces are visually complex, only partially observable from the user’s current viewpoint, and subject to continuous change. Our work takes the position that effective assistive navigation is inherently collaborative; the system performs continuous perceptual processing, while the user provides occasional natural-language guidance when the search becomes uncertain or inefficient. To this end, we propose a human–AI collaboration framework that utilizes a Vision-Language Model (VLM) as the perceptual and semantic backbone of a navigation agent. A human user, modeled by a simulated intervention controller, provides sparse and structured guidance, which is integrated with the VLM to update its semantic search hypotheses toward the likely location of the target object. Evaluation is conducted in the Habitat simulator on photorealistic scenes from the Habitat-Matterport3D dataset. Experiments analyze how human guidance affects task success and navigation efficiency, showing that guidance is most effective when it corrects the VLM's misaligned semantic search hypotheses, providing insights into the role of minimal human input in VLM-based assistive navigation systems.
Ahmed Ghita, Qiuyi Cao, Daniel Watzenig, Stefan Ehrlich
Open Access
Article
Conference Proceedings
Physiological precursors that precede the awareness of cognitive stress
In collaboration, early detection of partner’s dissatisfaction is important to keep communication in good shape. Such early detection is also essential for AI systems and agents that interact with humans. This paper examined whether physiological changes associated with cognitive stress precede the conscious experience of dissatisfaction during consensus building dialogue. Twenty male undergraduate participants engaged in controlled one-on-one prioritization tasks based on a snow-mountain rescue scenario. Participants continuously reported dissatisfaction in real time using a visual analogue scale (VAS). Physiological data were temporally aligned to peaks of subjective dissatisfaction. The results showed that subtle but sustained decreases in nasal skin temperature frequently occurred prior to subjective dissatisfaction peaks. Nasal skin temperature may serve as an early autonomic marker of emerging cognitive stress, whereas heart rate may reflect later or more intense stress responses. By elucidating the temporal dynamics between physiological responses and subjective stress, this study provides foundational insights for cognitive computing in sociotechnical systems and supports the design of adaptive dialogue support frameworks.
Takuya Endo, Toru nakata, Toshikazu Kato
Open Access
Article
Conference Proceedings
Human Cognitive Processing Strategies in the Detection of AI-Generated Synthetic Media
Deepfake technology generated by artificial intelligence (AI) is becoming increasingly realistic. This technology not only challenges people’s ability to judge what they see but may also influence individual thought patterns. Current deception research generally holds that applying cognitive load to deceivers can prompt them to reveal more deception cues, without considering the cognitive impact of deception detection methods on observers. Addressing this gap, this study moves beyond the limitations of focusing solely on media content or detection mechanisms by examining how individuals perceive and interpret signs of manipulation when evaluating deepfake material, particularly their attention and cognitive processing strategies in multimedia deepfake detection tasks. Participants were asked to assess whether textual, image, and video materials are authentic or fake. The results showed that participants used different strategies in allocating perceptual and cognitive resources across the three media. Text and image materials required longer reaction times and led to more extreme judgments, indicating that they involved more effortful processing and cautious deliberation. Video materials, on the other hand, induced fewer extreme judgments, implying that dynamic cues help individuals detect anomalies intuitively. Although video information is commonly perceived as the most challenging media format to authenticate, participants were able to determine its authenticity more quickly than in other tasks. These findings highlight the importance of considering observer cognitive load in deception detection and offer theoretical implications for integrating cognitive load theory with dual process models of judgment in HCI contexts.
Yue Liu
Open Access
Article
Conference Proceedings
Stress and Recovery Signatures from Wearable Biosignal Data in the Production Environment
Production work involves time pressure, variable workload, and shift schedules that may contribute to sustained psychophysiological strain. Wearable sensing can support continuous, low-burden monitoring of stress and recovery and may inform resilience-oriented workplace interventions. This pilot study examined stress and recovery signatures in an industrial electronics production context by combining multi-day wearable biosignal tracking with validated self-assessments. Sixteen employees, predominantly shopfloor operators, participated in a one-week field study across four shifts, including two day shifts and two night shifts. The sample included 8 women and 8 men (M age = 47.3 years, SD = 10.7). Participants wore a biosignal tracker during work, leisure time, and sleep. Measures included heart rate, heart rate variability[DS1.1][l1.2] (HRV), baseline calibration during sitting and standing, and questionnaires assessing resilience, perceived stress, well-being, affect, and short-term recovery-stress states. Analytics of wearable data focused on shift-level HRV changes and rule-based bouncing-back features, including stress peaks, peak amplitude, and recovery time. Results indicated mostly normal-to-high resilience, low-to-normal perceived stress, and generally preserved well-being, although some participants showed reduced well-being and high perceived stress. The short version of Perceived Stress Scale (PSS-4) total scores were negatively associated with HRV, and day shift analyses linked decreasing HRV to increasing mental strain. Additionally, the rate of stress peaks that caused long-term recovery periods was positively associated with the change in short-term physical-strain scores. The findings support the feasibility of wearable biosignal analytics for exploratory stress and recovery assessment in production work.
Lucas Paletta, Wolfgang Weiss, Ciprian Alexandrescu, Diana Nastase, Michael Schneeberger, Martin Pszeida, Sandra Draxler, Jochen A Mosbacher, Herwig Zeiner
Open Access
Article
Conference Proceedings
Emotional Regulation of Adults with high-functioning ASD Using Pupillometry from Real and Artificial Stimuli
Social interaction deficits are a core characteristic of Autism Spectrum Disorder (ASD) and are often linked to atypical attentional processing of socially relevant cues (Baron-Cohen, 1995). Understanding these mechanisms is essential for developing sensor-based learning analytics in serious games. Eye movement analysis provides a non-invasive and cost-effective approach to derive digital biomarkers for ASD (Frazier et al., 2018). In this study, we examined pupillometry features at rest and during emotion processing, building on earlier findings that pupil dynamics reflect autonomic and cognitive function in ASD (Anderson et al., 2013; Pszeida et al., 2025). We specifically tested whether video-based virtual stimuli elicit pupillometric responses comparable to real human faces.Participants included N = 20 neurodiverse (ND) adults with ASD (M = 26.20, SD = 4.78; 50% female) and N = 20 neurotypical (NT) controls (M = 23.71, SD = 2.93; 33% female). All participants completed a standardized psychological test battery and the digital Emotion Evaluation and Regulation Test (EERT), which presented video stimuli of six emotions (joy, sadness, fear, disgust, anger, neutral). Stimuli were drawn from (i) the validated FACES database (Ebner et al., 2010) and (ii) artificially generated virtual character faces (Poglitsch et al., 2024). ASD classification was supported by the RAADS-R screening tool (Ritvo et al., 2011), with the ND group showing markedly elevated scores (ND: M = 128.72, SD = 38.36; NT: M = 17.24, SD = 12.47). Cognitive profiles indicated a high-functioning ASD subgroup based on CFT-20R fluid intelligence scores (Weiß, 2008; ND: M = 116.95, SD = 11.22; NT: M = 114.48, SD = 13.05).Participants viewed four individual actors expressing all six emotions in real and virtual face videos, each lasting 2 seconds. The first 2 seconds following stimulus onset were analyzed relative to a 500 ms baseline period. Pupillometric features included peak dilation latency, dilation amplitude, and time-to-recovery.For emotion ‘anger’ expressed by real faces, ND participants showed a delayed time-to-peak dilation of approximately 200 ms, increased dilation amplitude by ~0.1 mm, and prolonged recovery time by ~600 ms compared to NT individuals. Neutral real faces produced group differences only in peak latency (~200 ms). Virtual anger stimuli yielded comparable effects: ND participants again demonstrated delayed peak latency (~200 ms), larger dilation amplitude (~0.1 mm), and delayed recovery (~500 ms).These results indicate that both real and virtual emotional face videos elicit distinct pupillometric signatures in high-functioning ASD. Latency to peak dilation, amplitude increase, and delayed recovery consistently differentiated ND from NT participants. The findings suggest that pupil dynamics may serve as reliable biomarkers of emotion regulation and attentional processing in ASD and highlight the feasibility of using virtual stimuli for scalable assessment in digital and game-based environments.
Martin Pszeida, Yannick Lieb, Wolfgang Weiss, Christian Poglitsch, Jördis Rausch, Manon Mannherz, Jochen A Mosbacher, Johanna Pirker, Knut Möller, Ludger Tebartz van Elst, Lucas Paletta
Open Access
Article
Conference Proceedings
Designing for the Six Human Needs: A Behaviour-Centred Framework for Emotional Connection in Physical Spaces
The conditions surrounding human behaviour are shifting in ways that directly affect how people experience interior spaces. This paper introduces a practical framework that adapts the Six Human Needs model for interior design and links it with research from environmental psychology and motivational theory. The Design Behaviour Interaction Model describes how human needs may shape emotional states, behavioural patterns, and the way people interpret physical environments.Methods combine thematic analysis of more than twenty design briefs, alongside structured observational analysis of user behaviour conducted over a two-year period across hospitality, workplace, and transitional environments (including cafés, restaurants, offices, and waiting areas), supported by a targeted review of psychological and acoustic literature.A case study of Penny Black Jazz Club illustrates how lighting, acoustics, materiality, proximity, and cultural cues can contribute to emotional clarity and strengthen connection, significance, and social ease. Observed patterns suggest increased focus, social warmth, and memory retention associated with specific environmental conditions.The paper argues that behaviour-centred design can reduce misalignment between designer and client and support the creation of environments that remain emotionally meaningful across generations.
David Macphee
Open Access
Article
Conference Proceedings
The SmartResponse Framework: A Semantic Digital Twin Approach to Remote Emergency Command
Remote incident commanders must coordinate disaster response without being physically present, while time pressure, incomplete information, and limited expert availability increase the risk. A novel framework – SmartResponse - develops the basis for future online command by integrating real-time 3D data acquisition, semantically enriched digital twins, and immersive, biosignal-assisted XR interaction for mission planning and communication with teams on site. This paper discusses the potential for digital twins to improve infrastructure management and remote emergency command. The workflow combines deployable sensing technologies, including LiDAR, calibrated cameras, inertial measurement, on-board processing, and broadband transmission, to stream 3D point clouds from an incident area to an operations centre. There, the scene is incrementally reconstructed into a semantically enriched digital twin that can be iteratively updated and augmented with hazard information, structural elements, relevant objects, and deviations from reference data. The resulting layer supports research on faster 3D situational awareness, hazard zoning, safer route planning, improved resource allocation, and more intuitive communication for distributed teams. A challenging dimension is the integration of emergency-critical human factors for system design and biosignal-based evaluation, including assessment under real emergency mission conditions.
Lucas Paletta, Georg Aumayr, Andreas Peer, Christian Perktold, Wolfgang Kallus, Patrick Luley, Stefan Ladstätter, Anna Weber, Sandro Eitzinger, Christian Schönauer, Friedrich Urbanek, Martin Söllner, Michael Schneeberger, Martin Pszeida, Christian Meurers
Open Access
Article
Conference Proceedings
Wearable Biosignal and Pupillometry Analytics for Stress-Probe Evaluation in Mixed Reality Illegal Checkpoint Training
Military first responders and civilian experts in international peace missions must make rapid, safety-critical decisions while managing conflict escalation under acute stress. Live field exercises such as the “Native Challenge” provide high realism but are costly and difficult to repeat systematically. The SmartSkills project addresses this gap by implementing Mixed Reality (MR) training with high-fidelity digital twins, standardized crisis scenarios, and a decision support concept that links human factors monitoring to instructor dashboards and debriefing.This work will demonstrate an extended SmartSkills analytics pipeline that focuses on physiological stress and emotion regulation during the Illegal Checkpoint MR scenario. Building on the SmartSkills concept of stress probes—standardized, time-locked stimuli placed at unavoidable key scenes to enable comparable measurements—we will analyze probe-evoked responses for stressors that are (a) artificially applied (e.g., scripted audiovisual cues, time pressure, unexpected instructions) and (b) executed by human agents (role players performing armed threat, separation, searching, and escalating communication).Participants of the pilot study were instrumented with unobtrusive sensing using smart biosignal shirts to capture complementary stress channels: (i) cardiovascular measures: ECG/PPG to estimate heart rate (HR) and heart rate variability (HRV), (ii) respiratory measures for breathing rate, and (iii) eye tracking–based pupillometry (MR headset embedding). Signals were synchronized with scenario events to quantify baseline-corrected, probe-evoked dynamics (peak reactivity, recovery slope, and habituation across repetitions). We will relate these markers to behavioral outcomes relevant to checkpoint management (e.g., de-escalation compliance, timing of key actions, and team coordination).Analytically, we will (1) compare stress signatures between artificial vs. human-enacted probes, (2) estimate individual differences in stress reactivity and emotion regulation (e.g., faster pupillary recovery and HRV rebound as putative indicators of effective regulation), and (3) provide the conceptual basis for developing interpretable multimodal models that map features to categorical stress/risk levels suitable for real-time visualization and after-action review within the SmartSkills decision support framework.At the conference presentation, we will demonstrate representative datasets and results, including event-related physiological traces aligned to stress probes, to illustrate how probe-based MR designs can support repeatable, data-driven evaluation and adaptive training for crisis operations.
Michael Schneeberger, Andreas Peer, Georg Aumayr, Martin Pszeida, Wolfgang Weiss, Jochen A Mosbacher, Florian Haid, Patrick Luley, Anna Weber, Stefan Ladstätter, Joachim Brandtner, Christian Schönauer, Benjamin Schuster, Martin Söllner, K Wolfgang Kallus, Martin Müller, Markus Öttl, Gudrun Walter, Lucas Paletta
Open Access
Article
Conference Proceedings
Rapid Military Triage of Traumatic Brain Injury Using Eye Tracking and Pupillometry
Rapid triage in military operations must reliably identify “hidden” neurological impairment among soldiers who appear only slightly wounded. Mild to moderate traumatic brain injury (TBI), concussion, and neurotoxic exposures can present with subtle early signs that are easily missed under stress, time pressure, and resource constraints. Eye tracking and pupillometry are promising for early triage because they probe brainstem and oculomotor function, can be captured quickly with deployable devices, and provide quantitative, repeatable measures suitable for decision support. This paper proposes a research program to develop an eye tracking–based, explainable rule base with operational thresholds to support early military triage. The goal is a field-ready decision logic that estimates the likelihood of clinically relevant impairment (e.g., concussion/moderate TBI, severe TBI warning signs, or neurotoxic syndromes) in “walking wounded” soldiers and outputs a transparent triage recommendation (e.g., high / medium / low risk), aligned with medic workflows. Methodology. We will implement a rapid assessment protocol combining (1) pupillary light reflex metrics, (2) basic oculomotor screening (e.g., gaze stability, smooth pursuit, saccadic control), and (3) short mobile cognitive-oculomotor tasks that can be completed in minutes. Candidate rules will be derived from a structured synthesis of prior clinical and applied evidence and formalized into interpretable “if–then” statements with thresholds and confidence flags. The rule base will be designed with explicit handling of confounds relevant to deployment (illumination variability, fatigue, stress, motion, and partial occlusion), including quality checks and fallback logic when signals are unreliable. Expected results. The project will deliver (i) a prototype triage tool integrating eye tracking data capture and automated rule evaluation, (ii) an initial threshold library for the most informative pupil and oculomotor features, and (iii) a validation plan and performance targets emphasizing sensitivity to clinically relevant TBI risk while maintaining practical specificity for operational use. We will prioritize explainability (rules that medics can interpret) and actionability (clear next steps such as observation, repeat testing, or expedited evacuation).Future work will include controlled studies for normative baselines, simulated field trials, and prospective evaluations in training environments, with iterative refinement toward robust deployment and integration with other physiological measures where beneficial.
Michael Schneeberger, Martin Pszeida, Dietmar Maurer, Lucas Paletta
Open Access
Article
Conference Proceedings
Mixed Reality-supported Training of First Responder Skills in International Crisis Situations: Evaluation of the SmartSkills Pilot Study
Military first responders and civilian experts in international peace missions must make rapid, safety-critical decisions while managing conflict escalation under acute stress. Live field exercises such as the biannual “Native Challenge” provide high realism but are costly, resource-intensive, and difficult to repeat in a standardized way. The Austrian SmartSkills project addresses this gap by implementing Mixed Reality (MR) training that combines repeatable crisis scenarios with an instructor-facing decision support concept and human factors monitoring to strengthen learning and debriefing quality.A central contribution of SmartSkills is the meaningful integration of digital twins into MR training—not as a static background, but as a functional component that anchors perception, navigation, cover/line-of-sight, and object interaction in a spatially correct replica of real mission environments. High-fidelity digital twins are generated via mobile mapping (e.g., drone/backpack/vehicle laser scanning) and processed into point clouds and textured 3D models. These assets are then optimized for real-time MR (balancing level-of-detail and data volume) and enriched with annotations to support scenario authoring and the consistent placement of interactive objects, hazards, and “checkpoint” events across repetitions.We report the design and evaluation plan of the SmartSkills pilot study in a simulation-center setting using the Illegal Checkpoint scenario, including a direct comparison between MR-supported training and a conventional real-world simulated training setup. Trainees must apply de-escalation, compliance, and team coordination procedures while being confronted with escalating threats. To enable comparable measurements across runs and between modalities, the scenario uses stress probes—standardized, time-locked stimuli embedded at unavoidable key scenes—implemented equivalently in both the MR and real simulated conditions.Participants are instrumented with unobtrusive sensing to capture complementary psychophysiological channels relevant to stress, emotion regulation, and situation awareness. Signals are synchronized with scenario events to relate these markers to behavioral and subjective outcomes. The pilot study thereby establishes an empirical basis for repeatable MR training that couples digital-twin realism with measurable human-factors outcomes, while also quantifying how MR compares to real simulated training in terms of perceived workload, stress response patterns, and training acceptance.At the conference presentation, we will demonstrate representative datasets and comparative results across MR and real simulated training, including questionnaire outcomes, qualitative feedback from trainees and instructors, and an overview of bio-signal sensor analytics (stress-analytical measures) derived from synchronized scenario events and stress probes.
Georg Aumayr, Andreas Peer, Benjamin Schuster, Martin Müller, Christian Schönauer, Thomas Seirlehner, Sabrina Hackl, Michael Schneeberger, Martin Pszeida, Wolfgang Weiss, Jochen A Mosbacher, Patrick Luley, Anna Weber, Stefan Ladstätter, Joachim Brandtner, Florian Haid, Markus Bergen, Markus Öttl, Friedrich Urbanek, Gudrun Walter, Wolfgang Kallus, Lucas Paletta
Open Access
Article
Conference Proceedings