AHFE Open AccessHuman Factors in Robots, Drones and Unmanned Systems
Editors: Alexandra Medina-Borja, Krystyna Gielo-Perczak
Topics: Robots, Drones and Unmanned Systems
ISBN: 978-1-964867-94-6
DOI: 10.54941/ahfe1007237
Table of Contents
Workshop: Orchestrating Synthesized Human and AI-Agentic Workflows: AI Agency Benefits, Disruptions and Management
For our Special Interest Group (S.I.G.), we propose papers that address the orchestration of teams by synthesizing their workflows into a coherent whole, whether these teams are composed of human, machine, Generative AI (gen-AI), robot or AI-Agentic members. The bigger picture of interdependence, teamwork and Gen-AI indicates the need by organizations to build a library of human and artificial agents with bidirectional agency (responsibility) to achieve operational goals (missions), considering agentic risk tolerances, available skills, and vulnerabilities across a complex trade space among the skills available versus those needed for the tasks assigned to complete an operation. In this trade space, agents (human or artificial) from multiple systems with the requisite skills to accomplish a designated task and timeline combined to form a hierarchy of humans, robots, machines and AI. This complex system produces workflows that must be synthesized into a unit(s), then orchestrated to accomplish the goals assigned to it, yet remain trusted even in competitive and uncertain environments. Once synthesized into a unit (e.g., a team), Gen-AI provides the opportunity to not only advance the science of teams by orchestrating team products and performances, but also has raised several concerns (viz., AI used for deception, superintelligence, blackmail, or existential threats to humans). For our S.I.G., We are interested in orchestrating teams: What are the benefits, drawbacks, and, most importantly, can humans, machines and Agentic AI be synthesized and managed (orchestrated)?
William Lawless, Marco Brambilla, Stephen Russell
Open Access
Article
Conference Proceedings
The Risks, Challenges, and Potential Opportunities with GenAI
Artificial intelligence (AI) is a field where the masses offer declarations about novel advancements to machine intelligence and the everyday person feels like an AI “expert” in Generative AI (GenAI), such as ChatGPT and DALL-E. While 80+ years of research has led to the potential for GenAI to create new, “original” content, the public ought to understand that GenAI’s abilities are predicated on processing massive datasets. These datasets have many potential risks, including overtraining or novel datasets, foundational data science and metadata to AI models to cause incorrect decisions, bypass security, or extract sensitive information. Effective, trusted teaming with AI-agentic teams remains a critical research and development objective. Further, AI effectiveness becomes irrelevant if a human does not understand or trust the AI. This paper provides the foundations of AI and risks of GenAI, followed by a Use Case example of data management from a sensor edge node through actionable intelligence describing AI. This Use Case will walk through a data science strategy underpinning AI for enhancing trusted AI-agentic teaming, outlining the scientific research, challenges, and risks that can occur at each step that can directly impact the trusted relationship.
Kristin Schaefer, John Tomaselli, Larry Parrotte, Brandon Taylor, Antonio Magana, Henry Reimert, Selena Hamilton, Maggie Wignmess, Daniel Cassenti
Open Access
Article
Conference Proceedings
Agentic LLMs for Scalable, Verifiable System Health Digital Twins
System Health Management (SHM) digital twins have evolved from specialized engineering tools into enterprise-wide critical systems supporting diagnostics and lifecycle decision support, yet scaling the creation, validation, and maintenance of detailed causal models remains a bottleneck due to labor-intensive, expert-driven processes that do not scale with system complexity or lifecycle evolution. This paper presents an AI-driven framework addressing this challenge through a tightly integrated neuro-symbolic architecture that combines agentic large language models (LLMs) as constrained knowledge extraction agents with a rigorous symbolic reasoning core grounded in multi-functional causal modelling, enforcing structural, semantic, and logical constraints to transform extracted knowledge into verifiable, executable diagnostic models while shifting human expertise toward validation, governance, and continuous improvement. The framework implements an end-to-end “ingest–extract–structure–verify” pipeline converting artifacts (i.e., technical manuals, schematics, FMECA data) into formal causal models compatible with TEAMS and SysML-based representations, providing a single source of truth for downstream applications including fault detection and isolation, prognostics, sensor optimization, training scenario generation, and lifecycle-informed design. Demonstrated results show up to an 80% reduction in engineering effort and rapid model generation at previously impractical scales, with aerospace and space system deployments confirming accurate, scalable operational reasoning, while an enterprise operating model treats the digital twin as a governed, evolving asset integrated across design, operations, maintenance, and training, enabling continuous adaptation from field data and offering a practical path to trustworthy, adaptive digital twins that deliver sustained enterprise-scale value.
Chris Norton, Krishna Pattipati, Jordan Thurston, Deepak Haste, Sudipto Ghoshal, Somnath Deb, William Lawless
Open Access
Article
Conference Proceedings
When Workflows Stay Stable but Meaning Moves in Agentic Analyst Pipelines
Agentic analyst pipelines can keep a workflow object stable while its operational meaning shifts underneath. A coded event, alert, or dashboard tile may persist unchanged while the reporting frame moves from accident to attack, or from outage to state-linked operation. This semantic movement can reach the analyst undetected, degrading situation awareness while the interface appears organized. This research proposes a measurement and governance layer for detecting semantic frame shifts in agentic analyst workflows, using 1032 headline and snippet records organized into 37 adjudicated event-window states across 10 event families. Three findings bear directly on human-agentic workflow design. First, human coders outperformed all tested LLM coding agents on a binary frame-shift endpoint, with the best LLM condition reaching kappa 0.549 against adjudication compared to kappa 0.784 for human coders, confirming that human semantic judgment is a more reliable reference in analyst triage workflows. Second, our proposed embedding-based metrics outperformed every LLM condition as computational detectors: translational drift reached AUROC 0.894, neighborhood rewiring AUROC 0.841, and a combined reframing-pressure score AUROC 0.909, classifying workflow states into stable interpretation, smooth directional drift, semantic churn, and major reframing. Third, LLM coding agents varied materially by model and prompt policy, paralleling the variation observed across human coders, which means that consensus among human judgment, semantic metrics, and LLM signals cannot be assumed and must be treated as an active governance variable in workflow design. Together these findings support an orchestration layer that routes high-reframing or high-disagreement cases to human review while allowing semantically stable outputs to pass forward. As demonstration, the proposed metrics and approach are implemented in an open source intelligence system.
Stephen Russell
Open Access
Article
Conference Proceedings
Effects of Swarm Size Variability on Operator Workload
Real-world deployments of human-swarm teams depend on balancing operator workload to leverage human strengths without inducing overload. A key challenge is that swarm size is often dynamic: robots may join or leave the mission due to failures or redeployment, causing abrupt workload fluctuations. Understanding how such changes affect human workload and performance is critical for robust human--swarm interaction design. This paper investigates how the magnitude and direction of changes in swarm size influence operator workload. Drawing on the concept of workload history, we test three hypotheses: (1) workload remains elevated following decreases in swarm size, (2) small increases are more manageable than large jumps, and (3) sufficiently large changes override these effects by inducing a cognitive reset. We conducted two studies (N = 34) using a monitoring task with simulated drone swarms of varying sizes. By varying the swarm size between episodes, we measured perceived workload relative to swarm size changes. Results show that objective performance is largely unaffected by small changes in swarm size, while subjective workload is sensitive to both change direction and magnitude. Small increases preserve lower workload, whereas small decreases leave workload elevated, indicating workload residue; large changes in either direction attenuate these effects, suggesting a reset response. These findings offer actionable guidance for managing swarm-size transitions to support operator workload in dynamic human--swarm systems.
William Hunt, Aleksandra Landowska, Horia Maior, Sarvapali Ramchurn, Mohammad Soorati
Open Access
Article
Conference Proceedings
A Computer-Vision Approach to Accessible Robot Control: Hand Gesture Recognition for Users With Limited Mobility or Speech
Human–robot interaction increasingly demands intuitive, efficient, and accessible control mechanisms, particularly for users with physical or communication disabilities. Traditional interfaces—such as joysticks, keyboards, or voice commands—often impose significant cognitive or physical effort and may be unusable for individuals with impaired speech, hearing, or motor abilities. Recent advances in artificial intelligence and computer vision offer promising alternatives by enabling robots and autonomous systems to interpret human intentions directly from visual cues. This paper introduces a vision-based control framework that allows users to operate an autonomous drone through predefined hand gestures without any physical contact with a controller. The proposed system integrates real-time computer vision with control-system engineering to translate finger poses captured by a camera into actionable navigation commands. Our method employs PoseNet for robust hand-keypoint detection, combined with a custom gesture-classification module optimized for low-latency inference. The generated gesture classes are mapped to drone control instructions, enabling tasks such as takeoff, landing, directional movement, and hovering. The development process involved coordinated work across three subsystems: (1) Data Labeling, including dataset creation and annotation using CVAT and MATLAB; (2) Robot Interface and Connectivity, focusing on reliable communication between the vision module and the drone’s flight controller; and (3) AI Model Development, comprising model selection, training, and optimization using Python, OpenCV, TensorFlow, and Google Colab. Although the project encountered initial technical and organizational challenges, the iterative development cycle ultimately led to a stable, functional prototype. Experimental results demonstrate that the system can accurately recognize gesture commands in real time and maintain responsive drone control under various lighting and background conditions. The achieved performance highlights the feasibility of replacing traditional physical controllers with AI-driven gesture interfaces, providing an accessible alternative for users who cannot operate conventional input devices. Overall, this work contributes a practical and innovative solution for enhancing human–robot interaction through contact-free control. The presented framework has potential applications not only in assistive technologies but also in fields such as rescue operations, manufacturing, and interactive robotics, where intuitive and hands-free control is advantageous. The project also offered valuable interdisciplinary experience in computer vision, robotics, and software engineering, demonstrating the effectiveness of merging AI-based perception with control-system design.
Amin Majd, Mehdi Asadi, Juha Kalliovaara
Open Access
Article
Conference Proceedings
Emotive Design Heuristics: A Methodology for Creating and Validating Empathetic Design Heuristics for Human-Robot Interaction
Design of emotional interaction between AI-based technologies and humans will be key to their successful deployment and implementation in a wide range of domains and real-world applications. Empathetic design focuses on the development of systems and technologies to which humans can connect and empathize with. In this research we have generated a novel methodology for developing evidence-based, empathetic design heuristics that can be applied by designers and evaluators of human-robot interaction and conversational AI agents in order to guide in their design and evaluation. The objective is to maximize effective and positive emotive response by humans in their interaction with robots and related AI technologies. The methodology described involves a number of steps, beginning with consideration of the evidence-based published literature on human-robot empathetic design. This was followed by an expert panel extracting a set of design heuristics from the reviewed literature, with several rounds of heuristic development and subsequent validation of the resultant emotive design heuristics by an expert panel. Implications for the design and evaluation of social robots in healthcare are discussed.
Andre Kushniruk, Seper Rohani, Elizabeth Borycki
Open Access
Article
Conference Proceedings
User Perception and Sentiment Analysis of Knee exoskeletons for Hiking Based on Social Media Comments: A Preliminary Study
With the rapid development of wearable robotic technologies, knee exoskeletons especially for hiking have emerged as assistive devices intended to reduce physical fatigue during outdoor activities. As these systems gradually enter early-stage consumer markets, understanding users’ real-world perceptions is important for human-centered design and evaluation. This study investigates public perceptions of knee exoskeletons through analysis of user-generated comments from social media. Approximately 9,000 comments related to knee exoskeletons were collected from the Chinese social media platform Rednote over a six-month period. After data cleaning and preprocessing, 7,280 valid comments were retained. Chinese word segmentation and Term Frequency–Inverse Document Frequency weighting were applied to extract textual features, and K-means clustering was used to identify major thematic categories in user discussions. In addition, lexicon-based sentiment analysis using the National Taiwan University Sentiment Dictionary was conducted to examine emotional tendencies. The results indicate that user discussions mainly focus on product attributes and usage scenarios. Neutral expressions (4755) dominate the comments, while negative sentiment (1304) slightly outweighs positive sentiment (1221) among emotionally polarized comments, reflecting a cautious and pragmatic public attitude. Although users recognize the potential benefits of physical assistance, concerns regarding comfort, weight, usability, and practical value remain prominent. Overall, this study demonstrates the feasibility of social media comment analysis as a complementary approach for evaluating human factors in emerging wearable robotic systems.
Qingyuan Kan, Tiejun Ma
Open Access
Article
Conference Proceedings
Effects of Robot Non-Verbal Behaviors on Human Emotion Recognition in Human–Robot Communication
Social robots are increasingly used in daily environments, where effective emotional communication is essential for smooth human–robot interaction. This study investigates whether robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken narratives.Narrative-based emotional messages representing basic emotions derived from Plutchik’s emotion wheel were constructed and validated in a preliminary experiment with 132 participants. A main experiment was then conducted with twelve adult participants. Emotional narratives were presented under two conditions: with and without robot non-verbal behaviors expressing corresponding emotions. The robot executed predefined gestures, body movements, and gaze behaviors synchronized with key emotional sentences. After each presentation, participants rated perceived emotional content using a five-point emotion recognition scale. Results showed significantly higher emotion recognition scores when robot non-verbal behaviors were present in both joy and sadness conditions. These findings indicate that appropriate robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken messages. From a human factors perspective, this study provides design implications for developing social robots that support intuitive emotional communication.
Kimihiro Yamanaka
Open Access
Article
Conference Proceedings
Rule-Based Interpretable AI for Concurrent Collision Detection in Industrial Robot Manipulators
Safe human-robot collaboration in industrial environments demands collision detection systems that are both computationally efficient and interpretable. Existing approaches — based on geometric modeling, bounding volume hierarchies, or physics engines — impose significant computational overhead that limits real-time performance, particularly for high-degree-of-freedom manipulators operating in complex workspaces. This reliance on expensive computation also perpetuates manual path teaching and validation practices that reduce deployment efficiency and increase operator workload.This paper proposes a rule-based artificial intelligence framework that replaces iterative geometric calculations with a learned, symbolic representation of the collision function. Joint configurations are sampled across the robot's operational space within a simulation environment and labeled according to their collision state. An ensemble learning method is trained on this dataset to approximate the collision boundary directly from joint space, bypassing the need for explicit kinematic or geometric modeling at query time.The central contribution of this work is the systematic extraction of decision rules from the trained ensemble model. These rules are compiled into a structured knowledge base, which an inference engine queries to evaluate collision states in constant time — independent of scene complexity or robot configuration. This architecture offers two critical advantages over classical methods: a substantial reduction in computational cost during operation, and a transparent, inspectable representation of system behavior that supports validation and human oversight.The proposed method is evaluated on a six-degree-of-freedom industrial manipulator in a controlled simulation environment. Results demonstrate a significant speed-up in collision checking relative to physics-based engine calculations, achieving real-time performance suitable for integration into motion planning pipelines. Prediction accuracy remains within acceptable bounds for practical deployment, and the rule-based structure allows collision logic to be audited without specialized simulation tools.From a human factors perspective, the approach reduces dependence on manual robot teaching and path validation — tasks that remain labor-intensive and error-prone in current industrial practice. By lowering the computational and operational barriers to collision-safe motion planning, the proposed system supports safer, more efficient human-robot collaboration in manufacturing environments.
Hesam Jafarian, Marzieh Zare, Uras Ayanoglu, Juha Kalliovaara, Jarkko Paavola
Open Access
Article
Conference Proceedings
Human Factors in the Design of Human–Machine Interfaces for Counter-Drone Systems
This paper examines human–machine interfaces (HMIs) for counter‑UAS systems and identifies the interaction patterns that most strongly affect operator performance in multi‑sensor, time‑critical environments. By analysing how operators interpret fused tracks, manage alerts, and verify targets under workload, we highlight key design principles: clear presentation of uncertainty and data freshness, one‑action cross‑cueing to sensor views, compact evidence bundles that consolidate relevant cues, adaptive visual decluttering during multi‑track surges, and alerting schemes that combine severity with confidence while limiting overload. We propose a concise set of HMI‑focused performance indicators—time‑to‑acknowledge, time‑to‑track, time‑to‑verification, alert hygiene, interface stability under load, and evidence‑bundle completeness—to shift evaluation from sensor‑centric to operator‑centric metrics. The findings show that effective C‑UAS performance depends on interfaces that minimize cognitive friction, compress critical actions, and maintain situational clarity even under stress.
Daniela Doroftei, Geert De Cubber, Xavier Depreytere
Open Access
Article
Conference Proceedings
Human-Friendly Control of Drones and Drone Swarms Using Natural Language and AI-Based Task Decomposition
The deployment of drones and swarms of small drones for operational purposes is rapidly increasing the load on human operators especially if interaction relies on low-level control interfaces. This paper reports work carried out as part of a project on human-centric drone control by natural language interaction and AI-augmented task understanding. With the proposed method, operators are able to give high-level commands to single or small groups of drones and task-representations are structured from these commands and executed with predefined mission primitives. The focus of the system is to promote transparency and operator supervision through explicit feedback about the interpretation of tasks and task execution.
Geert De Cubber, Daniela Doroftei
Open Access
Article
Conference Proceedings
European University–Industry Collaboration for Civil Counter-Drone Protection: A Human-Centered, AI-Game-Based Socio-Technical Systems Approach
This paper presents a comprehensive framework for civilian counter-drone (C-UAS) systems, combining AI-based detection, data fusion, and decision-support tools with human-in-the-loop operational concepts. The research emphasizes early-warning and socially acceptable mitigation strategies to protect critical infrastructure and public spaces while adhering to European legal and data protection standards. The project further incorporates gamified, simulation-based training and evaluation approaches to support iterative testing, decision-making under uncertainty, and competence development. In addition, it integrates stakeholder engagement, interdisciplinary student involvement, and European cooperation to ensure both practical applicability and long-term workforce development in the field of civil drone defense.
Helmut Wittenzellner, Marcin Wardaszko, Willy Christian Kriz
Open Access
Article
Conference Proceedings
Possibilities of using commercial unmanned aerial vehicles due to the level of electromagnetic radiation emissions
This article presents the use of commercial unmanned aerial vehicles (UAV’s) in the context of electromagnetic emissions and electromagnetic compatibility (EMC) requirements. With the increasing use of UAV’s in monitoring, telecommunications, inspection, and measurement applications, understanding the impact of electromagnetic interference (EMI) generated by on-board propulsion systems, power converters, radio communication modules, and flight control electronics is essential. Commercial UAVs are typically not designed as low-emission platforms, limiting their use in missions requiring high electromagnetic purity. The presented research results demonstrate typical sources of high-frequency interference in UAV’s and their impact on both on-board systems and interference-sensitive radio sensors. UAV applications are classified according to their sensitivity to electromagnetic interference, indicating situations in which commercial platforms can be used directly and situations in which additional EMC measures are required. A key part of the work consists of laboratory measurements of radiated emissions in the frequency range from 30 MHz to 10 GHz for six commercial drones of different classes, conducted in an anechoic chamber. The results indicate significant differences in emission levels, particularly in frequency bands associated with engine harmonics, radio communications, and switching power supplies.
Rafał Przesmycki, Marek Bugaj, Jarosław Bugaj, Paweł Skokowski, Krzysztof Malon, Michał Kryk
Open Access
Article
Conference Proceedings
From Simple Auditory Inspection to Acoustic Feature Extraction of Drone Brushless Motors
As the application of Unmanned Aerial Vehicles (UAVs) continues to expand globally, the operational health of propulsion components such as brushless motors is critical for ensuring flight safety. Traditional inspection typically relies on manual auditory diagnostics; however, this method is inherently subjective. Because human hearing sensitivity fluctuates across different frequencies, significant discrepancies often exist between objective sound pressure levels and human perception—especially at frequency extremes. Consequently, the reliability and consistency of such auditory-based fault detection are frequently scrutinized. This research establishes an objective, multi-dimensional acoustic feature extraction framework to provide scientific, quantified data that supports human judgment, thereby enhancing diagnostic accuracy. The methodology integrates time-domain, Envelope Analysis, and Time Synchronous Averaging (TSA) techniques to extract key signal features. Analysis of a specific audio sample revealed a signal dominated by intense high-frequency noise peaking at 5,871.09 Hz, exhibiting sharp impulsive characteristics with a Crest Factor (CF) of 4.23. Following TSA processing, asynchronous noise was attenuated by approximately 84.3%, successfully isolating a periodic impact signal at 40.28 Hz, which is precisely synchronous with the shaft rotation speed. The resulting CF of 3.31 confirms the presence of regular, persistent impacts, suggesting potential bearing looseness. The proposed framework effectively isolates weak, fault-related signals from high-intensity noise environments. These objective, quantified results provide a robust scientific basis to assist operators in making more consistent and precise assessments of UAV health. Future research will focus on expanding the experimental dataset and integrating machine learning models to develop a fully automated diagnostic system.
Yan An Lin, Ming-Lang Yeh, Yu-Cheng Lin
Open Access
Article
Conference Proceedings
Centralized Control Architecture for Human-Supervised Mission-Critical Unmanned x Systems Swarms
Unmanned x Systems (UxS) are increasingly deployed in defence, search and rescue and infrastructure monitoring missions, often as heterogeneous swarms. As such, one important question is how to optimally control such swarms. The control of heterogenous swarms requires architectures that are not only operationally efficient, but also compatible with meaningful human oversight. This paper presents a comprehensive framework for centralized control of mission-critical UxS swarms, with a focus on the human-factors requirements that arise from the inherent complexity of swarm operations. The paper first establishes a rigorous definition of UxS swarm as coordinated collections of three or more unmanned platforms operating under unified command authority toward a shared mission objective. Five structural components and the role of the human in each component are identified: The Command and Control (C2) layer, Communication Infrastructure, Individual Platform Capabilities, Coordination Algorithms and the Operator Interface as the critical link between human decision-makers and autonomous swarm behaviour. The C2-layer can be designed as a centralized, decentralized or hybrid control architecture. These control architectures are first related to different levels of autonomy and their implications for the human operator. Then a systematic comparative analysis of different C2-architectures is conducted against mission-critical performance metrics including operational efficiency, fault tolerance, decision predictability, human command integration and adaptability. The analysis demonstrates that centralized architectures, when augmented with redundant command nodes and mesh-based communication, deliver superior performance across all mission-critical metrics. In contrast, decentralized architectures, while theoretically scalable, exhibit fundamental limitations in human oversight and global optimization that are incompatible with high-consequence operations. While centralized control architectures are recommended, different levels of system autonomy within a centralized C2-system result in different requirements regarding human-system integration. For example, if the human operator is required to monitor the behaviour of a highly autonomous system concurrently, decision transparency has to be a fundamental design principle. The centralized architecture has to enable the generation of comprehensive decision rationales, reasoning and confidence metrics that support operator trust calibration and meaningful intervention. In addition, the situation awareness of the human operator has to be ensured through optimal representation of the situation awareness requirements relevant for the current situation on the human machine interface. Further design requirements for C2 of heterogenous swarms are derived in the current paper. These findings are directly relevant to the design of next-generation human-swarm interaction systems in safety-critical environments.
Batuhan Özcan, Birte Thomas-friedrich
Open Access
Article
Conference Proceedings