Toward Human-Centered Swarm Control: A VR-based UAV Simulator for Training and Cognitive Evaluation

Abstract

This paper introduces a virtual reality (VR)-based unmanned aerial vehicle (UAV) simulator designed to support immersive training and early-stage human factors evaluations in rescue and emergency mission contexts. The system simulates a leader-follower UAV configuration, where a human operator controls a lead drone through a VR headset and joystick, while autonomous drones maintain any geometric formations using onboard sensing and dynamic obstacle avoidance [1]. This swarm-based coordination reflects real-world search and rescue scenarios, where rapid decision-making, spatial awareness, and teamwork are essential.The simulator is developed with a focus on human-in-the-loop design, providing an immersive teleoperation experience that places the user in high-pressure environments with dynamic spatial constraints. Visual and auditory feedback, along with the ability to switch between multiple drone perspectives, is intended to support situational awareness, mission control, and error recovery in complex terrain.From a human factors perspective, the simulator serves as a flexible testbed for evaluating cognitive and ergonomic variables in safety-critical tasks. It allows the assessment of operator workload, interface usability, attention allocation, and situation awareness under realistic but controlled conditions. Subjective evaluation tools such as the NASA Task Load Index (NASA-TLX) and the Situation Awareness Rating Technique (SART) [2] can be embedded directly within the VR experience. At the same time, objective data from head and hand movement, control inputs, and UAV performance are recorded to gain insight into operator behavior.The simulator integrates high-fidelity UAV dynamics using closed-loop reference model adaptive controllers [3], and can be equipped with tools such as eye tracking, physiological sensors for workload estimation, and visual attention assessment methods like ATTENDO [4]. These additions support deeper analysis of how operators acquire information, prioritize tasks, and shift attention during mission-critical events.Overall, the VR-based UAV simulator addresses a growing need to evaluate and enhance human performance in multi-agent control systems used in emergency operations. It offers a scalable, immersive environment that supports both training and cognitive engineering. Its modular structure allows for future integration of real UAVs, digital mapping capabilities, and adaptive interfaces, making it a valuable platform for advancing human-centered UAV system design.References [1] Saunders, J., Call, B., Curtis, A., Beard, R., & McLain, T. (2005). Static and dynamic obstacle avoidance in miniature air vehicles. In Infotech@ Aerospace (p. 6950).[2] Braarud, P. Ø. (2021). Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work. International Journal of Industrial Ergonomics, 86, 103233.[3] Eraslan, E., & Yildiz, Y. (2021, December). Modeling and adaptive control of flexible quadrotor uavs. In 2021 60th IEEE Conference on Decision and Control (CDC) (pp. 1783-1788). IEEE.[4] Oberhauser, M., & Dreyer, D. (2017). A virtual reality flight simulator for human factors engineering. Cognition, Technology & Work, 19, 263-277.

Keywords: VR simulation, UAV training, situational awareness, digital twins, swarm robotics

DOI: 10.54941/ahfe1006904