AI-Powered Auditory Control and Augmented Reality Interfaces for UAVs - A Contactless Control and Situation Awareness Concept

Abstract: Unmanned Aerial Vehicles (UAVs) are increasingly utilized in military and civilian tasks like search and rescue, however, traditional operation methods can be risky in hazardous situations. This article presents a novel UAV control concept leveraging artificial intelligence (AI) and Augmented Reality (AR) technology, allowing operators to manage drones without handheld devices through audio-based input and output. The suggested system employs headsets and AR glasses to provide real-time visual feedback, enhancing situational awareness and decision-making by displaying critical data such as UAV position and detected hazards within the operator's field of view. The concept comprises five key components implemented within the Robot Operating System (ROS): Audio Input, Task Allocation, UAV Control, Situation Picture, and Output Units. Speech is processed using models such as Whisper, and commands are interpreted by a Large Language Model (LLM) like GPT-4, ensuring accurate recognition even in noisy environments. Initial experiments show high command recognition accuracy, indicating the concept's potential for reliable UAV control in real-world scenarios. Overall, this approach aims to improve operational efficiency and safety in UAV operations, with future work focusing on system refinement and advanced language processing.

Keywords: UAV-control, augmented reality, artificial intelligence, speech-based interaction

DOI: 10.54941/ahfe1005917

Cite this paper:

​