Integrating Robotics, AI, and Immersive Technologies: A Modular Framework for Human-Metahuman-Robot Collaboration

Abstract: Collaborative robots have been rapidly increasing across industries, particularly in manufacturing settings. This advancement allows humans and robots to work side by side to complete tasks more efficiently. Moreover, with the development of synthetic actors like Metahumans, humans can now enter immersive environments where these Metahumans act as guides and help humans in task executions. However, there is limited implementation of combining both technologies (synthetic actors and collaborative robots) in the industrial field.This paper proposes a system that combines speech recognition, object detection, motion planning, and AI-enabled Metahuman guidance within an immersive environment. The system architecture demonstrates the simple fetching and positioning of components through a robotic arm commanded by a human guided by a Metahuman. The system ensures seamless communication between nodes by utilizing ROS (Noetic version) along with advanced tools for speech recognition, object detection, and motion planning. Such modular architecture allows each component—voice recognition, command parsing, object detection, and robotic motion—to function independently while collaborating through ROS communication protocols. This ensures flexibility, scalability, and ease of maintenance that makes the system adaptable to various environments and use cases. For example, voice recognition simplifies human-robot communication, while computer vision ensures accurate object detection and localization, allowing the robot to perform precise manipulation tasks.Using a Metahuman in an immersive environment enhances the user experience by providing real-time guidance and feedback. This paper aims to demonstrate how metahumans can enhance efficiency and guidance for humans and how collaborative robots can assist them in an industrial context. We present an illustration showcasing how a Metahuman can guide a human, who then commands a robotic arm according to the instructions of the Metahuman during a simple pick-and-place task on an assembly line. The goal of this work is to improve the pedagogical curve of an assembly worker by introducing Metahumans and collaborative robots.However, the system has some limitations that warrant future exploration. Currently, humans are the bridge between the metahuman and the robot. As a result, the user needs to verify with the Metahuman after every step whether the robotic arm has successfully completed its task. If direct communication between the Metahuman and the robotic arm can be established, the Metahuman could modify its commands in real time based on the robot's performance. Another limitation is the dependency on predefined object detection models, which may struggle in cluttered or dynamic environments. Similarly, the speech recognition module could benefit from enhanced capabilities to understand complex or domain-specific commands. Future research could explore reinforcement learning to improve the robot's adaptability and integrate advanced natural language processing models to handle more nuanced interactions.In conclusion, this work demonstrates a practical approach to combining robotics, artificial intelligence, and immersive technologies to create an intuitive and efficient human-robot collaboration system. The modular design facilitates ease of use and flexibility and lays a foundation for future advancements in the field. By bridging the gap between humans and robots, this system paves the way for innovative applications in industrial automation, education, and beyond, showcasing the immense potential of integrating emerging technologies to redefine human-robot interaction. Moreover, the integration of the Metahuman enables non-technical users to effectively interact with advanced robotics, making the system accessible and user-friendly.

Keywords: Collaborative Robot, Human Computer Interaction, Metahuman, Virtual Reality

DOI: 10.54941/ahfe1006373

Cite this paper:

​