An AI-Based Adaptive Pipeline for Automated Feedback in Immersive Robotics Learning
Open Access
Article
Conference Proceedings
Authors: Mohammadreza Akbari Lor, Bhanu Vodinepally, Tisa Islam Erana, Bhavleen Kaur, Giancarlo Perez, Seth Corrigan, Shu-ching Chen, Mark Finlayson, Biayna Bogosian, Shahin Vassigh
Abstract: In this paper, we present a pipeline and framework for the Intelligent Immersive Learning environment for Programming Robotics Operations (IL-PRO), a novel AI-based approach to assess and enhance learner capabilities in an immersive virtual reality (VR) environment. By integrating telemetry data (both continuous and discrete) and speech data, the IL-PRO pipeline evaluates users' motor skills and cognitive understanding to deliver personalized, real-time feedback that links their conceptual understanding with motor skill performance. Telemetry data captures precise physical human-system interactions which are processed and analyzed using Machine Learning (ML) tools to capture and rate motor skill capabilities, while speech data is analyzed using Natural Language Processing (NLP) techniques in concert with a Large Language Model (LLM) to simultaneously assess comprehension and task-related knowledge. These insights are then integrated and used to provide feedback and adapt the learning environment dynamically, tailoring tasks and modules to the learner’s specific needs and progress. To demonstrate the feasibility of this approach, we apply the pipeline to a VR task focused on robot acceleration, which emphasizes how motor skills and cognitive understanding work together when learning about inertia in industrial robotic arms. This use case illustrates the pipeline's comprehensive workflow: data collection, multimodal processing of telemetry and speech using machine learning and AI, integration of cognitive and physical insights, and generation of adaptive, real-time feedback. The IL-PRO pipeline framework advances the development of immersive learning systems, enables research on how users combine motor skills with cognition, and enhances skill acquisition in applied training contexts such as robotics.
Keywords: Immersive Learning, Artificial Intelligence, Machine learning, Large Language Models, Multimodal Analysis, Adaptive Feedback Systems
DOI: 10.54941/ahfe1006372
Cite this paper:
Downloads
17
Visits
62