Understanding the Needs and Challenges of Developing Robot Teleoperation Applications using Mixed Reality Headsets

Abstract

Robot teleoperation enables humans to control robots to perform tasks and collect data to train robot intelligence. Compared to traditional interfaces, extended reality (XR)-based robot teleoperation offers more natural, efficient, and scalable interactions with reduced cognitive load. However, developing such applications involves interdisciplinary challenges across hardware, integration, interface design, and manipulation. To understand current practices and challenges, we conducted semi-structured interviews with 15 developers, ranging from novice prototypers to industry experts. While prior work focuses on end-user usability, this study explores the developer experience (DX) bottlenecks from the perspective of developers at varying levels of expertise, including undergraduate prototypers, graduate researchers, and industry practitioners. We identify a "Middleware Gap" where network instability and protocol mismatches hinder reproducibility, and a "Data Utility Crisis" where current XR tracking lacks the fidelity required for robust imitation learning. We contribute a refined taxonomy of XR teleoperation and a set of prioritized design implications, moving beyond generic wish lists to specific architectural requirements for interoperability, sensory substitution, and human-in-the-loop safety.

Keywords: Robot Teleoperation, Extended Reality, Mixed Reality, Developer Experience

DOI: 10.54941/ahfe1007501