Mixed reality control of a mobile robot via ROS and digital twin

Abstract: In recent years, mobile robotics has been increasingly used in agricultural production due to technological progress and increasingly powerful IT systems. Today, drones are already being used to spray agricultural areas with pesticides, for example. In the field of mobile robotics, control is currently based primarily on a controller or an app on a smartphone or tablet. If spatial target coordinates are to be specified, which the robot has to control, both systems quickly reach their limits. Using a mobile robot developed for the management of monocultures, this paper explains how spatial target coordinates can be approached with the help of a digital twin, ROS and a mixed reality interface using an MS HoloLens. The fusion of the technologies accordingly opens up new possibilities for a "human-robot interface".When developing an operating concept for the mobile robot, problems quickly arise regarding the determination of coordinates. The explanation for this is: The coordinate systems of the HoloLens and the mobile robot are not identical and, above all, are not known to the user. However, these obstacles can be eliminated by using a Digital Twin, which is displayed on the real robot using the HoloLens. By this measure, the coordinate system of the mobile robot is made known to the mixed reality device and it is possible to send the position coordinates as an offset to the current position of the robot. However, it should be noted that Unity (on the HoloLens) uses the left-hand rule, while ROS (on the robot) uses the right-hand rule. Therefore, the position data must be transformed accordingly.The same applies to the rotations, which must be converted as the equivalent of the coordinate transformation. Afterwards, the converted pose, which is now available in a "base_link" frame, can be transferred to the mobile robot via ROS.For the communication between the mixed reality device and the mobile robot, a ROS Bridge Client, which provides the messages in JSON format, is used. When sending target poses, it is important to note that the current time-stamped must be mandatory included in the message, otherwise the transformation from the "base_link" frame will not be converted to "map" coordinates. The transformation of the pose must also be done for the visual solution of the path planning. For the visualization, a distinction is made between the local and the global path planner A digital twin is superimposed on the real mobile robot for control and path planning. In the course of the project, problems with accuracy or performance have been discovered. In the intended application area - agriculture - the performance is weighted higher after an evaluation, so that slight position inaccuracies are accepted. The paper presents the concept as well as the interaction possibilities between the human and the mobile robot via the digital twin.

Keywords: Mixed Reality, Augmented Reality, Mobile Robot, Digital Twin

DOI: 10.54941/ahfe1003751

Cite this paper:

​