Building a Multicamera and Multimodal 3D Skeleton-based Pose Estimation Dataset to Enhanced Human-Robot Collaboration

Abstract

Human Pose Estimation (HPE) is an essential computer vision task that plays a significant role in ergonomics, human-centered design of workplaces, and collaborative robotics, particularly in providing a safe, adaptive, and valuable human-robot collaboration (HRC). Nevertheless, the need for a concise and practical approach to setting up a multicamera and multimodal system for HPE dataset generation remains an open issue in the literature.The main goal of this work is to describe in concise steps a protocol that serves as a guide in the exploratory stage of the approach to construct an extensive multicamera and multimodal dataset used to enhance HRC. In this paper, the proposed protocol specifically addresses the challenge of designing a biomechanical model that can consistently reproduce complex and variable human motion analyses in an assembly task while considering ergonomic factors. Furthermore, the resulting work led to the definition of a marker set for one single-person future pipeline involving the placement of thirty-two reflective markers for 3D motion analysis, specifically emphasizing the upper segments of the human body, including the hands. The future generation of this dataset will hold significant promise for advancing the study of HRC. It will introduce reliable and precise multimodal data collection, such as human kinematics and video data, including depth data, which will then be used for posture metrics analysis. Finally, the dataset will be a valuable resource for the research community, enabling the training of machine learning models. These models will empower collaborative robots (cobots) to learn from human demonstrations, enhancing their efficiency and ergonomic performance in assembly tasks.

Keywords: Human Pose Estimation Dataset, Human-robot collaboration, Human-centered design, Ergonomics

DOI: 10.54941/ahfe1005526