From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects

Open Access
Article
Conference Proceedings
Authors: Yishan Hu, Liwen Liu, Qihang Yu, Sishuo Liu, Yiqi Wang
Abstract

Visual information plays a central role in many contemporary interactive experiences, while sound is often treated as a supplementary feedback channel. This study presents Contour Sound Chamber, an object-based tangible interactive music system that explores how physical object properties and bodily actions can be transformed into auditory experiences. The system integrates AI-based object recognition, spatial sensing, sound mapping, and tangible interaction. Objects are classified according to material and color temperature, forming four predefined categories. Material determines the basic timbre, color temperature influences tonality and instrumental layers, and contour data acquired through ToF scanning are quantized into rhythmic patterns. Users further participate in the generated music through four tangible interaction modes: striking, plucking, pressing, and stroking, corresponding to different instrumental sounds. A user study with 34 participants evaluated object recognition, interaction task performance, and subjective perception. The system achieved a 97.1% first-attempt recognition accuracy across 136 trials, while the interaction tasks achieved an overall mean efficiency of 90.1% and a completion rate of 96.3%. The subjective evaluation showed an overall Likert score of 4.60/5. Participants were able to perceive differences between sounds generated by different objects, recognize the influence of their actions on musical output, and perceive certain object characteristics through sound. Together, these findings provide initial support for a perceptible relationship between object properties, sound representation, and user interaction, highlighting the potential of sound as an active component of multisensory interaction.

Keywords: Human-Computer Interaction, Human-Centered Design, Tangible Interaction, Interactive Music, Multimodal Interaction, Object Recognition, Spatial Sensing

DOI: 10.54941/ahfe1008243

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional SynchronizationThe Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services →
View all articles in Human Factors in Design, Engineering, and Computing →