From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
Abstract
Visual information plays a central role in many contemporary interactive experiences, while sound is often treated as a supplementary feedback channel. This study presents Contour Sound Chamber, an object-based tangible interactive music system that explores how physical object properties and bodily actions can be transformed into auditory experiences. The system integrates AI-based object recognition, spatial sensing, sound mapping, and tangible interaction. Objects are classified according to material and color temperature, forming four predefined categories. Material determines the basic timbre, color temperature influences tonality and instrumental layers, and contour data acquired through ToF scanning are quantized into rhythmic patterns. Users further participate in the generated music through four tangible interaction modes: striking, plucking, pressing, and stroking, corresponding to different instrumental sounds. A user study with 34 participants evaluated object recognition, interaction task performance, and subjective perception. The system achieved a 97.1% first-attempt recognition accuracy across 136 trials, while the interaction tasks achieved an overall mean efficiency of 90.1% and a completion rate of 96.3%. The subjective evaluation showed an overall Likert score of 4.60/5. Participants were able to perceive differences between sounds generated by different objects, recognize the influence of their actions on musical output, and perceive certain object characteristics through sound. Together, these findings provide initial support for a perceptible relationship between object properties, sound representation, and user interaction, highlighting the potential of sound as an active component of multisensory interaction.
Keywords: Human-Computer Interaction, Human-Centered Design, Tangible Interaction, Interactive Music, Multimodal Interaction, Object Recognition, Spatial Sensing
DOI: 10.54941/ahfe1008243
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image
- A Sustainable Assistive Technology for Children with Visual Impairments


AHFE Open Access