Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals: Insights from Multi-Element Perception Tasks

Abstract: Many Deaf and Hard-of-Hearing (DHH) individuals enjoy listening to music and singing. However, the way music sounds to them varies from person to person, and even among those with the same level of hearing loss, perceptions differ. Furthermore, DHH individuals focus on different elements when listening to music. Although music training systems for DHH individuals have existed before, many were designed and implemented in laboratory settings. We aimed to create a music training system for DHH individuals that could be used playfully in real-world environments, such as on their own smartphones or tablets, and for researchers and educators to provide an easily extensible framework. This presentation describes the versatile framework and trial experiments with DHH participants. The results revealed the importance of selecting training materials and setting training difficulty levels, especially for DHH participants whose music perception is affected by several factors, including hearing levels, interest in musical elements, and musical experiences. We envisioned a music training system integrating subsystems designed to help DHH individuals hear more of what they want to hear when listening to music. The subsystems share the same user interface, presenting the target audio and four alternative audio samples. Separate instructions indicate which selection to make, without visual or haptic cues. As users must listen carefully to the target sound and compare it with the alternatives, we call this subsystem “Music Memory.” We created Music Memory for four musical elements using this simple interface. We set problems for each of these: 1) selective listening, 2) instrument identification, 3) recognizing melody or rhythm variations, and 4) tempo recognition. To investigate whether the difficulty levels set for each of the four Music Memories were appropriate and whether the acoustic data for the alternatives presented matched the target audio, experiments were conducted with DHH participants. The problem audio sources were selected from J-POP and anime songs, which DHH frequently listens to. Between 8 and 20 young DHH participants took part in the four experiments. The DHH participants in the four Music Memory experiments differed (with some overlap).The accuracy rates for the four memory experiments were: 1) mostly ceiling effect observed, 2) ceiling effect observed, 3) slightly above 50%, and 4) slightly below 25%, namely below the chance level. For condition 3, a tendency was observed where higher musical experience correlated with higher accuracy rates. The accuracy rates, though averages, indicate the problem settings that are either too easy or too difficult. This suppresses the user's motivation to continue training and makes it difficult for trainers to refine the provided materials. For use in training, the slightly above 50% accuracy rate observed in 3) might be appropriate among the four Music Memories. However, in 3), concerns were raised about the lack of criteria for creating alternatives involving melody or rhythm alterations, potentially leading to subjective provision of acoustic data.These findings highlight the necessity of carefully designing training difficulty levels and instructional materials before applying the system in practice. Not only for 3), but all three other Music Memories need the objective provision to provide alternatives. For trainees to keep using Music Memory, the interface will need to give feedback on their answers, and we may consider additional sensory modalities to reduce the burden of using Music Memory for DHH people. This research has established a foundation for the practical implementation of a music training system for DHH individuals. However, future challenges include refining the personalized adaptation algorithm, verifying long-term training effects, and conducting evaluations with a larger participant sample. Then the system will contribute to the realization of music training environments that are optimized for each individual DHH and to construct inclusive music education environments. This research indicates a new research direction in the interdisciplinary field of assistive technology for the DHH people and music education.

Keywords: Deaf and Hard of Hearing, Music, Training System

DOI: 10.54941/ahfe1006949

Cite this paper:

​