The Mediated Moon: Semantic Mediation Between Gesture and Generative Image

Open Access
Article
Conference Proceedings
Authors: Shu-Yu Lin, Jiaming Day, Derlor Way
Abstract

This paper investigates how vision-language model (VLM)-based semantic mediation can reorganize the relationship between bodily input and real-time generative output in interactive art. The study develops a scaled interactive prototype from Luminous Night, an existing moon-inspired spatial light artwork. The system combines MediaPipe hand tracking, a locally deployed Gemma 3 model, and StreamDiffusion-based real-time image generation. A participant’s hand contributes both numerical tracking data and visual input; the VLM generates four semantic keywords within an artist-defined semantic field, which then participate in conditioning generative imagery presented through palm projection and LED output. The paper uses semantic mediation to describe this intermediate transformation from bodily input to language and image, and proposes the designed gap as an artistic design proposition for a partially indeterminate interaction in which bodily input remains influential without completely specifying the result. Observations made during prototype development provide an initial account of how participants adjusted and repeated gestures while encountering changing system responses. The contribution lies in articulating semantic mediation as an interaction configuration for generative art and establishing a practical basis for examining the relationship among bodily influence, generative variation, and perceived control.

Keywords: Semantic Mediation, Vision-Language Model, Embodied Interaction, Human-AI Interaction, Interactive Art

DOI: 10.54941/ahfe1008246

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design EducationA Sustainable Assistive Technology for Children with Visual Impairments →
View all articles in Human Factors in Design, Engineering, and Computing →