The Mediated Moon: Semantic Mediation Between Gesture and Generative Image
Abstract
This paper investigates how vision-language model (VLM)-based semantic mediation can reorganize the relationship between bodily input and real-time generative output in interactive art. The study develops a scaled interactive prototype from Luminous Night, an existing moon-inspired spatial light artwork. The system combines MediaPipe hand tracking, a locally deployed Gemma 3 model, and StreamDiffusion-based real-time image generation. A participant’s hand contributes both numerical tracking data and visual input; the VLM generates four semantic keywords within an artist-defined semantic field, which then participate in conditioning generative imagery presented through palm projection and LED output. The paper uses semantic mediation to describe this intermediate transformation from bodily input to language and image, and proposes the designed gap as an artistic design proposition for a partially indeterminate interaction in which bodily input remains influential without completely specifying the result. Observations made during prototype development provide an initial account of how participants adjusted and repeated gestures while encountering changing system responses. The contribution lies in articulating semantic mediation as an interaction configuration for generative art and establishing a practical basis for examining the relationship among bodily influence, generative variation, and perceived control.
Keywords: Semantic Mediation, Vision-Language Model, Embodied Interaction, Human-AI Interaction, Interactive Art
DOI: 10.54941/ahfe1008246
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- A Sustainable Assistive Technology for Children with Visual Impairments


AHFE Open Access