SeeBeyond: An AI-Powered Mobile AR System for Context-Aware Color Assistance

Abstract

People with color vision deficiency (CVD) face challenges in perceiving and distinguishing colors. Existing assistive approaches primarily rely on technical color correction and visual feature substitution. However, these methods often lack contextual awareness, intuitiveness, and universality, making it difficult to effectively support users' cognition and decision-making in everyday life. To address this issue, this study investigates the real-world needs of people with CVD through questionnaire surveys, semi-structured interviews, and situational simulations. Based on our findings, we propose a Context-Aware Color Interpretation Framework. This framework categorizes daily situations into three hierarchical levels based on task urgency and response requirements: instant decision-making, daily perception, and experience enhancement. Guided by this framework, we designed SeeBeyond, an AI-powered mobile and augmented reality (AR) system. The system integrates real-time color recognition with multi-modal feedback, providing personalized interaction adaptations tailored to the three contextual levels. By instantiating this framework through SeeBeyond, we demonstrate the feasibility of delivering context-aware, multi-modal assistance in everyday scenarios. This work shifts the focus of CVD assistive technologies from mere visual correction to holistic, context-driven cognitive support, providing a novel design paradigm for accessible interaction.

Keywords: Color Vision Deficiency (CVD), Accessible Interaction, Color Assistance, Augmented Reality, Artificial Intelligence

DOI: 10.54941/ahfe1007293