Communicating Uncertainty in AI-Based Decision Support Systems: A Comparative Study of Numerical and Visual Representations

Abstract

AI-based Decision Support Systems (AI-DSS) are increasingly recognized for their significance in professional environments. A key challenge in human-AI interactions is effectively communicating the uncertainty inherent in AI recommendations, as this can influence performance outcomes. Various methods exist for representing uncertainty, primarily through numerical data or visual cues. While users often favor numerical probabilities for their perceived precision, these figures can be difficult to interpret. Conversely, visual representations may enhance understanding but tend to be less accepted by users. The existing literature lacks clear conclusions regarding the impact of these communication designs on user performance and cognitive load. This research examines the effects of two forms of uncertainty communication—numerical (decimal numbers) and visual (traffic light system)—on user performance and cognitive load. An online experimental study was conducted with 104 participants assigned randomly to either condition within an AI-supported customer service context. Participants responded to support request emails using AI-ranked response modules while retaining decision-making authority. Each participant engaged with ten vignettes and completed questionnaires measuring task load afterward; performance was assessed based on correctly answered vignettes. Results indicated no significant differences in task load between groups. However, notable variations in performance emerged when systems made errors, influenced by the communication design used. These findings suggest that effective uncertainty communication strategies may vary based on context and audience, offering valuable insights for designing AI-DSS.

Keywords: Uncertainty Communication, AI-based Decision Support, Human-AI-Interaction

DOI: 10.54941/ahfe1007986