Visualizing Uncertain Real Time Threat Information in Augmented Reality Aided Target Recognition: Lessons Learned from Virtual Reality Simulation

Abstract: Recent advancements in artificial intelligence (AI), edge computing, and head-worn augmented reality (AR) technology are bringing the prospect of sophisticated aided target recognition (AiTR) systems from sci-fi to reality. Future AI algorithms could augment AiTR with real-time threat assessments (RTAs) that augment Soldier decision making by providing a binary threat assessment alongside an estimate of epistemic algorithmic uncertainty through the fusion and interpretation of multiple data sources. Yet, visual representations of probability are often misinterpreted, which could have consequences when relying on uncertain RTAs. To investigate, we designed simulated uncertain RTAs in virtual reality (VR) using emerging probabilistic visualization techniques, such as hypothetical outcome plots (HOPs; Hullman et al., 2015) and discrete-outcome framing (Franconeri et al., 2021), and quantified their impact on lethal force decision making. Specifically, we extended a VR decision making task from our previous work (Gardony et al., 2022; Frontiers in VR), in which participants categorized a Soldier target advancing towards them as friendly or enemy based upon their worn camouflage pattern, overlaying continuous- and discrete-outcome framed uncertain RTAs and introducing gamification elements to encourage rapid decision making. We found that when targets were easy to distinguish, participants were more conservative when categorizing targets as enemy vs. friendly, reflecting a learned decision-making heuristic. Importantly, under conditions of relatively low perceptibility (i.e., for far-away targets), our findings suggest trust in and reliance on RTAs increased, as evidenced by attenuated conservativity that deviated from the default heuristic. These findings contribute to the emerging literature on trust in AI and have implications for the design and deployment of effective military human-AI interfaces.

Keywords: Aided Target Recognition, Uncertainty, Visualization, Decision Making, Augmented Reality, Virtual Reality

DOI: 10.54941/ahfe1005027

Cite this paper:

​