Developing an Approach to Measure Trust in Flight Testing of Autonomous Aircraft Systems: An LLM-Assisted Robotic Journaling Proof of Concept

Open Access
Article
Conference Proceedings
Authors: Nathalie Ochoa, Martin Ha, Justin Stofik, Rachel Jager, Brenda Haslup, Boyoung Kim, Nhut Ho
Abstract

Integrating autonomous systems into high-stakes military aviation requires robust, repeatable methods for evaluating operator trust. While developmental flight testing generates exhaustive technical log data during envelope expansion, these quantitative records capture system states but fail to document the nuanced cognitive evolution of operator trust. This paper presents a case study for applying the Robotic Journaling methodology originally developed and validated for heterogeneous human-machine teams in R3 (Real users, Real systems, Real consequences) environments to autonomous flight tests. As a precursor to Phase 2 flight testing, this Phase 1 study utilizes simulated data from a software control environment to establish and validate a Trust Calibration Indicator Codebook for flight test maneuvers. The approach uses a Large Language Model (LLM) to aggregate multi-modal data on system behavior, alerts, operator interventions, and communications into a coherent narrative for qualitative analysis. Flight logs contain extensive data about what events took place during a test point; LLMs aggregate that information into a narrative that non-engineers can understand, which can then be correlated to an operator’s retrospective account of their experience on that flight. These narratives are then used as the basis for a cross-referencing analysis, a key tool for scoping the boundaries of trust calibration. This study establishes a replicable framework for analyzing how an operator's trust evolves in response to an autonomous system's performance during flight test maneuvers. The central contribution is the Trust Calibration Indicator Codebook: a structured set of observable codes derived from over-trust and under-trust theory that operationalizes trust calibration for flight tests. Phase 2 is positioned to empirically calibrate and validate these indicator codes against actual flight test data.

Keywords: Trust Calibration, Robotic Journaling, LLM-Assisted Analysis, Autonomous Flight Test, Cross-Referencing, Over-Trust, Under-Trust, Human-Autonomy Teaming, Developmental Test & Evalu

DOI: 10.54941/ahfe1008317

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← Exploring Usability and User Perception in an Immersive Robotics Learning EnvironmentBrailleBridge: A Cost-Conscious Mechatronic Prototype for Digital Text-to-Tactile Braille Conversion →
View all articles in Human Factors in Design, Engineering, and Computing →