Developing an Approach to Measure Trust in Flight Testing of Autonomous Aircraft Systems: An LLM-Assisted Robotic Journaling Proof of Concept
Abstract
Integrating autonomous systems into high-stakes military aviation requires robust, repeatable methods for evaluating operator trust. While developmental flight testing generates exhaustive technical log data during envelope expansion, these quantitative records capture system states but fail to document the nuanced cognitive evolution of operator trust. This paper presents a case study for applying the Robotic Journaling methodology originally developed and validated for heterogeneous human-machine teams in R3 (Real users, Real systems, Real consequences) environments to autonomous flight tests. As a precursor to Phase 2 flight testing, this Phase 1 study utilizes simulated data from a software control environment to establish and validate a Trust Calibration Indicator Codebook for flight test maneuvers. The approach uses a Large Language Model (LLM) to aggregate multi-modal data on system behavior, alerts, operator interventions, and communications into a coherent narrative for qualitative analysis. Flight logs contain extensive data about what events took place during a test point; LLMs aggregate that information into a narrative that non-engineers can understand, which can then be correlated to an operator’s retrospective account of their experience on that flight. These narratives are then used as the basis for a cross-referencing analysis, a key tool for scoping the boundaries of trust calibration. This study establishes a replicable framework for analyzing how an operator's trust evolves in response to an autonomous system's performance during flight test maneuvers. The central contribution is the Trust Calibration Indicator Codebook: a structured set of observable codes derived from over-trust and under-trust theory that operationalizes trust calibration for flight tests. Phase 2 is positioned to empirically calibrate and validate these indicator codes against actual flight test data.
Keywords: Trust Calibration, Robotic Journaling, LLM-Assisted Analysis, Autonomous Flight Test, Cross-Referencing, Over-Trust, Under-Trust, Human-Autonomy Teaming, Developmental Test & Evalu
DOI: 10.54941/ahfe1008317
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image


AHFE Open Access