Multimodal Assessment of Pilot Cognitive Workload Using ECG and Eye-Tracking Features in Simulated Flight Tasks

Abstract

Pilot cognitive workload is a critical determinant of task performance, decision-making quality, and overall flight safety, rendering its objective and accurate assessment essential. However, current cognitive load assessment methods rely mainly on electroencephalography（EEG） signals that are easily disturbed, which limits their real-world use. Therefore, this study focuses on assessing pilot cognitive workload in realistic application scenarios, and proposed a multimodal assessment method of pilot cognitive workload base on Electrocardiogram (ECG) and eye-tracking signals, which their have high reliability and applicability. Firstly, flight tasks with graduated difficulty were designed in a simulated environment to induce three distinct levels of cognitive workload, and at the same time, the pilots are required to complete corresponding Overall Workload Scale (OWL) and the NASA Task Load Index (NASA-TLX) scale in different flight tasks. In the objective assessment, the multimodal signals of the pilot were collected based on ECG and eye-tracking signals for feature extraction. Then, effective feature indicators were selected using ANOVA, and redundant features were removed through Spearman correlation analysis. Finally, a multimodal pilot cognitive workload assessment model based on ECG and eye-tracking signals was built using a stacking ensemble learning framework with decision-level fusion. Results demonstrated the effectiveness of the workload task paradigm. Subjective ratings increased progressively with task difficulty, as confirmed by both NASA-TLX and OWL. Physiological analysis revealed distinct trends under increased workload. For ECG metrics, heart rate rose significantly, while heart rate variability indices—specifically Mean NN, RMSSD, LF, HF, TP, and Lempel-Ziv complexity—demonstrated significant decreases. Regarding ocular metrics, fixation duration, blink interval, and pupil diameter increased significantly, whereas saccade duration and blink frequency decreased. Following feature refinement via Spearman correlation analysis to remove redundancy (coefficient > 0.8), eleven key features were retained: six ECG features (Heart Rate, RMSSD, LF, HF, ApEn, and Lempel-Ziv complexity) and five eye-tracking features (Fixation Duration, Saccade Duration, Blink Count, Blink Interval, and Pupil Diameter). Based on these refined features, the multimodal assessment model, built with a stacking ensemble framework using decision-level fusion and employing Random Forest and K-Nearest Neighbors as base classifiers for ECG and eye-tracking data, achieved an accuracy of 0.959. Collectively, these findings substantiate the validity and sensitivity of fusing ECG and ocular metrics, providing critical data and technical support for the advancement of pilot physiological monitoring and early-warning systems.

Keywords: Cognitive Workload Assessment, human Factors In Aviation, electrocardiogram (ECG), Eye Tracking, multimodal Fusion Model

DOI: 10.54941/ahfe1007396