Multimodal Learnability Assessment of a Touch-based Large Area Display with Eye Tracking and Optical Brain Imaging

Abstract: Multifunctional Large-Area Displays (LAD) have become an integral part of modern airplane cockpits, offering pilots flexible access to flight controls and mission-critical information. The modern glass cockpit paradigm is expected to reduce the workload of pilots by reducing the complexity and clutter of traditional cockpit layouts and improve their situational awareness by providing flexible access to rich information through the interface. On the other hand, the new paradigm has led to interaction design challenges in utilizing the affordances of the novel interface components, presenting barriers agains effective use of the interface by the pilots. In the present study, we report empirical investigations of such a LAD interface and its learnability via behavioral performance, established upon the neuroergonomics approaches, such as eye tracking and optical brain imaging measurements. Two test pilots, who had prior experience in traditional cockpit layouts were recruited to perform flight tasks in a flight simulator incorporating a LAD touch screen to interact with basic flight instruments of a trainer jet. The pilots were first given standardized training to familiarize with the basic information layout and features of the LAD interface. Following the initial training, both flew a scenario in the simulator that included various tasks (e.g., functional layout assignment, setting communication parameters, monitoring barometric settings and fuel levels, etc.) over the LAD interface for the before take-off, taxi, take off, climb, level flight, approach, and landing phases. The Pupil Labs Invisible mobile eye tracker and the fNIR Imager 1100 system were used to monitor the pilots’ eye movements and prefrontal oxygenation changes during the flight. Two weeks later, the tasks were replicated for the learnability assessment. The results indicated that pilots’ task completion times decreased for the majority of the tasks, accompanied by an increase in eye fixation durations, a decrease in the number of fixations, and a decrease in right dorsolateral prefrontal cortex activation. Overall, these results suggest supporting evidence for the learnability of the new interface paradigm through task measurements in multiple scenarios.

Keywords: Multimodal Analysis, Eye Tracking, fNIRS, Large Area Touch Display, Aviation

DOI: 10.54941/ahfe1003007

Cite this paper:

​