Decoding Internal Decision Making During Reverse Engineering Tasks

Abstract: Neural decoding is often limited to tasks with known stimuli and limited response options. Real world tasks, however, are often completely stimulus free with unconstrained user response possibilities. To address this problem, we here aimed to achieve online decoding of unprompted moments of recognition as well as the subjects internal value judgements of each recognized entity. In short, we wanted to decode when the subject noticed something, and how they felt about it. This is a fundamental first step in creating automated, interactive, and intelligent human machine teaming. We here present a novel method of decoding moments of recognition and their associated internal value judgments in the context of highly complex reverse engineering tasks. This is done through a combination of P300 detection and the Engagement Index to determine whether an item has been identified as relevant to the task (to be further explored) or irrelevant to the task (to be quickly ignored). P300s are a neural signature of novelty detection, and can be classically evoked in both Go and No Go conditions - i.e., both when a Target stimuli is presented, as well as when a Distractor stimuli is presented. Go and No Go P300 waveforms are very similar, and are usually differentiated only post-hoc if at all. Go & No Go P300s were here evoked in each subject using a classic Auditory Oddball test; Artificial Neural Networks were then trained to identify P300s in each subject individually. These personalized networks were used to identify P300s in each subject during reverse engineering tasks. Dimensionality reduction of neural data during the tasks showed the existence of separately clustering subgroups of P300s; these subgroups specifically showed differences in Engagement Index, a ratio of neural power bands that has been linked to subject engagement in the task at hand. In the case of reverse engineering tasks where the goal is to find vulnerabilities in the code, this is interpreted as noticing something relevant to identifying a vulnerability in the code (P300 + High Engagement), or noticing something that allows you discard sections of code as likely not containing a vulnerability (P300 + Low Engagement). Subgroups of P300s differentiated by High vs. Low Engagement were further verified as distinct groupings with pupil dilation and user behavior metrics. This decoded information can be used to modify the task interface to aid in the reverse engineering process. Specifically, sections of code that trigger a Go response (P300 + High Engagement) can be highlighted in yellow while sections of code that trigger a No Go response (P300 + Low Engagement) can be greyed out. This allows for cognitive offloading of the user’s own decision making onto the visual interface in a completely automated and personalized fashion. Online detection of Go vs. No Go P300 events represents a significant advance in domain of real-time neural decoding, and opens up many further possibilities for usage in a broad range of intelligent human systems integration applications.

Keywords: Human Machine Teaming, Neural Decoding, Brain Computer Interface

DOI: 10.54941/ahfe1004481

Cite this paper:

​