Neuroergonomics and Cognitive Engineering

AHFE International

Accelerating Open Access Science in Human Factors Engineering and Human-Centered Computing

Neuroergonomics and Cognitive Engineering

book-cover

Editors: Alex, er M. Yemelyanov, Lisa Jo Elliott

Topics: Neuroergonomics and Cognitive Engineering

Publication Date: 2024

ISBN: 978-1-964867-02-1

DOI: 10.54941/ahfe1004732

Articles

Using Cardiac and Electrodermal Activity as Cognitive Markers for Interruptions and Distraction in a Surveillance Simulation

Security surveillance is frequently used to increase public safety. Characteristics of the surveillance rooms, however, pose many cognitive challenges pertaining to distraction and interruptions, which may affect surveillance performance. Affective computing could represent a potential solution. It involves the recognition and the interpretation of human states using, for instance, different psychophysiological measures. As a first step toward this goal, the present study aimed at assessing whether cardiac and electrodermal activity, could be used as potential markers of interruptions and distraction during a surveillance simulation. A total of 126 participants went through a simulation involving four 8-min scenarios using a high-fidelity urban security surveillance microworld. Task interruption in the form of a realistic secondary task to perform and distraction in the form of background noise representative of a busy operational centre were also implemented into the simulation. Different features of the electrocardiographic (ECG) signal varied with the presence of distraction, but also as a function of time on task. Electrodermal (EDA) features mainly varied as a function of time. These results suggest that distraction and time on task specifically impacted cognitive functioning, potentially increasing sympathetic activity through cognitive workload, and that EDA and ECG measures may represent relevant markers to use from an affective computing perspective to particularly pinpoint periods of distraction and hypovigilance. Implications for the development of user-adaptive systems are discussed.

Alexandre Marois, Damien Mouratille, Yvan Pratviel, Cindy Chamberland, Sébastien Tremblay

Conference Proceedings

Real-Time Machine Learning for ICU Hypoxia Prediction: A Pilot Study

Continuous and precise monitoring in the intensive care unit (ICU) are essential, particularly for patients with respiratory disorders necessitating mechanical ventilation, as they represent a critical cohort. Hypoxia can be defined as a medical condition characterized by an inadequate supply of oxygen to bodily tissues and organs, leading to an inability to meet their metabolic needs. This condition can manifest in diverse scenarios, such as high-altitude environments for example mountain climbing or in high-altitude flights. Common symptoms of hypoxia include breathing difficulties, confusion, elevated heart rate, and impaired cognitive and physical functions. If left unaddressed, hypoxia can lead to tissue damage, organ failure, and, in severe cases, even death. Traditionally, hypoxia detection relies on post facto measures, with methods implying peripheral capillary oxygen saturation (SpO2) providing valuable but delayed insights. Models providing real-time levels of hypoxia, or even early detection and intervention, would thus be relevant to prevent such a state. They could in fact provide timely detection to trigger automatic ventilation or to alert healthcare personnel promptly via adaptive automation. The goal of this study was to produce a real-time hypoxia detection model using machine-learning techniques in the context of ICU. Methods. We used the open-source eICU database, consisting of critically ill patients treated in ICU across the United States. It contains vital signs recorded at 5-min intervals and many hypoxia events. We utilized parameters according to a feature selection such as SpO2, heart rate (HR), and respiratory rate (RR) to make hypoxia predictions from few prior lags. We selected three hypoxia levels to achieve a supervised learning classification for hypoxia: No Hypoxia (SpO2 > 93%), Low level of Hypoxia (88% < SpO2 < 93%), and Strong level of Hypoxia (SpO2 < 88%). Furthermore, we exported the trained model to the Open Neural Network Exchange (ONNX) format to facilitate real-time predictions deployment in clinical settings, thus offering a valuable tool for early hypoxia detection and proactive patient care in ICU.Results. Mann-Whitney U tests and paired t-tests raised significant differences across levels of hypoxia for the SpO2, HR and RR measures. We tested several machine-learning models and our results showed that a Random Forest model could provide accurate predictions of impending hypoxia events (5 minutes prior to the event). We performed a random search fine-tuning method and a group 5-fold cross-validation and achieved a predictive accuracy of 0.937, a precision rate of 0.85, and a balanced accuracy of 0.813. The ONNX implementation of the model had an inference of 1 millisecond, which allows real-time hypoxia prediction. Discussion. Hypoxia is frequently encountered in the ICU as a result of a wide range of pathologic characteristics. Therefore, detecting hypoxia events before they occur is of paramount importance to ensure timely countermeasures through adaptive automation and prevent fatal outcomes for patients. The use of an ONNX model further enables real-time predictions, enhancing the clinical utility of our approach. The versatility of this approach extends beyond the medical domain, for example in aviation where hypoxia poses a serious threat. Such models have potential for being integrated in real time for pinpointing instances of pilots hypoxia, thus contributing to their health and to overall flight safety.

Victor Niaussat, Raphaelle Giguere, Tanya Paul, Patrick Archambault, Alexandre Marois

Conference Proceedings

NeuroTeaming: Using Power Spectral Density for Adjusting Teaming Dynamics in Pilot-AI Task Allocation

Human-autonomy teaming (HAT) is becoming a subject of high interest in the human factors literature. It has several applications, including the collaboration between a human and an autonomous unmanned aerial vehicle (UAV) for security and defence use cases (e.g., for search and rescue tasks). This work is focused on methods for task-allocation between human and autonomous UAV agents. The proposed approach is human-centred, using a coactive design framework which relies on enabling adaptive team dynamics where different agents might act as key players for specific tasks based on an interdependent relationship. This method helps solve complex issues in understanding and adjusting to complementary team dynamics where agents might have different skill levels, experiences, roles, and helps understand which agent is more competent to perform a task. Additionally, such a framework promotes transparency towards the control and task-allocation strategies. To demonstrate this task-allocation strategy, this study looked at the use of neurophysiological features as indicators of task-specific capacities in UAV operations, more specifically electroencephalogram (EEG) signals, which opens up for the development of task-allocation adaptive systems, dependent upon variations in brain activity. Results found that EEG spectral power bands have potential to help determine different task-based abilities across groups (i.e., obstacle avoidance vs. target identification), hence contributing to pinpointing variations in the type of autonomous support needed. Overall, this research explores how task-dependencies can be observed through EEG signals for better transparency and explainability of adaptive control in pilot-AI teaming.

Tanya Paul, Daniel Lafond, Alexandre Marois

Conference Proceedings

Training stress models on open-access data for a continuous human state monitoring platform

Stress can be an indicator of discomfort with a task, which is of relevance for training in safety-critical fields. Knowing a trainee’s stress level could be especially useful when objective performance outcomes are unclear or when success in training tasks alone is insufficient to predict proficiency in real-life safety-critical scenarios. In this study, stress classification models trained on open-access physiological data and integrated in Sensor Hub, a multi-sensor system for near real-time monitoring, were developed. To obtain ground-truth neurophysiological data recorded under high-stress conditions, raw electrocardiogram (ECG) and respiration data in an open-access database sourced from PhysioNet, consisting of 57 participants with arachnophobia watching spider videos, was used. Machine learning algorithms were trained on features extracted from these raw signals. A first set of algorithms focused on heart rate, respiratory rate, and heart rate variability (HRV) features. The second set included feature normalization according to an individual’s baseline. Models based on individually normalized features reached balanced prediction accuracy >80%. A pilot data collection was conducted with a different sensing device than the device used to obtain these measures. Qualitative analysis revealed that real-time R-R intervals from the new sensors were sensitive to artifacts, suggesting that the model relying on HRV features may not be reliable. The model that used only the baseline normalized heart and respiratory rate was selected as the final choice, exported in the Open Neural Network Exchange format and integrated into the Sensor Hub platform, providing predictions every second. This research demonstrates the potential of open-access data for providing a solid starting point for training cognitive models, while also highlighting the necessity of real-time testing to confirm that models can generalize across different sensors and processing pipelines.

Danielle Benesch, Tanya Paul, Alexandre Marois

Conference Proceedings

Validation of Vigilance Decline Capability in A Simulated Test Environment: A Preliminary Step Towards Neuroadaptive Control

Vigilance is the ability to sustain attention. It is crucial in tasks like piloting and driving that involve the ability to sustain attention. However, cognitive performance often falters with prolonged tasks, leading to reduced efficiency, slower reactions, and increased error likelihood. Identifying and addressing diminished vigilance is essential for enhancing driving safety. Neuro-physiological indicators have shown promising results to monitor vigilance, paving the way for neuroadaptive control of vigilance. In fact, the collection of vigilance-related physiological markers could allow, using neuroadaptive intelligent systems, a real-time adaption of tasks or the presentation of countermeasures to prevent errors that would ensue from such hypovigilant situations. Before reaching this goal, one must however collect valid data truly representative of hypovigilance which, in turn, can be used to develop prediction models of the vigilant state. This study serves as a proof of concept to assess validity of a testbed to induce and measure vigilance decline through a simulated test environment, validating controlled induction, and evaluating its impact on participants’ performance and subjective experiences. In total, 28 participants (10 females, 18 males) aged 18 to 35 (M = 23.75 years), were recruited. All participants held valid driving licenses and had corrected-to-normal vision. Data collection involved Psychomotor Vigilance Task (PVT), Karolinska Sleepiness Scale (KSS) and the Stanford Sleepiness Scale (SSS) along with neuro-physiological specialized equipment: Enobio 8 EEG, Empatica E4, Polar H10 and Tobii Nano Pro eye tracker. Notably, this study is limited to demonstrating the results of PVT, KSS, and SSS, with the aim of assessing the effectiveness of the test setup. Participants self-reported their loss of vigilance by pressing a marker on the steering wheel. To induce hypovigilance, participants drove an automatic car in a low-traffic, monotonous environment for 60 minutes, featuring empty fields of grass and desert, employing specific in-game procedures. The driving task included instructions for lane-keeping, indicator usage, and maintaining speeds of up to 80 km/h, with no traffic lights or stop signs present. Experiments were conducted before lunch, between 9 am and 12 pm, ensuring maximum participant alertness, with instructions to abstain from caffeine, alcohol, nicotine, and cannabis on the experiment day. Results showed that the mean reaction time (RT) increased from 257.7 ms before driving to 276.8 ms after driving, t = 4.82, p < .0001, d = -0.61 whereas the median RT changed from 246.07 ms to 260.89 ms, t = 3.58, p = 0.0013, d= -0.53 indicating a statistically significant alteration in participant's psychomotor performance. The mean number of minor lapses in attention (RT >500ms) to the PVT increased from 1.11 before driving to 1.67 after driving, but was not statistically significant t = 1.66, p = 0.11, d = -0.28. KSS showed a considerable rise of sleepiness, with a mean of 4.11 (rather alert) before driving increasing to 5.96 (some signs of sleepiness) after driving, t = 5.65, p < .0001, d = -1.04. Similarly, the SSS demonstrated an increase in mean values from 2.57 (able to concentrate) before driving to 3.96 (somewhat foggy) after driving, t = 8.42, p < .0001, d = -1.20, signifying an increased perception of sleepiness following the driving activity. Lastly, the mean time of the first marker press was 17:38 minutes (SD = 9:47 minutes) indicating that the self-reported loss of vigilance occurred during the first 30 minutes of the driving task. The observed increase in PVT reaction time aligns with the declined alertness reported on both the KSS and SSS responses, suggesting a consistent decline in vigilance and alertness post-driving. In conclusion, the study underscores the effectiveness and validity of the simulated test environment in inducing vigilance decline, providing valuable insights into the impact on both objective and subjective measures. At the same time, the research sets the stage for exploring neuroadaptive control strategies, aiming to enhance task performance and safety. Ultimately, this will contribute to the development of a non-invasive artificial intelligence system capable of detecting vigilance states in extreme/challenging environments, e.g. for pilots and drivers.

Andra Mahu, Amandeep Singh, Florian Tambon, Benoit Ouellette, Jean-françois Delisle, Tanya Paul, Foutse Khomh, Alexandre Marois, Philippe Doyon-poulin

Conference Proceedings

Decision-making augmentation system for solving the problem of risk reduction

The decision-making augmentation system for solving the problem under risk and uncertainty is demonstrated. This system helps decide on the most satisficing alternative for solving the problem of risk reduction. Satisficing alternative is an alternative that satisfies requirements for risk reduction and is sufficient for the decision-maker. The process of solving the problem is self-regulating, where the problem goal, initially set up as an uncertain “sufficient risk reduction”, should be clarified in the process of problem-solving to reflect the formation of the mental model, while the activity goal should be accordingly modified by adding corresponding objectives as criteria for success to reflect the formation of the level of motivation. This iterative process ultimately leads to the most satisficing solution to the problem. Given human limitations in computational capacity due to the size of working memory, the augmentation system supports computation on various levels, encompassing motivation, self-efficacy, and risk reduction. This system is implemented in ED2® mobile web apps, addressing both reactive and proactive risk reduction for present or future risk events, respectively.

Alexander M. Yemelyanov

Conference Proceedings

Wide scope of applications of the SSAT toward the optimization of user experience

In this paper we are demonstrating application of the Systemic Structural Activity Theory (SSAT) to a wide range of applications. We will show how SSAT has been applied to the reduction of task complexity, to the enhancement of the efficiency of performance, to the improvement of user friendliness of various applications and to the reliability of performance. Our purpose here is to firstly demonstrate how the founder of SSAT Gregory Bedny applied his theory and demonstrated it efficiency.We will also show how other scientist implemented SSAT in their research in a wide variety of areas such as medicine, decision making, human-computer interaction, Web design, AI, healthcare, etc.Our goal here is also to show that SSAT has great potential for future application in a variety of research areas.We, for example are considering to apply this theory to the study of the user friendliness of cell phone apps that have their own specific restrictions and users’ demands. SSAT is a high-level generality theory. There are a number of methods of task and human performance analysis that have been developed within this framework. Our paper will briefly describe these methods and mention examples of utilizing these methods in order to enhance product quality, improve user experience, minimize human errors and increase the probability of successful results of human activity in a variety of areas of human activity demonstrating that behavior action require mental efforts for their regulation. SSAT provides a deep understanding of the relationship between external motor and internal cognitive action that are basic elements of human activity. Suggested by SSAT methods can be applied to traditional and computer based human activity.This paper will also discuss motivational aspects of human activity and the role they play in success or failure of an enterprise. We will touch on such aspects of human activity as its complexity and difficulty and their correlation with motivation. SSAT pays special attention to the cognitive components of human activity and demonstrates the methods of making the activity less complex and reducing the memory load. These aspects are equally important in both production and non-production environments.The above-mentioned methods can be successfully utilized in the design of a variety of products from apps and AI to kitchen appliances because they allow to take into account human emotions, motivation, memory load, usability aspects, etc. They can also be applied to devising the most efficient methods of work or non-work-related tasks. We will discuss the analytical level of product design and the implementation of the suggested methods to the training process.This paper will cover the existing applications of SSAT and the opportunities of using its methods in the future research.

Inna Bedny

Conference Proceedings

On the Theory of Regulating Educational and Professional Activity

This paper is based on the experimental studies that demonstrate that motivational and target-oriented component are important in the structure of educational and emotional regulation of professional activity.It is known that the external motive of activity does not fully correspond to the cognitive process. This motive is mainly aimed at the final result, that is, the desire to avoid a bad assessment or desire for approval. Therefore, the emotions accompanying it, in this case, reflect the degree of relationship in the form of success or failure in the implementation of the chosen actions. A more complex structure has intrinsic motivation, which coincides with the target of educational and professional activity. The most generalized forms of manifestation of such activity are success or failure. The latter performs the function of a signaling device about the adequacy or inadequacy of intermediate performance results. Emotions are selective in nature, since they are associated with the intermediate results of the activity and are related to the achievement of its final product. They act in the form of a given product of activity, where the impulse arises from the qualitative and procedural motivation of the subject of activity, as a developing system. As we can see, there is a variety of targets that specify the development of the motive. At the same time, the emerging emotions are aimed not only at intermediate results, but also at significant connections and internal patterns of the subject content of cognitive activity. Based on the above-said and other data, the model of the mechanisms of emotional regulation of educational and professional activity can be represented by three phases. In the first phase, emotions reflect the ratio of external motives in the form of success or failure in the implementation of stereotypical action patterns. The nature of emotional manifestations is fixed, as a rule, in pleasure or not pleasure. At the same time, the mechanisms of emotional consolidation or emotional motivation for the target function. In the second phase, emotions reflect the relationship between internal motives and the final result or product. The states of success and failure that arise in this case, in contrast to satisfaction, perform the function of internal assessments of the adequacy-inadequacy of achieving intermediate results of the target. A specific mechanism of emotional regulation is the induction function, which manifests itself in confidence or doubt. In the third phase, emotional regulation characterizes the relationship between internal motives, against the background of a qualitatively developing subject content of educational and professional activity. All this allows here to apply the mechanisms of emotional regulation in the form of inductions or emotional evaluation of the result. The described functions in the conditions of real emotional regulation of educational and professional activity can be expanded: give incentive, activating, reinforcing, etc. at the first stage; reflective, selective, etc. at the second and heuristic, coordinating, and sanctioning at the final one.

Oleksiy Chebykin, Marianna Skoromna, Inna Bedny

Conference Proceedings

Workers’ Modes of Self-Expression and Behavioural Manifestations of Loyalty or Exit-Intentions When Engaged in Systemic Structural Activities

The purpose of this paper was to understand how employees’ modes of self-expressions affect their behavioural manifestations of loyalty or exit-intentions, when engaged in systemic strucutural activity and the influence of such activity on a firm’s organizational tolerance. The need for the study was informed by the conception that historicity of an individual’s self-regulation system relates with the individual’s subjective perception of complexity that influence the individual’s activity goal formation, which has implications on the person’s modes of self-expression while involved in a systemic activity as well as on organizational tolerance. Guided by Bedny and Karwowski's well-established knowledge that activities of individuals are realized by goal-directed actions, informed either by mental or motor conscious processes, as objects of the cognitive psychology of skills and performances, an attempt is made to understand the significance of workers mode of self-expressions on organizational tolerance in the different work setting of mining support firms in Ghana. This is based on the premise that the discovery of goals is essential to true activity, and the goals, being discrete elements of activities, can be transformed into contradictions, which may influence employees modes of self-expressions relative to prevailing organizational tolerance, and which contradictions can be expanded and generalized into a qualitatively new organizational activity structure and systemic activity contexts. Thus, building on the notion that an individual’s self-regulation system takes shape and gets transformed over lengthy periods of time, with its problems and potentials being understood only against its own history, the argument that an individual’s mode of self-expression may result in his/her (in)ability to accurately align with an organizational tolerance is explored conversely. Using the survey approach, a questionnaire was developed from standardized measurement scales and used to collect quantitative data from two hundred employees of mining-support firms. The data was analyzed using both descriptive and inferential statistics. It was found that employees’ active- and passive-constructive voices positively correlated with their behavioural manifestations of loyalties while their active- and passive-destructive voices positively correlated with their behavioural manifestations of exist-intentions. Organizational tolerance moderated the relationships between the employees’ active-constructive and passive-constructive voices, and their job behavioural manifestations of loyalty, but did not moderate the relationship between the employees active-destructive voice and their behavioural manifestations of exist-intentions. This study is the first to be carried out in the mining sector in Ghana and the findings provide useful insight toward improving the management of employees. The insights provided will enable managers in mining-support firms in Ghana develop organizational tolerance for managing all employees’ voice-types effectively to enhance their employees’ happiness and productivities in the work environment.

Mohammed Aminu Sanda

Conference Proceedings

An Introduction to Single-Case Experimental Designs for Applied Human Factors and Ergonomics

Experimental designs help human factors and ergonomics (HFE) scientists and professionals make decisions about the causal effects of interventions on measures of human cognition, emotion, and performance. HFE researchers have typically used traditional between-subjects, within-subjects, and mixed experimental designs to do so. Although these designs will continue to play an important role in HFE research, some research questions and applied problems do not easily lend themselves to the use of these designs. This is particularly true when a study focuses on the performance of single individuals or two or more individuals performing as a single unit, and/or researchers find it difficult or impossible to obtain enough individuals from the population of interest to achieve sufficient statistical power for traditional experimental designs. In these situations, single-case experimental designs (SCEDs), can offer effective and flexible alternatives to traditional experimental designs. In this paper, we describe the general characteristics of SCEDs and the two most common designs, withdrawal and multiple-baseline designs using HFE examples. SCEDs have demonstrated potential to identify effective interventions for individuals in a variety of domains and contexts relevant to HFE.

Sean Laraway, Susan Snycerski, Sean Pradhan, Bradley Huitema, William Rantz, Geoffrey Whitehurst, Vernol Battiste

Conference Proceedings

Measuring Flow: Perceived Emotions & Arousal-Valence

Deep focus states, like Immersion and Flow are important parameters when it comes to an enjoyable experience during learning activities. To measure these mental states usually self-assessment questionnaires, answered by the subject after the experience, are used. Because of the shortcomings of this method, the ultimate goal is to establish an alternative measuring method through correlations of physiological sensor data. Exploring the Physiology of deep focus, in the course of prior studies, physiological data of participants was recorded during activities and inspected for correlations with Flow and Immersion. By broadening the experimental scope, this paper explores the effect of participant's emotions on reaching states of deep focus.

Ehm Kannegieser, Johannes Ratz

Conference Proceedings

Evaluation of Voice vs. Text Communication Modes in Simulated UAM Operations.

Urban Air Mobility (UAM) is a system that is expected to operate within and around metropolitan environments, utilizing electric, vertical takeoff and landing (e-VTOL) aircraft, to create on-demand, highly automated passenger and cargo-carrying air transportation services. Many stakeholders are developing such systems, although several barriers to UAM operation remain. Two barriers being addressed in our simulation facility are pilot training and air traffic management operations. Although the UAM industry is focused on autonomous operations, the initial UAM operations will have ground or onboard pilots, yet the forecasted pilot shortage will be problematic. Completely autonomous systems will result from an evolution, of pilot-based operations, to onboard operators flying with automated assistance and finally completely autonomous vehicles. We are currently investigating and evaluating concepts for simplified vehicle operations and air traffic management using a virtual UAM eVTOL vehicle in a CAVE virtual environment. In the first test of the UAM vehicle (e.g., Strybel et al., 2021) pilots and reported that the simulation was sufficiently realistic for tests of UAM operations. Here we report on a subsequent investigation of communication modes for pilots flying UAM routes over the San Francisco metropolitan area. The routes consisted of stops at six vertiports, either at airports or other locations, for picking up/dropping off passengers. UAM pilots communicated with air traffic control and vertiport managers using either voice or text messaging. Voice communications were consistent with current day air traffic control operations. Text communications were via a custom message pad application that enabled standard messages to/from ATC (requests and responses) via touch input. Six certified pilots flew two routes using each mode of communication. We evaluated the impacts of these communication modes on pilot performance (flight time and communication latency) and subjective responses (workload and subjective feedback). Feedback regarding the messaging application and simulation facilities were also collected and will be described in the presentation.

Thomas Strybel, Vernol Battiste, Kim-Phuong L. Vu, Panadda Marayong, Stacey Ahuja, Maegan Schmitz, Justin Cheung, Chloe Culver, Andrew Alfaroarevalo, Praveen Shankar

Conference Proceedings

A Comparative Case Study of Post-Industrial Regeneration Project Through Digital Footprint

Visitors’ satisfaction appears to be an essential indicator for the success of urban regeneration projects, as the finding reflects people's expectation of regeneration projects, which are the necessary design drivers for project designers in their ideation. The covid-19 pandemic has limited many conventional research approaches, in particular in the human-centred study that requires the face-to-face contact. Digital Footprint is data that is left behind when users have been online. Analysing the visitors’ digital footprint generated by the websites of TripAdvisor and Dianping, this method examines four post-industrial regeneration projects selected from China and the UK, to explore the factors that influence visitors’ satisfaction for regeneration projects within the two cultures. This paper extends the Digital Footprints method to the urban regeneration research, aiming to seek an alternative and practical method for researchers, designers, and project investigators in their future projects.

Xiaochun Zhan, Fang Bin Guo

Conference Proceedings

Impact of Anxiety on Eye Markers: Role of Visual Task Complexity

Anxiety and visual task difficulty may interact, impose cognitive demands, and affect task performance. Researchers have found that when comparing task performance across people with different levels of anxiety vulnerability, people with somewhat elevated anxiety vulnerability perform worse than people with lower anxiety susceptibility. Furthermore, vulnerability to anxiety is associated with a decreased ability to control the allocation of attention that is reflected while performing tasks. We aim to see the role of visual task complexity as potential mediators in the relation between different levels of anxiety and eye movements and to see whether increasing task complexity has an effect on visualization behaviour. We use eye-movement analysis to observe and analyze how people view and subsequently process visual information. After the screening, the study recruited 31 students (F= 11; M=20) aged between 21 to 35 years (M=28.02, SD=0.62). Exclusion criteria included individuals with visual impairments, with substance disorders and history of neurological disorders. All participants provided consent before participation. STAI-Y2 was used to categorize participants into low, moderate, and high (N=9, N=10, and N= 12). Visual task involved arithmetic problems and determining the accuracy of the answer displayed (true or false). Each level was divided into two subsets, each containing six equations with a combination of numbers, alphabet characters, and spatial symbols (>, <). To heighten task complexity, one subset required participants to respond synchronously using their right index finger for true and left index finger for false, while the other subset required asynchronous responses (left index finger for true, right index finger for false). Relevant information to solve the problems consisted of numbers and spatial symbols, while alphabet characters were considered irrelevant or noise. The First Fixation Duration (FFD), Time to First Fixation (TTFF), Total Visit Duration (TVD), Total Fixation Duration (TFD) were computed after ascertaining the Area of Interests (AOIs). The findings suggest that individuals with high trait anxiety tend to explore or scan irrelevant information more compared to low and moderate trait anxiety levels while performing the task, particularly during more complex visual tasks. Individuals in the low and moderate anxiety groups showed better ability to fixate on relevant information during the same tasks. The study's findings suggest that eye metrics, such as fixation duration and visit duration on relevant and irrelevant information, could serve as objective markers indicating anxiety related attentional differences during task performance. The results are in line with previous research supporting the correlation between anxiety and eye metrics. Our study corroborates with existing literature, which posits that highly anxious individuals may experience impaired attentional control during task performance. The study provides valuable insights into the impact of anxiety on attention and information processing during visual tasks of varying complexity.

Saurabh Sharma, Sonali Aatrai, Rajlakshmi Guha

Conference Proceedings

Random Gazes, Telling Eyes: Exploring Gaze Transition Entropy as a Performance Indicator in Evaluating Instructional Designs

This study explores gaze transition entropy (GTE) as an objective performance measure in industrial assembly training. GTE can be defined as a metric of randomness in gaze patterns over time, with low entropy indicating predictable and structured patterns and high entropy indicating unpredictable and irregular patterns. Using mobile eye-tracking glasses, 28 participants completed a cable assembly task while gaze patterns were analyzed across specific areas of interest (AOIs). Preliminary findings from six participants show decreased GTE with increased task familiarity and task experience. Additionally, preliminary trends suggest a positive relation between GTE and self-reported hesitation, indicating its potential as an objective gauge of uncertainty. Furthermore, a theoretical variation on gaze transition entropy where the effect of consecutive fixations at the same object is factored out is explored. This research offers insights into the potential of using gaze transition entropy to objectively assess hesitation and proficiency in training, providing a potential avenue for enhancing instructional content within instructional design using objective evidence. Further refinement and exploration of gaze transition entropy could significantly impact training quality assessment across diverse domains and can enable promising applications in other fields as well.

Jonas De Bruyne, Laetitia De Leersnijder, Durk Talsma, Jelle Demanet, Jelle Saldien, Klaas Bombeke

Conference Proceedings

An explorative study of nuclear operators' perception of groupthink

Operators in nuclear power plant (NPP) control rooms work together as a team where each team member has specific roles and areas of responsibility. In some situations, the crew dynamics are poor, and the group's effectiveness will be reduced. A phenomenon that often has been linked to poor crew dynamics is called groupthink. Due to potential negative outcomes of groupthink, it is important to gain more knowledge about the concept. In this paper we ask: How do NPP operators perceive causes, symptoms and consequences of groupthink and possible ways to avoid it? To explore this question, we conducted a study with NPP operators. Findings from the explorative study, as well as recommendations for further work, are presented in this paper.

Magnhild Kaarstad, Robert McDonald, Daniel Odéen

Conference Proceedings

The use of bespoke wearables to investigate neurological and physiological responses to microclimate stressors in quasi-formal academic contexts

Climate change caused by anthropogenic environmental pollution has become one of the most pressing issues of our modern world. For instance, heat waves have been shown to seriously impair students’ health and productivity (Lala & Hagishima, 2023). The general problem of climate change has influenced recent research to focus on redesigning and restructuring the living environment to improve human health and productivity. Yet, according to Palme and Salvati (2021), there have been relatively few studies on the relationships between microclimates and human health and emotions. This is particularly detrimental as the in-depth knowledge obtained can be used to enhance human health and productivity, as well as influence their attitude towards the environment (Doell et al., 2023). This paper reports a study conducted by students as an independent research project under the mentorship of a senior research scientist at the National Institute of Education, Singapore. It represents a multidisciplinary, citizen science and neuroergonomic approach to investigate the relationships between human neuro-physiological health and mental well-being. To investigate both physical health as well as stress, low-cost, bespoken wearables were built, such as a mini weather station and physiological wristband. Electrodermal activity (EDA) was also introduced as a non-invasive method to detect stress and emotional arousal (Rahma et al., 2022) and as a marker of sympathetic network activity (Zangróniz et al., 2017). EDA features such as mean of tonic component and TVSymp (spectral powers in specific frequency bands according to Posada-Quintero et al. (2016a; 2016b) and their normalised versions were focused on as they were found to be highly sensitive to orthostatic, cognitive, and physical stress (Posada-Quintero et al., 2020). PPG was also introduced as a second source of data for analysis of stress and emotions, since it is influenced by the cardiac, vascular and autonomic nervous systems, which are all affected by stress. Machine learning models were trained to investigate relationships between emotional arousal, stress and the surrounding environment. To elaborate, climate change might precipitate changes to microclimates to the extent that for those inhabiting these biomes the changes might be detrimental to physical and mental well-being. Therefore, investigating EDA data may unveil hidden relationships as to how microclimate is related to our perception of well-being at a granular level. In this way, the present study builds on prior work (eg, Lim et al., 2022) that documented changes in microclimate on affective states. It is hoped that analyses of EDA and PPG data will further strengthen the emerging model describing the intersections between local microclimate, physiological stress and emotion. In the present study, we apply this paradigm to the use of EDA in the context of students’ scholastic activity. We seek to understand factors influencing the affective states of learners. Our preliminary findings suggest implications for the design of living and studying conditions with respect to the interaction of microclimate and human health and comfort.

Duc Minh Anh Nguyen, Nguyen Thien Minh Tuan, Kenneth Y T Lim, Hugo Posada-quintero

Conference Proceedings

Influences of Information Processing Modality on Mental Workload Recognition Performance Based on EEG Feature Extraction

Accurate recognition of mental workload is significant for optimizing the human-machine interaction and avoiding the regrade of task performance levels due to overloading or underloading of mental workload. In past studies, the use of Electroencephalogram (EEG) signals has shown high performance in the recognition of operators' mental workload levels, however, most of the studies were conducted using a single visual modality task or dual visual modality tasks. But in real-world operational tasks, auditory-visual modalities tasks are commonly involved, and there have been relatively few researches on the EEG recognition of mental workload levels in auditory-visual modalities tasks. Therefore, in this research, visual single modality task scenario and audio-visual dual-modality task scenario were set up based on simulated flight experiments. For each task scenario, two levels of mental workload were induced by the differences in task complexity. Twenty subjects were recruited and their NASA-TLX scales and EEG signals were collected during the experiment. Two types of feature extraction methods were used, including Power Spectral Density (PSD) and Common Spatial Pattern (CSP), to recognize the mental workload levels. The research results indicated that the information processing modality did not have a significant influence on the performance of recognition for mental workload based on EEG feature extraction.

Sinan Guo, Wanchen Jia, Lin Ding, Chongchong Miao

Conference Proceedings

Assessing the Effects of Fatigue on Cognitive Performance for Shift Workers in the Petrochemical Industry: A Scoping Review

Each industry has its challenges and risk factors for workplace fatigue, shaped by the nature of the work, the work environment, and regulatory frameworks. The petrochemical industry requires shift systems to keep their productions operational continuously. These systems pose significant risks associated with worker fatigue. Workers are frequently exposed to hazardous chemicals, posing a risk to their health and further exacerbating fatigue. It might lead to decreased safety and productivity. To investigate this issue, we conducted a scoping review of literature published according to PRISMA guidelines in the past two decades using Scopus, Web of Science, and Embase databases. The objective is to investigate the question: "What does the current literature indicate about the effects of fatigue during shift work in the petrochemical industry?". Our review highlights the complex relationship between shift work patterns and their physiological and psychological impact on workers in this industry. Our paper aims to provide industry-specific recommendations and interventions that effectively reduce the adverse effects of shift fatigue in the petrochemical sector.

Ashraf Alhujailli

Conference Proceedings

Rapid detection of near-infrared spectral response of neural activity in prefrontal cortex

Due to its non-invasive neuroimaging properties and wide applicability in the field of aviation, fNIRS is chosen as the main tool for studying human factors in aviation. Currently, fNIRS-based brain-computer interface (BCI) and neurofeedback learning systems have a detection time of 4-6 seconds, which does not meet the needs of rapid decision-making by pilots in emergency situations. To shorten the detection time, we look for features to respond faster to subjects' stimuli and improve the feasibility of fNIRS for real-time application in aviation human factors. In this paper, two features of the NIR signal are extracted: the degree of variability feature and the oxygen exchange feature. By calculating the standard deviation, the degree of variability feature is compared between before and after stimulation, and the larger standard deviation indicates the more obvious activation effect of the stimulation on the brain prefrontal. In addition, by assessing the oxygenation between oxygenated hemoglobin and deoxyhemoglobin in each channel, the activation response between different brain regions can be recognized, thus reflecting the occurrence of the stimulus more accurately.that the results shown that the standard deviation of the first-order derivatives of the oxygenated hemoglobin concentration of some of the channels increased compared with the resting state 1.5 s after the stimulus onset, suggesting that the difference between before and after stimulation brain frontal increased. Analysis of the degree of oxygen exchange of channel oxyhemoglobin and deoxyhemoglobin before and after stimulation revealed that in 85% of the trials, the degree of oxygen exchange of certain channels changed significantly between 1 and 2 s after stimulation. Our study suggests that subjects' responses to stimulation and brain prefrontal activity could be detected on the basis of changes in the standard deviation of the first-order derivative of oxyhemoglobin and changes in the degree of oxygen exchange of oxyhemoglobin and deoxyhemoglobin in certain channels within 1 to 2 seconds after stimulation. This finding may enhance the feasibility of NIR imaging in future real-time applications of human factors.

Xiaodan Wang, Tianrui Qi, Yiyuan Zheng, Shan Fu, Yanyu Lu

Conference Proceedings

A potential rapid detection of cognitive status in the brain: an fNIRS study

In the realm of human factors, objective assessment of cognitive states is crucial for the safe completion of tasks. Functional near-infrared spectroscopy (fNIRS) can be employed to recognize the cognitive activities and evaluate the mental workload associated with cognitive-executive processes. This study proposes a method for facilitating the faster detection of changes in cognitive states based on the correlation coefficients of adjacent channels, which enables the extraction of local connectivity (LC) features from fNIRS data. The results indicate that the extracted new features can reflect changes in the activation patterns of specific brain regions during the early stages (0~2.5s) of the task. It is suggested that these features could be used to identify the brain's task states.

Tianrui Qi, Xiaodan Wang, Yiyuan Zheng, Shan Fu, Yanyu Lu

Conference Proceedings

NeuroDesign2.0: A Framework of Visual Perception in Visual Communication

This paper presents NeuroDesign 2.0, an advanced interdisciplinary framework that combines principles of visual design, cognitive process, and general working methods of NeuroDesign to understand the intricacies of visual perception. By utilizing functional near-infrared spectroscopy (fNIRS) and applying poster samples based on the International Typographic Style, our approach aims to clarify the selection mechanisms that explain the visual perception of audiences and influence the effectiveness of design in conveying messages. The study transforms from strategies that focus on the designers to the audiences to discuss the mass reception of visual information and further guide the designers’ practices, Where the audiences’ impression and intuition determine the selection of visual messages. Based on the existing research on design and neuroscience, the study presents a novel experimental method for analysing how the brain perceives visual information. It enhances our comprehension of the complex correlation between human engagement with visual stimuli and the corresponding selection reactions.

Jun Chen, Mengyao Guo

Conference Proceedings

Study of Emotional contagion through Thermal Imaging: A pilot study using noninvasive measures in young adults

Emotional contagion, the process of unconsciously mirroring others’ emotions [6], occurs through various channels including facial expressions, vocal tone, and body language, influencing social interactions and responses to cultural stimuli like music and movies [3], [4], [1]. Facial expressions, analyzed using the Facial Action Coding System (FACS), provide insights into emotional transmission [2]. Thermal imaging, a technique for measuring facial temperature changes, offers a noninvasive method to study emotional responses [5]. However, the facial thermal response to emotional contagion remains understudied. This study aims to investigate how emotional contagion affects facial blood flow among highly emotionally contagious individuals, identified using noninvasive measures. Thermal imaging will capture temperature changes across ten designated facial regions of interest (ROIs), shed-ding light on facial muscle activation. By interpreting temperature variations in these ROIs, researchers seek to understand the physiological processes underlying emotional contagion. Previous studies have shown inconsistent findings regarding facial temperature changes during emotions like fear and joy, highlighting the need for further investigation. This research aims to clarify these discrepancies and advance our understanding of facial thermal responses to emotional contagion, contributing to the broader field of emotion research and potentially informing therapeutic interventions and communication strategies.Initially, Eighteen participants participated in the study. Two groups of standardized emotionally contagious video stimuli (Happy, Fear) were used to induce emotional contagion.The videos started with a one-minute relaxing clip to help participants achieve a neutral emotional state before watching the emotional contagion clips. Following the two-minute emotional contagion video, a blank screen was displayed for one minute to observe the aftereffects of the emotional contagion on participants. Facial temperature was recorded from Fluke Ti 400, and facial expressions were recorded from the webcam. Participants were asked to fill out an emotion-intensity feedback form to rate the experienced emotion and its intensity during video stimuli. Eight participants’ data was removed from further analysis because of inconsistencies. Out of the remaining ten, we further shortlisted five highly emotionally contagious participants with the help of the emotional contagion scale. Ninety baseline and arousal thermal images (10 seconds each) were identified and analyzed using FACS. Ten important regions of interest(ROIs) were selected for facial thermal variations. The interpretation of temperature patterns on selected ROIs produces a physiological time series signal, reflecting changes in blood flow associated with emotional responses. As previously discussed, blood flow radiates across the blood vessels when an emotion happens, which is why a gradual shift in the baseline occurs when an emotion takes place. To assess significant differences in facial thermal temperatures from baseline to emotional contagion, the Mann-Whitney U test and average temperature differences were used. During both emotions (fear and joy), the temperature of the nose decreased on the faces of participants. However, during fear, the temperature dropped in the forehead, left eye corner, and right cheek, while during joy, it increased in the left eye upper region. Additionally, while in fear, the left eye upper, right eye upper, and nose exhibited decreased temperatures, whereas during joy, the forehead, left and right eye corners and nose showed reduced temperatures. Mann Whitney U test showed significant emotional arousal in all the ROIs. Only the right eye corner and left cheek in two participants during fear and the right eye corner during joy in one participant was showing insignificant differences.[1] Amy Coplan. Catching characters emotions: Emotional contagion responses to narrative fiction film. Film Studies, 8(1):26–38, 2006.[2] Paul Ekman. Facial expression and emotion. American psychologist, 48(4):384,1993[3] [3]Carolina Herrando and Efthymios Constantinides. Emotional contagion: a brief overview and future directions. Frontiers in psychology, 12:2881, 2021[4]Giuliana Isabella and Hamilton C. Carvalho.Chapter 4 - emotional contagion and socialization: Reflection on virtual interaction. In Sharon Y. Tettegah and Dorothy L. Espelage, editors, Emotions, Technology, and Behaviors, Emotions and Technology, pages 63–82. Academic Press, San Diego, 2016 [5]Sophie Jarlier, Didier Grandjean, Sylvain Delplanque, Karim N’diaye, Isabelle Cayeux, Maria Ines Velazco, David Sander, Patrik Vuilleumier, and Klaus R. Scherer. Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2:2–9, 2011.Eliska Prochazkova and Mariska [6]E. Kret. Connecting minds and sharing emotions through mimicry,Neuroscience Biobehavioral Reviews2017

Prachi Joshi, Hirak Banerjee, Avdhoot V Muli, Aurobinda Routray, Priyadarshi Patnaik

Conference Proceedings

Comparison of Alpha Waves and SSVEP Based on Ear-EEG Using Conductive Paste and Gel Sheet

The ear-electroencephalogram (ear-EEG) method is used to measure brain activity from regions around ear. It is becoming a popular EEG measurement method due to its characteristics such as wearability, simplicity, and long term measurability. In our previous study, we developed and evaluated a brain-computer interface (BCI) by means of steady-state visual evoked potential (SSVEP) measured via ear-EEG. Some ear-EEG-based SSVEP-BCIs have demonstrated practical performance in assisting people with disabilities in various ways. However, the EEG measurement electrodes used in these devices establish contact with skin through conductive paste that stains subject’s hair and skin. This impairs the simplicity of measurement, impacting the advantages of using ear-EEG. Hence, in this study, we measured and evaluated SSVEP and alpha waves by using electrodes coated with conductive gel sheets. A total of 20 channels of electrodes were installed around the subject’s left and right ears, of which EEG components of 10 channels each were made of conductive paste and gel sheet. While the SSVEP and alpha waves were detected by using both conductive paste and gel sheet as the electrode-skin interface, the electrodes attached to the skin with conductive paste showed better detection performance. This indicates that an ear-EEG-based BCI system can be constructed by using conductive gel sheets instead of conductive paste as the electrode-skin interface. In future studies, we aim to improve the signal detection performance of electrodes by using conductive gel sheets as the electrode-skin interface, subsequently developing SSVEP-BCI systems that are implemented for the progress of society.

Sodai Kondo, Hisaya Tanaka

Conference Proceedings

Experimental Paradigms and Their Knowledge Associations in Studies of Information Processing: A Review

Experimentation is the foundation of empirical science and is an important research method of mental cognition. The experimental paradigm, as an important part of experimental research method, is a relatively fixed task program, which is mainly used to describe certain psychological phenomena clearly and accurately. However, the application of experimental paradigms is characterized by mixed task content and complex paradigm type. This paper reviews typical experimental paradigms which are used to study information-processing such as perception, attention, working memory and long-term memory, in terms of paradigm origins, task procedures, research results and development. Based on the review of various experimental paradigms, the knowledge relevance between them are further sorted out in the aspect of task procedure characteristics and research trend. It provides reference for the application and design of experimental paradigms in studies of information processing.

Lan Zhang, Xiaoli Wu, Biao Yan, Qian Li, Xingcan Yang

Conference Proceedings

The Changing Rule and Intervention Strategy of Long-voyage Crew's Alertness

In order to reveal the change law of long-voyage crew alertness in the time dimension of voyage, determine the best time for long-voyage crew alertness intervention, and design the intervention strategy to improve long-voyage crew alertness, this study designed and carried out a series of long-voyage simulation experiments based on the needs of long-term voyages. The results of the study found that the rhythm of the simulated crew is disturbed and the best and worst changes are reversed on the prometaphase of the mid-long-term voyage, which may mean rhythm disturbance is one of the reasons for the alertness sharp declining of the crew. The ERPs latency of target locking the alertness test has no significantly different in the before, middle and after stages on mid-short voyage, that is, the time of detecting target is no delay, but the response time increases, indicating that the decrease of alertness of the crew on mid-short voyages may be related to the decrease of response ability. Based on the above results, the intervention timing for alertness was determined, intervention strategies were designed, and the effectiveness of intervention strategies was verified through experiments.

Jin Liang, Cong Peng, Si Li, XIN WANG, ZHEN LIAO, Ye Deng, Yang Yu, Liang Zhang, Xiaofang Sun, Yulin Zhang

Conference Proceedings