Neuroergonomics and Cognitive Engineering

AHFE International

Accelerating Open Access Science in Human Factors Engineering and Human-Centered Computing

Neuroergonomics and Cognitive Engineering

book-cover

Editors: Frederic Dehais

Topics: Neuroergonomics and Cognitive Engineering

Publication Date: 2023

ISBN: 978-1-958651-78-0

DOI: 10.54941/ahfe1003998

Articles

Applying Smart Assistants in Express Decision for Insurance Choices

In this work, the self-regulation model of decision-making is further expanded to help Express Decision apply voice smart assistants to provide a service through a particular version of Express Decision in insurance (ED2-Insurance-Choice) when deciding which insurance policy to buy. We demonstrate that with the help of Express Decision, existing smart voice assistants like Alexa can be used more efficiently, specifically when setting goals. They can support instrumental rationality of the self-regulation model of Express Decision not only by voice recognition, but also by recognizing intuition as an inner voice.

Sai Mukkavilli, Alexander M. Yemelyanov, Rahul Sukumaran

Conference Proceedings

Application of Systemic Structural Activity Theory to Web Design

In this paper we are demonstrating application of the Systemic Structural Activity Theory (SSAT) to the design of the WEB pages. These days every company has a WEB presence. The Websites come in all shapes and sizes.In order to demonstrate how SSAT theory and methods can be utilized in the design of WEB pages we choose two websites of similar businesses in order to compare them and show how application of SSAT makes it easy to identify which version is a better choice for its users. The advantage of the SSAT methods is that they are formalized and allow to perform the step-by-step task analysis at the design stage. One of such methods is the probabilistic event tree. It allows to analyze the task and determine if the task flow leads to the successful outcome or failure. And in the latter scenario the user has to start over.The other SSAT method is to create the human algorithm of the task. Such algorithm provides the performance time and the task flow and demonstrates the probabilities of various versions of the task. This information is then utilized to determine complexity of the task performance. The higher the complexity of the task is the higher is the probability of the human errors. The preferable design of any website is the one that is the least complex for the users and allows the easy access to the most commonly used relative sources of information. The main objective of this paper is presenting the analytical methods of SSAT and their application to WEB design. The existing methods outside of SSAT consist mostly of observation and experiments. However, observation and experiments are not always available and efficient, especially for new design. Such methods are not sufficient for prediction of the efficiency of the end product. The concept of design is substituted in ergonomics by experimentation which reduces the external validity of the task analysis. Website users are dealing with tasks that include numerous decisions and are exceedingly variable. So, cognitive components of activity are extremely important. Streamlining of web design and improving the user experience is vital considering that such users come in all ages and have various levels of web user experience. This paper will show the way to minimize abandoned actions and improve the user experience when they are in a process of looking up information on the company website or are trying to place an order. We will show how the well-designed application saves customers time and benefit the company by saving the resources and attract the customer that are looking for the fast and easy access to what they are looking for.

Inna Bedny

Conference Proceedings

Self-Regulation Problem Solving for Sufficient Risk Reduction

This paper proposes the self-regulation model (SRM) for sufficient risk reduction, which is based on the self-regulation model of the thinking process developed within the systemic-structural activity theory. SRM includes two sub-models: formation of mental model and formation of the level of motivation, as well as the regulation of their interaction by using feedback and feedforward controls. Feedback control is regulated by the factor of difficulty, and feedforward control is regulated by the factor of significance. With instrumentally rational goal setting, where “reduce risk sufficiently” is an uncertain goal, self-regulation helps the individual apply their personal beliefs and experiences to find a sufficient solution to the problem. We demonstrate how SRM is implemented in ED2-CPR-Choice, a web application designed for people with serious illness to help them decide whether to attempt CPR.

Alexander M. Yemelyanov, Alina A Yemelyanov

Conference Proceedings

Probabilistic predictive modeling in the critical human-in-the-loop (HITL) ergonomics engineering problems

Improvements in ergonomics engineering can be achieved through better work environment and other traditional efforts that directly affect human behaviors and performance. There is also a significant potential, however, for the improvement in ergonomics engineering tasks and problems through better understanding the role that various uncertainties play in the planner’s and operator’s worlds of work, when never-perfect human, never failure-free equipment and instrumentation, never hundred-percent-predictable response of the object of control (such as, say, car, train, or air- and spacecraft), and uncertain-and-often-harsh environments that contribute jointly to the never-zero likelihood of a mishap. By employing quantifiable and measurable ways of assessing the role and significance of such uncertainties and treating a human-in-the-loop (HITL) as a part, often the most crucial part, of a complex man-instrumentation-object-of-control system, one could improve dramatically the state-of-the-art in assuring success and safety of an ergonomics system. This can be done by predicting, quantifying and, if necessary, even specifying an adequate probability of a possible mishap. Nothing and nobody are perfect, of course, and the difference between a highly reliable ergonomics object, product, performance or a mission and an insufficiently reliable one is “merely” in the level of the never-zero probability of failure. Application of the probabilistic predictive modeling (PPM) concept provides a natural and an effective means for reduction failures. This is not the current practice though. The application of the PPM concept can improve the state-of-the-art in understanding and accounting for the human performance in a particular ergonomics undertaking. While the traditional statistical human-factor-oriented approaches are typically based on experimentations followed by statistical analyses, the PPM concept is based on, and starts with, physically meaningful and flexible probabilistic predictive modelling followed by highly focused and highly cost-effective experimentations geared to the chosen governing model(s). The PPT concept enables quantifying, on the probabilistic basis, the outcome of a particular HITL related ergonomics effort, situation or a mission. If the predicted outcome, in terms of the most likely probability of the operational failure, is not favorable enough, then an appropriate sensitivity analysis (SA) based on the developed and available algorithms can be effectively conducted to improve the situation. There are quite a few publications of the authors on the theme of the suggested presentation.The Systemic Structural Activity Theory (SSAT) is another tool that allows to analyze human performance and predict the probability of successful outcome not by just analyzing the existing software or equipment but to do it at the design stage. Application of SSAT improves efficiency and productivity and saves resources by making the design flows apparent at the early stages of the process.

Ephraim Suhir, Inna Bedny

Conference Proceedings

Validity and rationality of using neuroergonomics concept in exploring worker mental issues in systemic-activity theoretical research

It is known fact that the brain is the most complex organ in the human body. Over the last few decades, mapping of the human brain connectivity to human activity has gained considerable attention not only in the areas of neuroscience and cognitive neuroscience, but also in the field of human factors and ergonomics. The field has benefitted greatly from the inclusion and integration of neuroscientific methods and theory, with the argument that synergistic success of such integration could work in the other direction with the inclusion of neuro-field methods and theory of human factors, such as neuro-psychology or neuroergonomics., which incorporates knowledge on workload measures and theory. Thus, the field of human factors and ergonomics has benefitted from the committed inclusion of neuro-based methods and techniques, and it continues to develop and advance in a variety of interesting ways. In this wise, continuous efforts in the neuroergonomics field have been devoted to studying brain signals relative to human systemic activity at work and in everyday settings. Though the number of useful analytical approaches used in neuroergonomics research has rapidly expanded, there is the argument that the functional brain connectivity and network topology in the context of neuroergonomics is largely unknown. Hence, modern network science, entailing a synergetic mix of dynamic systems theory, graph theory, and statistics, is applied in studying the functional and structural brain connectivity network under various states and conditions. Such synergistic relationship is deemed to work in the reverse direction, with methods and measures of human factors and neuroergonomics benefitting other disciplines, such as the systemic structural activity theory (SSAT) approach. SSAT establishes that knowledge derived from ergonomics and activity theory is uniquely capable of engaging with different ways of knowing the world of work, generating new knowledge, and helping stakeholders understand and incorporate the results or lessons learned. Even though previous studies have succeeded in quantifying a great variety of cognitive and physical measures of human tasks, the SSAT approach has been used to understand the mental and physical systemic activities entailed in human dynamic temporal interactions during everyday tasks. This therefore brings to the fore the debate on the validity and rationality of using neuroergonomics concept in exploring worker mental issues in systemic-activity theoretical research. In neuroergonomics studies using the SSAT approach, mental workload is a multidimensional construct and widely invoked concepts, whose assessment has been of great interest. In the SSAT approach, the neuro-indices of cognitive workload have been discussed in the context of human mental load and working memory related to the process of storing and processing information, and which in the workplace require the manipulation and recall of information for decision-making and problem-solving. In this wise, this paper will argue on the validity and rationality of using neuroergonomics concept in the SSAT approach, which has been used in many situations to establish the relation between worker ability to recall and store information to fatigue, stress, and workload, which in turn affects attention levels, situational awareness, and learning performance.

Mohammed Aminu Sanda

Conference Proceedings

The contribution of Gregory Bedny's systemic-structural activity theory to the science of activity

In this paper we will make an attempt to present a brief overview of General, Applied and Systemic-Structural Activity Theories. The focus mostly will be on the creation and development of the Systemic-Structural Activity Theory (SSAT). We will consider some basic concepts of activity theory and will outline some difficulties that Western scientists experiense in interpretation and application of the theory. General activity theory (AT) was developed in the former Soviet Union. Three prominent scientists Rubinstein, Leont’iev and Vygotsky are responsible for the creation of the theory. For a long period of time attempts were made to use this theory for the study of human work. However, there was not enough significantly developed data for applying it to the study of human work. With the development of mechanization and automation in the industry, in transport, in the military sphere and other modern fields of human activity it became obvious that the direct application of general activity theory was not possible. To the response of technological progress a more advanced theory, an Applied Activity Theory (ATT), was developed by these scientists in 1970s. The most important fields where AAT was applied to were aviation systems, automated control systems for technological processes, remote control systems, software and some others. The further development of ATT led to the creation of the Systemic-Structural Activity theory. Thanks to the research of Gregory Bedny, the SSAT was born. The main postulate of SSAT is that it views activity as a structurally-organized self-regulating system, rather than the set of responses to multiple incentives. This system is considered to be purposeful and self-regulating in changing environmental conditions. Activity theory, and specifically its applied fields, AAT and SSAT, utilize teminology with a totally different meaning of what it is used in the West. It explains why the adaptation of general activity theory to the task analysis in general and to human-computer interaction specifically was ineffective. That was also the reason why analysis of basic concepts of activity theory in the West demonstrates an unfortunate failure of Western scientists in the attempts to capture the original meaning of activity theory terminology. Gregory Bedny illustrates it by the following example. The Russian word deyatel’nost’ loosely translates into English as activity. However, deyatel'nost is a much broader concept than the English word activity. Deyatel'nost is a coherent system of internal mental processes and external behavior actions, and motivation, that are combined and directed to achieve conscious goals. By analyzing, interpreting, explaining, and translating the general activity theory terminology, he provided a great gift to Western scientists, and thus made a significant contribution to understanding of the theory. The SSAT received recognition in the West, and particularly in the USA. In this work we will describe the terminology used in SSAT and the basic concepts of the theory: self-regulation, goal and task. The significance of the AAT and SSAT is that, based on the data developed, it can now be applied for the sudy and practice of human work.

Fred Voskoboynikov, Waldemar Karwowski

Conference Proceedings

Limitations on the use of eye-tracking data to understand operator awareness

In the last 20 years, a number of accidents and incidents in commercial aviation have pointed to poor flightcrew awareness of basic flight path parameters (e.g., airspeed, bank, pitch). As a result, there is a desire to improve pilot situation awareness and how attention is allocated. Eye-tracking has been a commonly used measure of awareness; it can aid in understanding whether specific indications were fixated, and perhaps how a pilot gathered information (that is, which indications in which sequence).In this paper, we discuss limitations on what eye-tracking data can reveal about pilot awareness and understanding. First, previous studies (e.g., Sarter et al., 2007) have shown that fixation on an indication may not ensure awareness or understanding. Further, an operator may have awareness of information not fixated. Additional measures—such as self-report or control inputs—can help to better establish the extent of pilot awareness and understanding.Second, sequences of fixations (scan patterns) have also become a performance measure. While a small number of recognized scan patterns have been validated for a small set of parameters on the Primary Flight Display (PFD), scan patterns have not been identified to support the broader context of flight path management or flight operations. More important is to understand the full set of drivers underlying the larger pattern of eye fixations; this approach moves away from the idea of well-established scan patterns as a marker of skilled performance and gives a larger role to pilot cognition. Pilots have various reasons to direct attention to specific elements on the interface, such as -feedback tied to control inputs-a check on compliance with flight path targets-a reaction to an alert or a call out-attempt to understand an unexpected indication-assess progress toward a flight path targetThe importance of cognition is further implicated in the finding that pilot interviews show that fixating typically is accompanied by expectation; generally, pilots have a strong expectation of what value or indication they will see, which allows more efficient integration of information and an ability to identify indications that suggest an alternative account of the current system state. We will describe a range of eye-tracking measures and how they should and should not be used.

Randall Mumaw, Dorrit Billman

Conference Proceedings

Cognitive Engineering in Training: Monitoring and Pilot-Automation Coordination in Complex Environments

This paper reports on a Cognitive Engineering approach to identify untaught skills and knowledge and to support design of learning tools. We investigate flightpath (FP) monitoring and the interaction of pilot and automation implicated in monitoring. We interviewed experienced pilots to understand the knowledge and skills underlying effective monitoring, and we developed an example learning environment to improve these skills. We explore how design of pilot training and learning, like the design of interfaces and of the underlying automation, benefits from cognitive engineering methods and perspective. In aviation, monitoring and managing FP are critical activities, affected by automation, control actions by the pilot, and by external factors, including weather and Air Traffic Control (ATC). Effective piloting depends on strategies for noticing, understanding, and anticipating these influences to monitor and manage FP. Although flightdeck automation is intended to aid pilot understanding and prediction, the Fight Management Systems (FMS) can also mislead the pilot, particularly when depending on old or incomplete information. Understanding such vulnerabilities is an important part of pilot-automation coordination. We identified skills and knowledge learned from experience but not from training; for less experienced pilots these are likely knowledge gaps and potential targets for learning. Using learner-centered principles, we developed a learning environment designed to help pilots build skills and knowledge for proactive FP monitoring. We consider how a broad cognitive engineering approach might inform both the "what" and "how" of learning in dynamic work domains.

Dorrit Billman, Barth Baron Jr, Randall Mumaw

Conference Proceedings

Multimodal Learnability Assessment of a Touch-based Large Area Display with Eye Tracking and Optical Brain Imaging

Multifunctional Large-Area Displays (LAD) have become an integral part of modern airplane cockpits, offering pilots flexible access to flight controls and mission-critical information. The modern glass cockpit paradigm is expected to reduce the workload of pilots by reducing the complexity and clutter of traditional cockpit layouts and improve their situational awareness by providing flexible access to rich information through the interface. On the other hand, the new paradigm has led to interaction design challenges in utilizing the affordances of the novel interface components, presenting barriers agains effective use of the interface by the pilots. In the present study, we report empirical investigations of such a LAD interface and its learnability via behavioral performance, established upon the neuroergonomics approaches, such as eye tracking and optical brain imaging measurements. Two test pilots, who had prior experience in traditional cockpit layouts were recruited to perform flight tasks in a flight simulator incorporating a LAD touch screen to interact with basic flight instruments of a trainer jet. The pilots were first given standardized training to familiarize with the basic information layout and features of the LAD interface. Following the initial training, both flew a scenario in the simulator that included various tasks (e.g., functional layout assignment, setting communication parameters, monitoring barometric settings and fuel levels, etc.) over the LAD interface for the before take-off, taxi, take off, climb, level flight, approach, and landing phases. The Pupil Labs Invisible mobile eye tracker and the fNIR Imager 1100 system were used to monitor the pilots’ eye movements and prefrontal oxygenation changes during the flight. Two weeks later, the tasks were replicated for the learnability assessment. The results indicated that pilots’ task completion times decreased for the majority of the tasks, accompanied by an increase in eye fixation durations, a decrease in the number of fixations, and a decrease in right dorsolateral prefrontal cortex activation. Overall, these results suggest supporting evidence for the learnability of the new interface paradigm through task measurements in multiple scenarios.

Ezgi Çigdem Şahin, Dilan Diğba Özmenoğlu, Ahmet Paker, Yasin Kaygusuz, Hakan Aydemir, Cengiz Acartürk, Murat Perit Çakır

Conference Proceedings

Guidelines for Artificial Intelligence in Air Traffic Management: a contribution to EASA strategy

Artificial intelligence has the potential to improve air traffic management through the consistent use of machine learning. AI can bring benefits to air traffic controllers in terms of workload, situational awareness, trust, and thus operational efficiency and safety. However, human problem-solving strategies can potentially collide with AI and lead to misunderstandings and a decrease in user acceptance of air traffic control systems. The proposed paper focuses on the design of the ML system, in particular providing insights and guidelines derived from results of recent field studies as they addressed the impacts of conformance and transparency on controller behaviour and survey responses. Several guidelines were distilled based on empirical insights obtained from experiments, feedback from controllers and workshop results. The guidelines are divided into different categories: ML/AI design, Personalization, Transparency, and HCI. The proposed paper also describes a contribution to a different use case to test the generalizability of the guidelines themselves, as well as a recent update in the explainability framework developed by a regulatory authority.

Matteo Cocchioni, Stefano Bonelli, Carl Westin, Ana Ferreira, Nicola Cavagnetto

Conference Proceedings

Multimodal characterization of mental fatigue on professional drivers

A non-adequate psychophysical condition represents a major factor in causing car accidents. In particular, 20% of car crashes are caused by mental fatigue and drowsiness, with dramatic consequences and fatalities. Nowadays strategies to reduce the risks while driving. The on-board systems equipping current vehicles are not able to intervene before the sudden onset of drowsy episodes and are affected by a poor accuracy resulting in several events’ misclassifications (i.e., false positive) causing drivers’ mistrust of technology.Being able to recognise in advance the occurrences of fatigue and drowsiness episodes would dramatically increase road safety and reduce car crashes. This is extremely relevant especially for professional drivers who drive for prolonged periods leading to an increase of risks due to not-proper psychophysical conditions. Even if professional drivers are trained to prevent, recognise, and minimize the effect of fatiguing, it must be considered that often the driver becomes conscious of drowsiness and mental fatigue onset too late, that is when it is already driving in a not-safe condition. The aim of this study was to adopt a multimodal approach to characterize the initial phases of fatigued mental state while driving, to develop an effective and timely detecting methodology. Ten volunteer professional drivers have been recruited to take part in an experimental protocol, performed in a car simulator. The experiment took place in the afternoon to increase the chance of eliciting mental fatigue and it consisted in driving for 45 minutes in a monotonous city-like environment. Before performing the monotonous driving task, participants were asked to drive for 15 minutes in a high-difficulty track race to induce fatigue, increasing the probability of a not-adequate psychophysical condition during the following monotonous driving task. Aiming at developing a neurophysiological model for mental fatigue characterization, a multimodal neurophysiological assessment was performed collecting the Electroencephalographic (EEG) and Electroculographic (EOG) signals. In parallel, behavioural assessment was performed through a secondary reaction task to detect eventual variation of performance from individual normal levels because of an altered psychophysical condition. Subjective measures were collected as well for the self-assessment of both fatigue and drowsiness state and task perception (high vs low demand). Behavioural and subjective measures have been so employed to (i) validate the experimental design; and to (ii) support and validate the employment of neurophysiological measures for characterizing the mental fatigue.

Andrea Giorgi, Vincenzo Ronca, Alessia Vozzi, Pietro Aricò, Gianluca Borghini, Luca Tamborra, Ilaria Simonetti, Simone Sportiello, Marco Petrelli, Rodrigo Varga, Marteyn Van Gasteren, Fabio Babiloni, Gianluca Di Flumeri

Conference Proceedings

Teamwork objective assessment through neurophysiological data analysis: a preliminary multimodal data validation

Teamwork efficiency and safety are inextricably linked. The capability of having online insights and access to objective information regarding cognitive and emotional aspects of the team members using neurophysiological measures (brain activity, skin conductance, heart rate) will endow a tool which can support Instructors during the assessment and management of teams. Such neurophysiological measures can be seen as the physical interface that will enable for gathering insights about all the aspects relating to Human Factors (HFs) of the operators. The study aimed at developing and validating a methodology able to objectively measure the teamwork dynamics and efficiency. This objective has been performed in a real surgery-related context. A data-driven approach based on machine - learning (ML) and multivariate autoregressive (MVAR) models has been employed to develop the Neurometrics - based teamwork model. Such a model considered the co-variations both within each HF (e.g., Low vs High Stress) and between different HFs (e.g., Attention vs Workload) to consider their simultaneous coexistence. The results of this preliminary study demonstrated that it is possible to quantify the teamwork of operators while dealing with real tasks and endow additional information for a more accurate teams assessment and management.

Gianluca Borghini, Vincenzo Ronca, Pietro Aricò, Gianluca Di Flumeri, Andrea Giorgi, Stefano Bonelli, Laura Moens, Lidia Castagneto Gissey, Maria Irene Bellini, Giovanni Casella, Fabio Babiloni

Conference Proceedings

EEG assessment of driving cognitive distraction caused by central control information

This study collected EEG data using a driving simulator and analyzed it using the average spectral power density of EEG features to study the assessment method of cognitive distraction in driving caused by central control information. The results showed that Theta, Beta1 and Beta2 brain waves in the frontal lobe and central region could reflect the driver's cognitive load and cognitive processes. As cognitive difficulty increases, Theta and Beta2 brain waves in the frontal lobe and central region gradually calm down, and Beta1 becomes more active. By recording the driver's EEG signals and analyzing changes in brain waves, the impact of in-vehicle central control system de-sign on driver cognitive distraction can be evaluated. This EEG-based evaluation method can provide a more objective and accurate assessment, providing a scientific basis for optimizing and improving the design.

Yan Zhao, Xiaonan Yang, Yahui Wang, Wanni Wei, Heng Zuo, Yuan Yu, Yan Zhao

Conference Proceedings

Validation of affective images of the IAPS set in children

Emotions, in the field of Neuropsychology, are a topic of contemporary relevance due to the impact they have on a person's development. At the brain level, emotions arise from the activation of brain circuits in response to stimuli. Treatments have focused on traditional psychological intervention, which today, thanks to technological advances, includes the use of technology, so it is necessary to have available material that has been validated and standardized in the area where it is to be used, which is the subject of this study. The validation of 16 affective images from the International Affective Picture System (IAPS) set was carried out with the participation of 223 children between 6 and 8 years of age from a private, co-educational educational institution in Cuenca, Ecuador applying validation processes followed in other countries, adjusted to our context. The children's emotional responses were similar to those of the original test according to the statistical tests applied.

Gabriela Gómez-Ochoa, Patricia Ortega-Chasi, Martha Cobos-Cali, Omar Alvarado-Cando, Karla Andrade-Castro

Conference Proceedings

Towards continuous mental state detection in everyday settings: investigating between-subjects variations in a longitudinal study

Maintaining mental health can be quite challenging, especially when exposed to stressful situations. In many cases, mental health problems are recognized too late to effectively intervene and prevent adverse outcomes. Recent advances in the availability and reliability of wearable technologies offer opportunities for continuously monitoring mental states, which may be used to improve a person’s mental health. Previous studies attempting to detect and predict mental states with different modalities have shown only small to moderate effect sizes. This limited success may be due to the large variability between individuals regarding e.g., ways of coping with stress or behavioral patterns associated with positive or negative feelings. A study was set up for the detection of mental states based on longitudinal wearable and contextual sensing, targeted at investigating between-subjects variations in terms of predictors of mental states and variations in how predictors relate to mental states. At the end of March 2022, 16 PhD candidates from the Netherlands started to participate in the study. Over nine months, we collected data in terms of their daily mental states (valence and arousal), continuous physiological data (Oura ring) and smartphone data (AWARE framework including GPS and smartphone usage). From the raw data, we aggregated daily values for each participant in terms of sleep, physical activity, mental states, phone usage and GPS movement. First results (six months into the study at the time of writing) indicate that almost all participants show a large variability in ratings of daily mental states, which is a prerequisite for predictive modeling. Direction, strength and standard deviations of Spearman correlations between valence, arousal and the different variables suggest that several predictors of valence and arousal are more subject dependent than others. In future analyses, we will test and compare different versions of predictive modeling to highlight the potential of wearable technologies for mental state monitoring and the personalized prediction of the development of mental problems.

Lea Berkemeier, Wim Kamphuis, Herman De Vries, Anne-Marie Brouwer, Jan Ubbo Van Baardewijk, Maarten Schadd, Hilbrand Oldenhuis, Ruud Verdaasdonk, Lisette Van Gemert-Pijnen

Conference Proceedings

Investigating Feature Set Decisions for Mental State Decoding in Virtual Reality based Learning Environments

In modern workplaces with rapidly changing skill requirements, suitable training and learning environments play a key role for companies to remain competitive, effective and ensure job satisfaction. To provide an immersive, interactive, and engaging learning experience, Virtual Reality (VR) has emerged as a revolutionary technology. Especially when erroneous behaviour is associated with severe consequences or great resources, VR offers the opportunity to explore actions and visualize consequences in safely and at affordable costs. In addition, it provides an easy way to personalize educational content, learning speed, and/or format to the individual to guarantee a good fit with skills and needs. This is decisive, since insufficient or excessive workload during training sessions results in demotivation and reduced performance. In the latter case, persistent professional exhaustion, pressure to succeed and stress can lead to long-term psychological consequences for employees. Besides skill and ability, current physical conditions (e.g., illness or fatigue) and psychological states (e.g., motivation) also affect the learning performance. To identify and monitor individual mental states, Brain-Computer Interfaces (BCI) measuring neurophysiological activation patterns, e.g., with an electroencephalography (EEG), or functional near-infrared spectroscopy (fNIRS) can be integrated in a VR-learning environment. Recently, fNIRS, a mobile optical brain imaging technique, has become popular for real-world applications due to its good usability, portability, and ease of use. For the reliable online decoding of mental states, informative neuronal patterns, suitable methods for pre-processing and artefact removal, as well as efficient machine learning algorithms for the classification need to be explored. We, therefore, investigated and decoded different working memory states in a free moving fNIRS experiment presented in VR. different working memory states in a free moving fNIRS VR experiment and the possibility of decoding these states properly. 11 volunteers (four female, right-handed, mean age of 23.73, SD = 1.42, range = 21−26 years) participated in the study. The experimental task was a colour-based visuo-spatial n-back paradigm adapted from Lühmann and colleagues (2019) with a low (1-back) and high working memory load condition (3-back) and a 0-back condition as active baseline. Brain activity was recorded using the mobile NIRx NIRSport2. To capture brain activation patterns associated with working memory load, optode montage was designed to optimally cover the prefrontal cortex (PFC; in particular, dorso- and ventrolateral parts of the PFC) with some lateral restriction by the VR head-mounted display (HMD). fNIRS signals were processed using the python-toolbox mne and mne-nirs. For the decoding of working memory load, we extracted statistical features, that are peak, minimum, average, slope, peak-to-peak, and time-to-peak, from epochs of oxygenated (HbO) and deoxygenated (HbR) hemoglobin concentration per channel. A Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) and Gradient Boosting classifier (XGBoost) were explored and compared to a Dummy classifier (empirical chance level). We also investigated which cortical regions contributed to the decoding when choosing single features and which feature combination was suggested to optimize performance. With this study, we aim to provide empirically supported decision recommendations to reach the next step towards future online decoding pipelines in real-world VR-based learning applications.

Katharina Lingelbach, Daniel Diers, Michael Bui, Mathias Vukelić

Conference Proceedings

Evaluating the restorative impact of nature through multimodal mobile sensing of neural, physiological, and behavioral activity in ambulatory settings

One of the fundamental principles of neuroergonomics is that human cognition is profoundly shaped by the environment in which it operates. In the modern world, this environment can often be highly artificial, noisy, barren, and intentionally distracting. On the other hand, natural environments compare favorably as they may offer not only an appreciation of beauty but a rich array of sensory and contextual information which can be undemanding to the observer. Attention Restoration Theory (ART) proposes that exposure to natural environments can provide various benefits to stress, health, and cognition. Understanding how the brain responds to natural environment presentation poses a crucial hurdle to using traditional neuroimaging techniques as many approaches necessitate highly controlled and resultingly, low-fidelity stimuli presentation to mimic the environmental effects of nature. Functional near-infrared spectroscopy (fNIRS), a non-invasive brain monitoring technology that relies on optical techniques to detect changes in cortical hemodynamic responses to human perceptual, cognitive, and motor functioning, is an ideal candidate tool for understanding the brain in natural environments. In this paper, we will describe an experimental setup that involves the integration of mobile fNIRS systems with simultaneous wrist-based optical heart rate monitoring (OHRM) and electrodermal activity (EDA) recordings that can record the cognitive and physiological responses of individuals to natural settings.

Adrian Curtin, Yigit Topoglu, Saqer Alshehri, Michael Woodburn, Lynelle Martin, Rajneesh Suri, Hasan Ayaz

Conference Proceedings

A Data-Driven Framework to Model Physical Fatigue in Industrial Environments Using Wearable Technologies

Industry 4.0 is the tendency towards automation and data exchange in manufacturing and the process sector. However, many manual material handling and repetitive operations can still cause the operators fatigue or exhaustion. Once the operator experiences physical fatigue, their performance decreases. The consequences may result in reduced production quality and efficiency and increased operational human errors that could give rise to incidents and accidents. Over time, physical fatigue can result in more adverse effects for the operators, such as Chronic Fatigue Syndrome (CFS) and Work-related Musculoskeletal Disorders (WSMD). For this reason, from an occupational health and safety point of view, the operator's physical fatigue must be managed. The increasing availability of wearable devices combined with health information can provide real-time measuring and monitoring of physical fatigue in the operational environment while minimally influencing the primary job. This paper presents a physiological signal-based approach using a non-intrusive wristband for continuous health monitoring to predict physical fatigue in industrial-related tasks. These data are used as input to classification algorithms to detect physical fatigue. Accurate and real-time physical fatigue detection helps to improve operator safety and prevent work accidents. Future work will deploy the model in a real-world environment in the industry.

Carlos Albarrán Morillo, Micaela Demichela

Conference Proceedings

Mental Workload Classification during simulated flight operations based on cardiac and neural dynamics recorded using the MUSE 2 low-cost system

The advancement of low-cost and highly portable physiological systems presents promising opportunities for monitoring human cognitive processes during daily-life activities and more complex tasks such as operating an aircraft. The Muse 2 system combines electroencephalography (EEG) and photoplethysmography (PPG) sensors allowing the extraction of neural dynamics features in the time and frequency domains and heart rate. In a study, we equipped five pilots with the Muse 2 system while they performed a low-load and high-load traffic pattern task along with a passive auditory oddball task. The group-level analyses revealed that participants exhibited higher average heart rate, lower power spectrum density in the alpha band, decreased P300 amplitude in the high-load compared to the low-load condition. These results are in line with previous laboratory research conducted in highly controlled settings and research-grade instrumentations. The classification of the two levels of mental workload reached 93.2% accuracy on a single-trial basis based on EEG frequency features. Post-hoc analysis revealed that the classifier mainly relied on motion artefact features in the beta and gamma bands. The classifiers using heart rate and ERPs features reached 76% and 77.8% classification accuracy, respectively. Despite its interest, this system presents some limitations for mobile and neuroergonomics applications notably with regards to the limited number of electrodes preventing the use of advanced signal processing techniques to address noise and artifacts in the signals.

Frederic Dehais, Simon Ladouce, Juan Torre Tresols, Ludovic Darmet, Daniel Callan

Conference Proceedings

Neuroergonomics of Cursor Control Devices in Spacecraft Cockpits for Spaceflight Participants

The commercial space transportation industry is rapidly growing with increasing numbers of spaceflight participants (SFPs). These private individuals receive considerably less training than astronauts before embarking on space missions, which presents an urgent need to develop the cognitive ergonomics that simplify spacecraft cockpit design. Neuroergonomics is an emerging area within cognitive ergonomics, which Parasuraman described as “the study of brain and behavior at work”. This experimental study investigated the neuroergonomics of cursor control devices (CCDs) for spacecraft cockpits by applying electroencephalography (EEG) power indices as objective measures of concentration, relaxation, effort, fatigue, arousal, valence, and absorption during task performance. Data for this study were collected from a sample of twenty-seven participants who performed a Fitt’s cursor control task in PsyToolkit with a counterbalanced device sequence of four different CCDs, i.e., touchpad, touchscreen, joystick, and numpad. The devices were affixed to, and configured in the variable positioning Adaptive Spaceship Cockpit Simulator, which was used to simulate the microgravity environment of space using head-down tilt (HDT). The index of difficulty of the cursor control task trials was varied according to Fitt's law across easy, medium, and difficult levels. The orientation of the simulator varied between upright and HDT orientations. We administered a HDT treatment before the experimental trials in the HDT orientation to induce the physiological effects associated with increased intraocular pressure, which results from microgravity. A HDT recovery period was administered after the experimental trials in the HDT orientation. Participants completed a subjective questionnaire to capture perceived effort at the end of each experimental track. Using the Flow Choice Architecture, we processed EEG signals to compute EEG power indices for a Multivariate Analysis of Variance. There were significant findings in concentration across CCDs during the two orientations. The HDT orientation demanded more concentration than the upright orientation across the devices. This result indicated that there was additional cognitive workload induced by manipulating the CCDs in the HDT orientation. There were significant differences in fatigue across the two orientations. The HDT orientation was associated with greater fatigue levels. An important finding in the subjective questionnaire was the perceived effect of the HDT orientation on cognition. The touchpad consistently demonstrated differences relative to the other CCDs. Task difficulty did not significantly impact any of the EEG indices. No significant interactions were observed in the EEG indices across the orientations, devices, and task difficulty levels. A striking result emerged during the HDT recovery period where most participants exhibited a sleepy-like EEG signature characterized by a consistently high relaxation index. Overall, these results indicated that computational neuroergonomics may produce objective insights about the human spaceflight experience related to orientation and cursor control devices. We recommend that strategies to enhance spacecraft cockpit design include neuroergonomics of CCDs, control devices, and user interfaces, in general.

Troy Weekes, Kazuhiko Momose, Thomas Eskridge

Conference Proceedings

Do increased engagement effects in lecture videos improve comprehension?

Students are choosing more and more to enroll in online courses due to convenience or acclimation from distance learning during the COVID-19 pandemic. However, instructors must learn to utilize principles of cognitive load and student engagement when designing online courses, especially when creating asynchronous lecture videos. This study examined the effects of content difficulty (Easy vs. Hard) and percentage level of engagement effects in videos (10%, 25%, 50%, 75%) on comprehension of course material. Participants were asked to watch one easy content and one hard content video and answer questions on the video topic after each video. Perceived usability, mental effort, and engagement behavior tendencies while watching instructional videos were also measured. Results showed a significant interaction between content difficulty and subject pool, with student participants performing better than Amazon participants, specifically on hard content. Participants rated lower levels of engagement effects as more usable, and participants overall rated easy content requiring less mental effort to understand than hard content. Implications and further research topics based on these findings are discussed.

Jessica Mar, Kim-Phuong L. Vu

Conference Proceedings

Remembering Passwords: The Role of Instructions

Most users follow predictable patterns and create weak passwords because they are unaware how to generate strong, secure passwords (Ur et al., 2015). Yet, more secure, system-generated passwords tend to be more difficult to remember (Vu et al., 2003). The current study examined whether system-generated passwords could be made more memorable through use of different types of instructions that help the users associate text and/or images to the passwords or password components. Over 100 participants were asked to memorize three system-generated passwords for three fake accounts: bank, email and social media, either in a lab-based setting with a moderator or completely online. Participants were given no instructions, text-based instructions, image-based instructions, or a combination of both text and image-based instructions to help them understand and memorize each password. Participants were then asked to recall their password after no delay or a short delay. We found that users were able to remember complex system-generated passwords when provided with detailed text-based or image-based instructions to help the users map the password components to a structure. Our findings did not clearly show which instructional technique was better. Future studies should explore additional instructional techniques for password generation and memorization.

Kim-Phuong L. Vu, Ha Nguyen, Uyen Bui

Conference Proceedings

Evaluation of Pedometer Interfaces for Mobile Apps

The use of mobile health apps has been on the rise, as they allow people to get their health information more conveniently. Many people are using their mobile health apps to track their health status (KC et al., 2021), but there are known issues with people being unable to use their health apps effectively due to poor design. According to Wildebos et al. (2019), if users are continuously failing to get the information they need, they could develop feelings of insecurity and stop using the app. To mitigate these negative interface design impacts, Universal Design Principles (Story, 1998) and Gestalt’s Principle of Perceptual Grouping (Smith-Gratto & Fisher, 1999) could be used to improve the interfaces. In the present study, we evaluated several interfaces of pedometer apps that varied in terms of flexibility (low and high) and three levels of simplicity (simple, intermediate, complex). Ninety six participants were recruited from MTurk. The participants responded to questions on a survey that require them to extract information from a pedometer interface. After answering the comprehension questions for the specific interface, participants were asked to indicate their perceived ease of use (Brooke, 1996) and the likelihood of utilizing the pedometer app (Pasha & Indrawati, 2020). We found that participants had higher accuracy scores with the interface that was intermediate in terms of simplicity, but they preferred the simple or complex interface design. Results of this study suggest that users may not prefer designs that lead to better task performance and designers will need to balance features that enhance performance versus those that users find to be more attractive or desirable for continued use.

Alina Tran, Kim-Phuong L. Vu

Conference Proceedings

Right visual field is advantageous in detecting different color: an implication for appropriate arrangement of digital graphics on a display

Many studies have so far revealed that the visual field biases influence visual tasks. These biases should be taken into consideration to arrange an Excel worksheet, draw a CAD (computer-aided design) blueprint and other digital graphics effectively. In this study, we investigated visual field biases in identifying different and same color lines shown on a computer display. Twelve male and two female college students (21-22yrs.; mean age 21.2yrs.) were recruited to participate in the experiment. All were physically and mentally healthy and had visual acuity (including corrected visual acuity) that does not interfere with the task performance and vision that allows color perception. All participants were right-handed. Visual stimuli were displayed on a 17-inch CRT monitor. White circle on a quadrant (1000 ms) was followed by Test1 image (1500 ms), checker pattern image as a distractor (4100 ms) and Test 2 image (4000 ms). White circle showed the quadrant to which a participant must pay attention in order to discriminate different and same colored lines shown in Test 1 image and Test 2 image. Immediately after a participant press a correct response key, a correct answer was fed-back to a participant. The participants were instructed to detect different and same colored lines in the visual field and press a correct response key as quickly as possible. The results demonstrated that the participants could detect different colored lines faster than match of color in both upper- and lower-right visual fields have advantages (F (1,13) =28.814, p<0.000; F (1,13) =8.120, p<0.014, respectively). However, there was no change in identifying different and same colored line in both upper- and lower-left visual fields (F (1,13) =13.000, ns.; F (1,13) =13.000, ns., respectively). Rates of correct responses to different and same colored lines were almost same in these visual fields (upper-right, lower-right, upper- left, lower-left visual fields) (F (1,13) = 0.027, ns.; F (3,39) =0.073, ns, F (1,13) = 0.071, ns; F (1,13) =1.779, ns, respectively). These results showed the advantage in identifying difference in colored lines in right visual field compared with left. This finding implies that right side is appropriate to arrange a manuscript for proofreading, a calculating worksheet, a CAD blueprint and other digital graphics for modification.

Seiko Kawashima, Motoharu Takao, Senri Komiya, Mana Hattori

Conference Proceedings

Use of eye-tracking system to evaluate selective attention in Children with motor difficulties.

This study aimed to use the Eye Tracking System to assess selective attention. Researchers applied the Margarita's Test in a pilot study with a non-probabilistic convenience sample (N=30). Participants were schoolchildren aged 8 (15 males and 15 females). The measurement instruments used were the Margarita's and the Tobii Glasses 2 eye-tracking system. The results showed that 37.11% of correct responses were obtained if the first fixation coincided with the motor response. When analyzing the last fixation, the coincidence of fixations and motor response reached 81.3%, of which 77.2 % were correct. These results suggest that the use of the eye-tracking technology, jointly with the Margarita's test, has the potential to evaluate selective attention in children with motor difficulties.

Daniela Idrovo, Martha Cobos, Nataly Alvarado, Patricia Margarita Ortega Chasi, Esteban Mora, Jaime Burbano, Francisco D Salgado

Conference Proceedings

Exploring the Role of Visual Attention in Aggressive Behavior: Evidence from Eye-Tracking Measurements

This study explores the relationship between eye-tracking measurements (fixation count and duration) and aggressive traits. The research involved 60 female and male participants between 12 and 17 years. The standardized questionnaire used to measure aggressive traits was CAPI-A to assess premeditated and impulsive aggression. The sample was divided into two groups based on aggressive traits' presence (n=30) or absence (n=30). The participants were exposed to a validated subset of the OASIS affective images as visual stimuli, using the Gazepoint GP3 device to capture eye-tracking information. The study found that participants with aggressive traits had higher fixation durations and fixation counts on negative stimuli than non-aggressive participants. These findings suggest that aggressiveness may be related to selective attention towards negative stimuli, which may impede a person's ability to perceive positive stimuli in their environment. This study provides insight into potential underlying mechanisms contributing to aggressive behavior in adolescents.

Alexandra Calle, Patricia Margarita Ortega Chasi, Andrea Argudo-Vásconez, Martha Cobos, Omar Alvarado

Conference Proceedings

Unlocking Human Potential: The Power of Neural-Interface Technology measuring Cognitive Ability and Traits

Technoking has recently launched cutting-edge neural-interface technology (Hamilton 2022). This kind of technology has the potential to enhance human performance and safety across a variety of fields. To explore its possibilities, this study has set up measurement situations by pre-registering EEG measurements and established future applications based on neuroergonomic laboratory-based training. Positive system intelligence has been found to enhance performance and reduce stress, and neurofeedback-based adjustment has been shown to increase performance for both athletes and manpower training. To test the effectiveness of this technology, the researchers employed a system integrator's approach from collecting real-time streaming data based on vignettes to measuring cognitive load alleviation via dockers, testing package-based cloud computing interfaces through web technologies. The results of this study demonstrate that system intelligence measures can be used to ignite practical innovation processes and select high-performing manpower, ultimately leading to gains in innovativeness. The study provides a pre-registration analyzing example adaptable case model that can be applied to diverse host platforms and databases in Unix machines. Specifically, it explores the use of system intelligence measures in the context of human resources employment criteria using post-modernity competency assessment for innovative recruitment practices. The study also investigates the main cost of the study and the safety of the equivalence based on the requirements of the environment.

Janne Heilala, Waldemar Karwowski

Conference Proceedings

MRI Image Segmentation using Fuzzy C-Means in MATLAB

Image segmentation is challenging because an image has to be partitioned into regions with pixels that contain similar attributes. To accomplish this task, many use MATLAB, Python, or other environments that allow them to set thresholds for the image, to group similar pixels together, or remove undesirable pixels from the image. When approaching the image segmentation problem, many use the fuzzy c-means algorithm to group similar pixels with the same features in an image. Hence, it makes it easier to analyze specific areas in the image, such as a tumor that may be present. This paper presents a tutorial on performing image segmentation using MATLAB and Fuzzy C-Means. Through this tutorial, we examine how fuzzy c-mean is used to solve the image segmentation problem and the methodologies used to properly cluster data.

Brisaac Johnson, Chris Crawford

Conference Proceedings

Improvement of the Accuracy of SSVEP-BCI with In-Ear EEG Using Multiple Regression Analysis

A brain-computer interface (BCI) is an interface that reads brain activity and operates a computer. Typically, BCI involves wearing a device on the head to measure brain waves. EEG meters are cumbersome for the user when worn and require individuality techniques to reduce impedance. In-Ear EEG, which is obtained from electrodes placed near the inner and outer ear, is expected to solve these problems. Users can use it to have a comfortable BCI experience. However, in-Ear EEG has the problem that the EEG used for BCI uses EEG located far from the actively observable areas of the brain, which results in poor signal quality. Steady state visual evoked potential (SSVEP) is an EEG that is predominantly observed in the primary visual cortex in the occipital region during gazing at a flashing stimulus, and SSVEP-BCI is expected to be used as a new communication tool. Ear EEG will greatly advance the social implementation of SSVEP-BCI. In this study, electrodes were placed at 10 locations near the ear and 4 locations on the back of the head. SSVEP-BCI with 28 different inputs including alphabets was designed and evaluated. The accuracy of the SSVEP-BCI with signals obtained from the occipital region was 84.92%, and that of the BCI with signals obtained from the ear was 28.17%. Using the occipital and intra-ear EEG data sets as the teacher data, multiple regression analysis was performed to improve the accuracy, and an accuracy of 55.83% was achieved. These results indicate that improvements are needed to make in-Ear EEG based SSVEP-BCI practical. It also suggests that a communication tool using in-Ear EEG based SSVEP-BCI may be feasible in the future.

Sodai Kondo, Hisaya Tanaka

Conference Proceedings