A Model-based Environment for building and running Situation-aware Interactive Applications

Abstract: Situation-aware software systems open unique perspectives for improved goal achievement characteristics for both users and software suppliers. Such interactive systems give users individualized user experience, provide runtime-adaptive user in-terfaces, allow for situation-dependent task planning capabilities and can trigger ser-vice activations when the situational context changes.Situation-awareness is an evolution of the concept of context-aware computing that was first proposed for distributed mobile computing in [7]. In addition to software responsiveness when dynamically migrating applications to other devices and loca-tions, the notion of context then also included aspects related to the interacting people and the environment.The term situation-awareness was first established in psychology and the cognitive sciences. It was used for describing support systems for human operators in complex situations. By defining situation-dependent requirements correct and smooth interac-tive task accomplishment could be facilitated [3], [4]. For such systems, considering the changes of the interactive context over time plays an important role. The complete cognitive process of perception of the users is dependent on contextual information encountered during the interactive processing of tasks. To capture the individual requirements of a situation, Chang in [1] has suggested that a situation specification must cover the user’s operational environment E, the user’s social behavior B, by interpreting his or her actions, and a hidden context M that includes the user’s mental states and emotions. A situation Sit at a given time t can thus be defined as Sit = <M, B, E>t. A user’s intention for using a specific soft-ware service for reaching a goal can then be formulated as a temporal sequence <Sit1, Sit2, …, Sitn>, where Sit1 is the situation that triggers the usage of a service or a task execution and Sitn is the goal-satisfying situation.This model provides a good starting point for the construction of situation-aware applications. However, for recognizing and evaluating situations, the hidden context M, i.e., the user’s emotions, intentions and cognitive load must be made visible and exploitable. Adaptive situation-aware software systems need cognitive and analytic capabilities to interpret the multitude of available emotional and bio-physical signals as well as application and contextual meta-data to being able to infer a goal-reaching set of adaptations.Rather than providing situation-awareness separately for each developed applica-tion, we present an integrated model-based environment for building and running dynamically adaptive situation-aware applications. Therefore, we follow a combined approach by linking the PaMGIS (Pattern-Based Modeling and Generation of Inter-active Systems) MBUID-framework [2] and its domain and user interface models with SitAdapt, a high-resolution observation platform with an integrated user-centric adap-tation component [5]. SitAdapt offers functionality for visual emotion recognition, eye- and gaze-tracking, pulse-rate-tracking, EEG-signal recording, as well as various other bio-physical signal monitoring capabilities. For integrating the necessary rea-soning capabilities, we have introduced situation rules that have access to both, the observed raw-situations, and the modeling resources of the MBUID.We have tested the SitAdapt system with different e-commerce web-applications, e.g., an existing beauty products shopping portal that was enhanced with situation-awareness capabilities and runtime adaptivity [6]. In this environment we also con-ducted two user studies for evaluating the monitoring features and adaptive power of the system with respect to functionality and user experience.In this paper we will discuss the exemplary construction process and runtime-operation of an adaptive situation-aware web-application for travel-booking. We will demonstrate the different steps of the building process that derive application structure and functionality from the task and domain models of the application and apply vari-ous HCI-patterns as well as design and layout models for generating the interactive parts of the source code. We then will describe the observation process that records individual situation profiles during interactive sessions. We will finally discuss the adaptation process that creates a priori adaptations based on the parsing of user and environment models and dynamic runtime-adaptations that are triggered by situation rules fired when a user-specific situation profile encounters conditions that were speci-fied in the situation-rule model of the application. We will conclude the paper with a discussion of ethical and privacy issues arising when confronting users with such powerful monitoring systems. We will also give a short outlook on assessing situation-rule performance and optimizing the rules by implementing a reinforcement learning-component in the future.

Keywords: Situation-Awareness, Adaptivity, Emotion Recognition, Model-based User Interface Development Environments

DOI: 10.54941/ahfe1002754

Cite this paper:

​