Thesis Defense of Koussaila Moulouel
Koussaila Moulouel, a PhD candidate from the SIRIUS team, defended his thesis on May 19, 2023, in the RT Amphitheater at the Vitry-sur-Seine campus of UPEC—120 rue Paul Armangot, 94400 Vitry-sur-Seine.
Title: Hybrid AI Approaches for Context Recognition: Application to Activity Recognition and Anticipation, and Management of Context Anomalies in Ambient Intelligence Environments
Thesis Supervisor(s): Yacine Amirat
Abstract:
Ambient Intelligence (AmI) systems aim to provide users with assistive services that improve their quality of life in terms of autonomy, safety, and well-being. Designing AmI systems capable of accurate, fine, and consistent recognition of the spatial and/or temporal context of users—considering the uncertainty and partial observability of AmI environments—presents several challenges for better adapting assistance services to the user’s context. This thesis proposes a set of contributions addressing these challenges.
First, a descriptive and narrative ontology of context is proposed to model contextual knowledge in AmI environments. The purpose of this ontology is to model the user's context, considering various context attributes, and to define the commonsense reasoning axioms necessary to deduce and update the user's context. Unlike state-of-the-art ontologies, the proposed context ontology includes (i) a TBox representing the core domain ontology defined by concepts and relations, (ii) an ABox of propositional formulas corresponding to instances of context attributes, and (iii) an RBox, represented by an ASP logic program, consisting of rule models such as event effect specification, triggered event specification, context component aggregation, and detection and assistance action planning. The TBox, ABox, and RBox form the foundation of the frameworks developed in this thesis, playing a crucial role in enhancing user context recognition.
The second contribution is a hybrid ontology-based framework that combines commonsense probabilistic reasoning and probabilistic planning to recognize user context, particularly context anomalies, and provide context-aware assistive services in the presence of uncertainty and partial observability in environments. This framework leverages predictions of context attributes, such as user activity and location, provided by deep learning models. In this framework, the commonsense probabilistic reasoning is based on the proposed context ontology to define the axiomatization of context inference and planning under uncertainty. Probabilistic planning is used to characterize abnormal context by addressing the incompleteness of contextual knowledge due to the partial observability of AmI environments. Moreover, probabilistic planning allows for adapting the assistive services provided to the user based on their context. The proposed framework was evaluated using transformers and CNN-LSTM models on the Orange4Home and SIMADL datasets. The results demonstrate the framework’s effectiveness in recognizing user contexts, such as user activity and location, as well as context anomalies in uncertain and partially observable environments.
Third, a hybrid framework combining deep learning and probabilistic reasoning for anticipating human activities based on egocentric videos is proposed. The probabilistic commonsense reasoning used in this framework is based on abductive reasoning to anticipate atomic and composite human activities, and on temporal reasoning to capture changes in context attributes. Deep learning models, namely YOLOv5 and ResNet, were used to recognize context attributes such as objects, human hands, and people’s locations. The context ontology is used to model relationships between atomic and composite activities. The evaluation of the framework shows its ability to anticipate composite activities over a time horizon of a few minutes, unlike state-of-the-art approaches that can only anticipate atomic activities over a time horizon of a few seconds. It also demonstrated strong performance in terms of accuracy in classifying anticipated activities and computational time.
Finally, a stream-based reasoning framework is proposed for anticipating atomic and composite human activities based on streams of contextual attribute data collected on-the-fly. Deep learning models YOLOv7 and ResNet were employed to recognize contextual attributes such as objects used in activities, hands, and user locations. The stream-based reasoning system performs causal, abductive, and temporal reasoning using contextual knowledge obtained in real time. Dynamic effect axioms were introduced to anticipate composite activities that may be subject to unforeseen events, such as the skipping or delaying of an atomic activity. The proposed framework was validated through experiments conducted in a kitchen environment. The high performance in terms of the number of activity anticipations demonstrates the framework’s ability to leverage past contextual knowledge needed to anticipate composite activities. Its performance in terms of contextual knowledge inference time indicates that the framework is well-suited for real-world applications.