Deep Fusion of Neurophysiological and Facial Features for Enhanced Emotion Detection

Abstract

The fusion of facial and neurophysiological features for multimodal emotion detection is vital for applications in healthcare, wearable devices, and human-computer interaction, as it enables a more comprehensive understanding of human emotions. Traditionally, the integration of facial expressions and neurophysiological signals has required specialized knowledge and complex preprocessing. With the rise of deep learning and artificial intelligence (AI), new methodologies in affective computing allow for the seamless fusion of multimodal signals, advancing emotion recognition systems. In this paper, we present a novel multimodal deep network that leverages transformers to extract comprehensive features from neurophysiological data, which are then fused with facial expression features for emotion classification. Our transformer-based model analyzes neurophysiological time-series data, while transformer-inspired methods extract facial expression features, enabling the classification of complex emotional states. We compare single modality with multimodal systems, testing our model on Electroencephalography (EEG) signals using the DEAP and Lie Detection datasets. Our hybrid approach effectively captures intricate temporal and spatial patterns in the data, significantly enhancing the system's emotion recognition accuracy. Validated on the DEAP dataset, our method achieves near state-of-the-art performance, with accuracy rates of 97.78%, 97.64%, 97.91%, and 97.62% for arousal, valence, liking, and dominance, respectively. Furthermore, we achieved a precision of 97.9%, a ROC AUC score of 97.6%, an F1-score of 98.1%, and a recall of 98.2%, demonstrating the model's robust performance. We demonstrated the effectiveness of this method, specifically for EEG caps with a limited number of electrodes, in emotion detection for wearable devices.