Video-Based Contactless Mood Detection Combining Heart Rate and Facial Expressions

Mood detection and vital signs monitoring based on commonly used contact or contactless devices can play an important role in psychological health monitoring and human-computer interaction. Existing methods cannot rely on the common smart bracelets or watches to monitor emotions in daily life. Thus, in this paper a novel framework for mood detection that combines vital signs and facial expression recognition using contactless sensors is proposed. Specifically, a heart rate (HR) estimation algorithm and a facial expression recognition (FER) module using a low cost and commercial camera are combined to evaluate the end user's mood. The proposed system can detect faces "in the wild" independently from the selected vision sensor and from face orientation, consequently increasing its usability. We are currently testing the overall system first in a controlled environment and then in a real environment to achieve our final goal. The findings of the preliminary experiments show promising results for HR and facial expression monitoring with a low average error expressed in terms of Root Mean Square Error for HR estimation and high accuracy regarding facial expression recognition.