Phonetic syllables applied to continuous emotion tracking

In the present work we take into account the needs of real-time systems to give an estimate of the emotional content of an utterance during its production rather than waiting for it to be completed. The potential impact of this approach on the design of affective computing systems is also analysed. Past works have shown the importance of syllables for the transmission of emotions, while classical research methods adopted in prosody show that it is important to concentrate on specific areas of the speech signal to study intonation phenomena. In this work, we concentrate on continuous emotion tracking on syllable-like segments as our smallest unit for feature extraction considering it as an independent unit, regardless of its overlapping with phonologically expected syllables. The method we propose is evaluated on a continuous, three-dimensional model of emotions built on the classical axes of Valence, Activation and Dominance relying on the annotation available in the SEMAINE corpus. In particular, we concentrate on the spectral content of syllabic nuclei by examining the correlation of a single syllable's feature vector with the instantaneously perceived emotional level. The proposed method is shown to be competitive with state-of-the-art performance showing significant results in the reduction of the amount of information to be processed, and introducing features weighting based on syllabic prominence, thus not considering all the units of analysis as being equally important.