A hybrid grammar-based approach to multimodal languages specification

Multimodal interaction represents the future of human-computer interaction. It aims at enabling users to communicate more freely and naturally with automated systems. Current approaches to multimodal dialogue systems are based on two different strategies to perform the modality fusion. The former implements the multimodal fusion at dialogue management level, whereas the latter at grammar level. In this paper we propose a new strategy based on a hybrid multimodal grammar that puts formal grammars and logical calculus together in order to overcome the drawbacks of grammar-based and dialogue-based methods. Moreover, an application of the proposed approach in the context of a multimodal phone book is given and the experiments that we performed to evaluate the approach are described.