A Survey on Multimodal Disinformation Detection

Recent years have witnessed the proliferation of offensive content online such as fake news, propaganda, misinformation, and disinforma- tion. While initially this was mostly about textual content, over time images and videos gained popularity, as they are much easier to consume, attract more attention, and spread fur- ther than simple text. As a result, researchers started leveraging different modalities and com- binations thereof to combat online multimodal offensive content. In this study, we offer a sur- vey that carefully studies the state-of-the-art on multimodal disinformation detection cover- ing various combinations of modalities: text, images, speech, video, social media network structure, and temporal information. Moreover, while some studies focused on factuality, others investigated how harmful the content is. While these two components in the definition of disin- formation - (i) factuality, and (ii) harmfulness, are equally important, they are typically stud- ied in isolation. Thus, we argue for the need to tackle disinformation detection by taking into account multiple modalities as well as both fac- tuality and harmfulness, in the same framework. Finally, we discuss current challenges and fu- ture research directions.