A deep learning approach for automatic video coding of deictic gestures in children with autism

Autism is a heterogeneous neurodevelopmental condition that may include several deficits affecting social-communication skills, learning and behaviours. In this scenario, the early detection of deictic gestures plays an important role for developmental behavioural intervention of children with autism. Despite it is widely acknowledged, it has not been sufficiently explored by artificial intelligence models. To achieve this, the paper proposes an automatic digital coding approach based on deep learning models. It has been applied on 37 video clips of naturalistic mother-child interactions with the aim to recognize four main deictic gestures: pointing, giving, showing and requesting. We evaluate a deep multi-frame model with transformer-based architecture in python code trained on 26 clips, validated on 5 clips and tested on 6 clips. The system is based on a preprocessing phase based on resize of each video frame 128x128 and a 1024 feature extractor based on Densenet121 pretrained on Imagenet database followed by a soft-max classification layer. Preliminary results based on highest-probability of 4 recognized actions showed respectively an accuracy of training set of 100%, validation set of 80% and testing set of 67%. These findings suggests that the deep architecture based on multi-frame modelling is a very promising approach for analysis of deictic gestures. As future work, we plan to perform further validations on a higher number of samples to reach higher and reliable performances.