PARLOMA - A Novel Human-Robot Interaction System for Deaf-blind Remote Communication

We present a novel system for remotely controlling an anthropomorphic robotic hand using gestures. Our system uses a low-cost depth sensor as the only input device. The basic idea behind the resented system is that a user is able to perform hand shapes in front of the depth sensor; the system is able to segment the user's hand from the background and to recognize a set of hand postures; recognized postures are then sent over the web and reproduced by the a remote robotic hand. The system is expected to enhance communication among Deaf-blind signers by allowing, for the first time, remote transmission of Sign Language (SL) in the tactile modality. Usually, Deaf-blind signers can only receive external information by tactile exploration of the environment. Communication is therefore possible by using a special variety of SL, the tactile-SL (t-SL). The proposed system will basically work as a "telephone" for Deaf-blind people. It will receive a SL input (which can be produced by all signers, including Deaf-blind) and it will provide a t-SL output (the only one accessible to Deaf-blind signers). The system will convert a SL message into its tactile variety. It does not manipulate the semantic nor the structural content of the message, and therefore it is language-independent (i.e. it can be used with any SL, t-SL pair). The system has been designed after consulting the main Italian Deaf-blind associations. A Deaf-blind representative also tested a preliminary version of the system. In this paper, we report the results of a first set of experiments, showing that the developed system is accurate enough to reproduce handshapes (a crucial component of the SL message). This system is a first prototype for the Parloma project, which aims at designing a remote communication system for Deaf-blind people, but could also be useful in other scenarios (e.g., tele-rehabilitation, tele-presence . . . ).