Deciding which skill to learn when: Temporal-Difference Competence-Based Intrinsic Motivation (TD-CB-IM)

Intrinsic motivations, the topic of this paper, can be defined by contrasting them to extrinsic motivations. Extrinsic motivations are directed to drive and guide the learning of behaviour directed to satisfy homeostatic regulations related to the organisms' survival and repro- duction. Intrinsic motivations, instead, are motivations that serve the evolutionary function of acquiring knowledge (e.g., the capacity to pre- dict) and competence (i.e. the capacity to do) in the absence of extrinsic motivations: this knowledge and competence can be later exploited for producing behaviours that enhance biological fitness. While knowledge- based intrinsic motivation mechanisms have been widely modelled and studied, competence-based intrinsic motivation mechanisms (CB-IM), usable for guiding learning on the basis of competence level or improve- ment, have been much less investigated. The goal of this paper is to clarify the nature and possible roles of CB-IM mechanisms for learning, in particular in relation to the cumulative acquisition of a repertoire of skills underlying the capacity to accomplish multiple tasks in the environ- ment, and then to review a specific CB-IM mechanism. This mechanism (TD-CB-IM) measures the improvement rate of competence on the ba- sis of the Temporal-Difference learning signal (TD-error) used in several reinforcement learning (RL) models. The effectiveness of the mechanism is supported by reporting the results of some experiments where the TD- CB-IM mechanism is successfully exploited by a hierarchical RL model controlling a simulated navigating robot to decide when to train different skills in different environmental conditions. The mechanism is one of the few CB-IM mechanisms proposed in the computational literature.