A Framework for Performance Assessment of Autonomous Robotic Controllers

This work describes a framework to assess the performance of autonomous software for robotics platforms. The motivation for such an effort is twofold: (a) the difficulty to generate intensive test campaign for a given robotic architecture; (b) the lack of a general approach to assess and compare different planning and execution approaches for the same robotic platform. The produced framework, called OGATE, supports the integration, testing and operationalization of an autonomous robotic controller. It allows to run series of plan execution experiments while collecting and analyzing relevant parameters of the system under a unified and controlled environment. The first part of the paper presents the framework, an evaluation methodology for autonomous controllers and an initial deployment of OGATE to support experiments using a specific robotic system as a case study. The second part of the paper summarizes the lessons learned exploiting the proposed methodology and OGATE as an automated testbench to analyze different planning policies for the deliberative component of the targeted robot.