When the performance of robot functionalities and robot software components is evaluated, functionalities and software components are usually assumed independent from characteristics of the robot system and environment in which they operate. However, these aspects influence the performance, e.g., the performance of a software component implementing a robot functionality depends on the robot system configuration, such as which sensors are used, the sensor properties, or the robot platform kinematics, characteristics of the environment where the robot operates, and the component configuration parameters. This thesis proposes a benchmarking methodology which models the impact of the characteristics of the robot system and its environment on the performance of functionalities and their implementation as software components. However, measuring the performance of a software component for every combination of the variables which influence the performance would be untractable. To make the problem tractable, we propose to sample a relatively small number of combinations, conduct experiments for each of them, and from these results estimate a statistical model of the software component performance, which we call component performance model. To study the performance dependency between components, we build component performance models for multiple functionalities of a robot system. A performance model allows the comparison of different components implementing the same functionality to determine the best one to be used in a given setting and its optimal configuration. Moreover, the performance models enable us to predict the performance of a robot system given the performance models of its components. Two case studies illustrate application of this methodology to extract performance models: a first case study about benchmarking the Simultaneous Localization and Mapping (SLAM) functionality and the second case study focusing on an autonomous navigation system composed of a localization component and a navigation component.
Methodologies for Benchmarking of Robot Tasks and Systems