Activity Recognition Based on RGB-D and Thermal Sensors for Socially Assistive Robots

For socially assistive robots, being able to recognize basic human actions is an important capability. The sensors, which are frequently mounted on most recent robots, such as RGB-D and thermal cameras, as well as the advances in deep learning have enabled the research on activity recognition to grow. In this paper, we collected our own dataset of actions in a home-like scenario, which contains thermal imagery in addition to RGB-D data and we proposed a method based on Long-term Recurrent Convolutional Networks (LRCN). We showed that our method has an accuracy comparable with the state-of-the-art. We also proved that thermal information can improve the recognition accuracy. Furthermore, we tested the real-time capability of our system and conducted a real-time experiment with a robot (Pepper robot from Softbank Robotics) so as to investigate the effect of a robot enabled with action recognition capability in a human-robot interaction.