Neural-Network-based Action-Object Semantics for Assistive Robotics

Human daily activities in domestic scenarios are often interactions with objects. Therefore, we address human action recognition as a complex task of integration and fusion of information from different visual analysis: objects present in the scene (object detection and recognition), human body pose and motion (action recognition) and the correlation between human and objects during an interaction.

To date, there is no common framework based on neural network techniques able to learn and represent the semantic relationship between objects and actions. We approach this problem by applying self-organizing neural architectures considering their inherent capability to not only extract significant prototypical examples but also to learn associations from the input data.

This project aims to design, implement and evaluate a self-organizing neural architecture able to jointly learn meaningful representations of body pose sequences and manipulated objects and to combine the obtained representations for learning semantic relationship between them. At a later stage of this project, it is intended to integrate the implemented architecture on a robotic platform for evaluating its capabilities in both learning and generating the learned object-manipulation sequences.