Journal Article Available Online on Robotic Object Manipulation with Language In- and Output

22 November 2023

Our journal article “Learning Bidirectional Action-Language Translation with Limited Supervision and Testing with Incongruent Input” is available online in the international journal of Applied Artificial Intelligence. In this work, we tackle action-to-language and language-to-action translation in the robotic object manipulation domain under the constraint of limited supervision. Our crossmodal Transformer-based approach demonstrates high performance in realistic scenarios where only few labelled training samples are available. We also test our approach with conflicting multimodal input inspired by the concept of incongruence in psychology. The results indicate that our method behaves plausibly when it encounters conflicting input.

The link to the article on the publisher's website (open access) can be found here.