EchoRob: Echo State Networks for Developing Language Robots

EchoRob is a research project developed in the Knowledge Technology group of Hamburg University funded by the Marie Curie IEF programme between March 2015 and February 2016.

Persons participating in the project

Leading Investigators: Dr. X. Hinaut, Prof. Dr. Stefan Wermter
Associates: J. Twiefel, Dr. S. Magg

Description

In this project, we propose to build a biologically plausible model of sentence comprehension based on recurrent neural networks called reservoirs. Human sentence comprehension is mainly handled by prefrontal cortex areas, which have highly recurrent connectivity. In both biological and artificial neural networks this recurrence is supposed to enable the management of different aspects of time such as working memory and contextual information processing.
Here, we propose to develop the Reservoir Computing (RC) paradigm — in particular Echo State Networks (ESN) with incremental learning — to model language comprehension at the sentence level given sequential inputs of words or phonemes. Based on our initial research, a model processing syntactic sentence structures was able to demonstrate generalisation and online prediction capabilities while processing sequential input.

For less frequent inputs, the model provided potential explanation for human electrophysiological data. Building on this research, a new model is proposed with the following objectives:

processing of all semantic information enabling contextual processing and richer representation of meaning,
implementing incremental learning with noisy supervision enabling realistic developmental language acquisition from simple to complex sentences,
demonstrating that the model can learn from naïve user's utterances in several languages, and
demonstrating the ability of this model when embodied in a robot to acquire extended language capabilities through human-robot interaction accounting for developmental schemes.

Outreach

On Saturday 7 November 2015, Dr. X. Hinaut and J. Twiefel participated at the 6th Nacht des Wissens (Night of Science) in Hamburg. At their booth, they showed a demo with a Nao humanoid robot talking to visitors. They also showed a video explaining how a robot learns to name objects based on Convolutional Neural Networks, the DOCKS system and the sentence comprehension model central to the EchoRob project. Several motivated Bachelor and Master students helped as well to explain to numerous visitors how the whole system works (in German and English).

Publications

Hinaut, X., Twiefel, J., Petit, M., Dominey, P., Wermter, S. A Recurrent Neural Network for Multiple Language Acquisition: Starting with English and French, Conference on Neural Information Processing Systems (NIPS 2015), Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, Montreal, Canada, 2015.
Hinaut, X., Twiefel, J., Borghetti Soares, M., Barros, P., Mici, L., Wermter, S. Humanoidly Speaking – How the Nao humanoid robot can learn the name of objects and interact with them through common speech. International Joint Conference on Artificial Intelligence (IJCAI), Video Competition, Buenos Aires, Argentina, 2015. Video: Humanoidly Speaking
Hinaut, X., Wermter, S. An Incremental Approach to Language Acquisition: Thematic Role Assignment with Echo State Networks. In Wermter, S., et al., editors. Proceedings of the 24th International Conference on Artificial Neural Networks (ICANN 2014), pp. 33-40, Springer Heidelberg. Hamburg, DE, September 2014.

Acknowledgments

This research project is supported by a Marie Curie Intra European Fellowship within the 7th European Community Framework Programme: EchoRob project (PIEF-GA-2013-627156).