Published Article in TCDS

11 July 2020

Photo: cis.ieee.org

Knowledge Technology got a new journal article published in IEEE Transactions on Cognitive and Developmental Systems as a result of a collaboration with the group of Prof. Zhiyuan Liu at the Natural Language Processing Lab, Department of Computer Science, Tsinghua University, Beijing.

Title: Integrating Image-Based and Knowledge-Based Representation Learning
Authors: Ruobing Xie, Stefan Heinrich, Zhiyuan Liu, Cornelius Weber, Yuan Yao, Stefan Wermter, and Maosong Sun
Abstract:
A variety of brain areas is involved in language understanding and generation, accounting for the scope of language that can refer to many real-world matters. In this paper, we investigate how regularities among real-world entities impact emergent language representations. Specifically, we consider knowledge bases, which represent entities and their relations as structured triples, and image representations, which are obtained via deep convolutional networks. We combine these sources of information to learn representations of an image-based knowledge representation learning (IKRL) model. An attention mechanism lets more informative images contribute more to the image-based representations. Evaluation results show that the model outperforms all baselines on the tasks of knowledge graph (KG) completion and triple classification. In analyzing the learned models, we found that the structure-based and image-based representations integrate different aspects of the entities and the attention mechanism provides robustness during learning.