Selected recent doctoral thesis projects
Fares Abawi
Multimodal Social Cue Integration for Attention Modeling and Robot Gaze Control (2025)
|
Social cues influence our attention leading to shifts in our gaze. In this thesis, we integrate non-verbal head-related social cues such as gaze direction, facial expressions, and localized sound into saliency models. We deploy these models on robots, enabling gaze control during social interactions. Using neural early fusion and late integration, we enhance audiovisual saliency prediction models. We then extend these models to predict personalized scanpath patterns. Finally, we introduce a distributed framework for deploying and evaluating our models on robotic platforms, allowing us to design robots with humanlike gaze for conducting human-robot interaction studies and cognitive robotic simulations. |
  |
Ozan Özdemir
Embodied Crossmodal Language Learning Using Neurocognitively Plausible Mechanisms (2024)
|
Despite the remarkable progress made by advanced foundation models in computer vision and natural language processing, the research on language-conditioned robotic object manipulation continues to lag behind, underscoring the need for more in-depth exploration. As evidenced by the research on early childhood development, embodiment, i.e. actively changing the environment while perceiving it via multiple senses, is crucial for language learning in human infants. Inspired by this phenomenon, embodied language learning for artificial agents tries to emulate the process of early language acquisition with computational models. As most approaches focus on language-instructed action execution but cannot produce language themselves, language learning for robots remains a challenge.
|
|
Mengdi Li
Active Vision for Embodied Agents Using Reinforcement Learning (2024)
|
Similar to humans and animals, embodied agents can only partially observe their surroundings from their egocentric perspectives at any moment. An embodied agent, when presented with a specific objective, is expected to proactively collect the necessary information to accomplish the goal. Nevertheless, the issue of insufficient observations could arise, posing challenges in the development of the active visual system.
|
|
Leyuan Qu
Neural Network Learning for Robust Speech Recognition (2021)
| Recently, end-to-end architectures have been influential for modeling of automatic speech recognition (ASR) systems, which aim to directly map acoustic inputs to character or word sequence and to simplify the training procedure. Plenty of end-to-end architectures have been proposed, for instance, Connectionist Temporal Classification (CTC), Sequence Transduction with Recurrent Neural Networks (RNN-T) and an attention-based encoder-decoder, which have accomplished significant success and performance on a variety of benchmarks or even reached human level on some tasks. However, although advanced deep neural network architectures have been proposed, in adverse environments, the performance of ASR systems suffers from significant degradation because of environmental noise or ambient reverberation. This thesis contributes to the robustness of ASR systems by leveraging additional visual sequences, face information and domain knowledge. We achieve significant improvement on speech reconstruction, speech separation, end-to-end modeling and OOV word recognition tasks. |
|
Tayfun Alpay
Periodicity, Surprisal, Attention: Skip Conditions for Recurrent Neural Networks (2021)
| Recurrent neural networks are powerful models used for sequence learning. This thesis addresses the typical limitation of recurrent neural networks to fully process and update for every timestep of an input sequence, even if the presented information is redundant or not relevant to the given task. Based on the conditional computation framework, we introduce and investigate periodicity, surprisal, and attention as constraints to model effective skipping. We evaluate the presented methodologies in the context of natural language processing and explore how we can balance potential trade-offs between model performance and skipping. | |
Tobias Hinz
Disentanglement, Compositionality, Specification: Representation Learning with Generative Adversarial Networks (2021)
| In this thesis, we introduce several novel approaches for learning good image representations with Generative Adversarial Nets (GANs) and evaluate these representations on challenging tasks such as image generation and translation, text-to-image synthesis, and few-shot learning. We focus on three different characteristics - disentanglement, compositionally, and specification - and motivate why these characteristics are beneficial for image representations. We then design several new GAN architectures to learn useful representations that exhibit these characteristics in both unsupervised and supervised ways. We show that the examined characteristics lead to meaningful improvements on many different tasks and point to several future directions in how these characteristics can be combined. | |
Chandrakant Bothe
Conversational Language Learning for Human-Robot Interaction (2020)
| Language is one of the complex but fascinating ways of communication, and it is continuously developed and maintained in the human brain. It is remarkable to study how humans understand each other in a conversation and continually learn and develop their communication skills. Understanding the meaning of the spoken or written language and interacting in that language differentiates humans from other species. Although it is difficult to define the exact working nature of the brain related to language acquisition and development, researchers find a strong relationship between different behaviours acquired based on social, cognitive, emotional and behavioural intelligence. Social robots and artificial human-like intelligent agents are the expected members of future society, where they are firmly expected to realize and exhibit verbal communication capability. In addition to the robot appearance, conversational understanding and behaviours are crucial aspects for their acceptance and co-existence in emerging society. This thesis aims to connect the knowledge from behavioural intelligence through conversational language learning with human-robot interaction (HRI). The socio-linguistic features, such as emotion, sentiment, politeness and dialogue acts, are the building blocks of the decision-making process in humans. This thesis presents extensive conversational analysis through artificial recurrent neural modelling that helps to build the robots aware of such linguistic cues. |
|
Johannes Twiefel
Robust Bidirectional Processing for Speech-controlled Robotic Scenarios (2020)
| In this thesis, we present a self-trained ASR system that performs better than Google’s cloud-based ASR on a benchmark data set. We also define a novel and simple natural language representation called Semantic Logic Predicates (SemaPreds). In our experiments, we show that we can successfully recognize SemaPreds from speech input. The approaches we developed make it possible to interpret SemaPreds, find and correct errors inside them, and evaluate their plausibility regarding a given situation. We test our novel bidirectional processing chain in a human-robot interaction scenario and show that it works robustly and performs better than a unidirectional processing pipeline. These results indicate that the novel representation and the bidirectional processing chain can be useful for other speech-controlled system scenarios. | |
Muhammad Burhan Hafez
Intrinsically Motivated Actor-Critic for Robot Motor Learning (2020)
| The performance of deep actor-critic reinforcement learning critically depends on the chosen exploration strategy. Random, undirected exploration is impractical in real-world robot learning and limits how quickly a robot can learn useful control policies. Another issue inevitable in complex robot learning tasks is using imperfect environment models for planning actions or generating experience, which results in a compounding of model errors and leads to poor task performance. In this thesis, we propose behaviorally and neurally plausible approaches to address these challenges associated with improving deep reinforcement learning for robot control. We show how learning progress-based intrinsic motivation can provide a directed exploration behavior and enable adaptive arbitration between model-based and model-free control, improving the sample efficiency of learning pixel-level control policies. Our intrinsic motivation encourages actions that lead to data that improves the model, which we use to generate imagined experiences, inspired by human mental simulation of motor behavior. | |
Nils Meins
Diversity-driven Hopfield Neural Network Ensembles for Face Detection (2019)
| In this thesis, we present a hybrid ensemble classifier approach for face detection and head-pose estimation. A key factor of ensemble classifiers is diversity. Therefore, we aim to use simple features and classifiers and focus on increasing diversity. We use the pixel sum of several rectangles, arranged in different geometrical structures as features and two classifiers which uses template matching and an additional Hopfield Neural Network to compare a feature with a learned pattern. Finally, we combine features and classifiers to a hybrid ensemble architecture to further increase diversity. | |
Jorge Dávila Chacón
Biomimetic Computation and Embodied Embedded Cognition for Spatial Audition in Humanoids (2019)
| Inspired by human behaviour, we propose an embodied embedded cognition approach to improve automatic speech recognition with robots under noisy conditions using binaural sound source localisation. Humanoids allow the generation of spatial cues in the entire audible range, without additional computational costs, and by simplifying existing biomimetic models for the extraction of spatial cues we are able to understand the principles that are important to increase accuracy. Our approach halves the sentence error rate in comparison to the standard approach of downmixing the input of both channels. | |
Luiza Mici
Unsupervised Learning of Human-Object Interactions with Neural Network Self-Organization (2018)
| The main goal of this thesis is the modeling of unsupervised learning architectures for the recognition of human-object interactions. We apply hierarchies of growing self-organizing neural networks for processing and integrating relevant cues from body motion and the action’s context. We propose and evaluate two extensions of such architecture to account for motion and action anticipation. Finally, we investigate a learning mechanism that uses the classification error for modulating neural growth. | |
Francisco Javier Cruz Naranjo
Teaching Robots With Interactive Reinforcement Learning (2017)
| We investigate learning approaches, more specifically interactive reinforcement learning to perform a domestic task. We use parent-like advice to explore two set-ups: agent-agent and human-agent interaction. This thesis contributes to knowledge in terms of studying the interplay of multi-modal interactive feedback and contextual affordances. Overall, we investigate which parameters influence the interactive reinforcement learning process and show that the apprenticeship of reinforcement learning agents can be sped up by means of interactive parent-like advice, multi-modal feedback, and affordances-driven environmental models. | |
Doreen Jirak
Inspection of Echo State Networks for Dynamic Gestures (2017)
| We investigate Echo State Networks (ESN), which implement a new training paradigm for recurrent neural networks. We first demonstrate their gesture classification performance considering two feature sets with very distinct complexity. Second, we introduce the recurrence analysis for qualitative and quantitative description of the gesture input and the system dynamics of an ESN, and show that the methodology complements classic stability measures. Finally, we address the reservoir itself and propose an algorithm for pruning connectivity in a one-shot learning scenario. |
German I. Parisi
Multimodal Learning of Actions with Deep Neural Network Self-Organization (2017)
| Understanding other people's actions plays a crucial role in our everyday lives. Human beings are able to reliably discriminate a series of socially relevant cues from body motion, with this ability being supported by highly skilled visual perception and other modalities. The main goal of this thesis is the modeling of artificial learning architectures for action perception with focus on the development of multimodal action representations. As a modeling foundation to address our research question, we focus on hierarchies of self-organizing neural networks motivated by experience-driven cortical organization. | |
Pablo Vinicius Alves de Barros
Modeling Affection Mechanisms using Deep and Self-Organizing Neural Networks (2016)
| One of the most important aspects of affective computing is how to make computational systems use emotion concepts in different situations. Although several types of research were done, we are still far away from having a system which can recognize and learn emotion concepts in a satisfactory way. In this thesis, we propose computational models which introduce a unified solution for emotional attention, recognition, and learning. These models are competitive in each of these tasks, and also provide an overview of a learning mechanism which adapts its knowledge according to a given situation. | |
Stefan Heinrich
Natural Language Acquisition in Recurrent Neural Architectures (2016)
| Our understandings of the behavioural and mechanistic characteristics for natural language are still in its infancy and we need to bridge the gap between the insights from linguistics, neuroscience, and behavioural psychology. To contribute an understanding of the appropriate characteristics in a brain-inspired neural architecture that favour language acquisition, recurrent neural models have been developed for embodied and multi-modal language processing, embedded in a developmental robotics framework. In this dissertation, the main contributions from the study of these models are reported. | |
Nicolás Navarro-Guerrero
Neurocomputational Mechanisms for Adaptive Self-Preservative Robot Behaviour (2016)
| We believe that a deeper understanding of innate and learned defensive mechanisms could also be helpful in developing future robot generations, making them more adaptable and robust. Therefore, in this thesis, we study and develop three neuro-computational self-preservative mechanisms in the context of humanoid service robots to demonstrate the potential and feasibility of including bio-inspired adaptive self-preservative mechanisms as part of real-world robotic systems. This thesis does not attempt to provide a comprehensive model of animal behaviour, but rather tries to draw attention to the need for it by presenting the potential of neglected aspects of animal behaviour such as self-preservative behaviour. | |
Wenjie Yan
Indoor Vision-based Robot Navigation: a Neural Probabilistic Approach (2016)
| We present a neural probabilistic robot localization and navigation that provides a mere concept while enabling far-reaching functionality. Through emulating several basic functionalities of the brain the system is able to achieve complex tasks such as robust target tracking, environment learning through observation, and flexible robot navigation in a home-like environment. The concept of our work is implemented and evaluated using a robot platform in a home-like environment, and the results show that our neural system helps a robot to realize different functions successfully. | |
Johannes Bauer
One Computer Scientist's (Deep) Superior Colliculus. Modeling, understanding, and learning from a multisensory midbrain structure (2015)
| The superior colliculus is a mid-brain region which integrates sensory input to localize multisensory stimuli. This thesis aims to close the gap between models of SC physiology and system-level behavior, and development thereof. A new model of the SC based on self-organized statistical learning is proposed and it is shown to replicate a variety of important biological phenomena. It is then applied to a problem in robotics: binaural sound-source localization. We show that our algorithm can learn to perform state-of-the-art sound-source localization. | |
Junpei Zhong
Artificial Neural Models for Feedback Pathways for Sensorimotor Integrations (2015)
| The brain comprises hierarchical modules on various physiological levels. Neural feedback signals modulate the neural activities via inhibitory or excitatory connections within/between these levels. They have predictive and filtering functions on the neuronal population coding of the bottom-up sensory-driven signals in the perception-action system. In this thesis, we propose that the predictive role of the feedback pathways at most levels of action and perception can be modelled by the recurrent connections in different artificial cognitive platforms. This will be examined by three recurrent neural network models. Furthermore, the three models and experiments with them show that the recurrent neural networks are able to model feedback pathways and to exhibit the feedback-related sensorimotor predictive functions. | |
Jens Kleesiek
Action-driven perception – neural architectures based on sensorimotor principles (2012)
| The active nature of perception and the intimate relation of action and cognition has been emphasized in philosophy and cognitive science for a long time. However, most of the current (computational) approaches do not consider the fundamental role of action for perception. Inspired by theories rooted in the research field of embodied cognition the scope of this thesis has been to design artificial neural architectures for the learning of sensorimotor laws. Guided by the core concept that an agent actually needs to act to perceive, a series of computational studies, including simulations and real-world robot experiments, have been conducted. It is shown that considering a set of common sensorimotor design principles results in goal-directed behavior of artificial agents. In this talk experiments as well as the underlying sensorimotor design principles will be presented and discussed. | |