CASY: Cognitive Assistive Systems

All our cognitive performance rests on the brain and cognitive performance based on the brain still outperforms computational processing in terms of interaction or navigation. Therefore, we are pursuing a research programme how brain-inspired findings from cognitive science and neuroscience can lead to new forms of embodied neurocognitive architecture based on robust, learned and embodied processing. The research builds on our previous brain-inspired models of higher-level cognitive functions related to language, vision, working memory, and intelligent control systems (Wermter et al. 2001, Wermter et al. 2004, Murray et al. 2009, Wermter et al. 2009). Our research has led to unsupervised learning algorithms that explain neural development in the visual and motor cortex (Wermter and Elshaw 2003, Wermter et al. 2005b). Biological evidence on mirror neurons (Rizzolatti and Arbib 1998) has contributed to our models of the motor cortex (Elshaw et al. 2004, Weber et al. 2005) and the relation to language representations (Panchev and Wermter 2006, Pulvermüller and Fadiga 2010). This neurocognitive evidence from the brain is considered for the design of a computational methodology and hybrid robotic architecture based on computational neural networks and statistical symbolic representations. This brain-inspired embodied robot approach may not only lead to a better functional understanding of brain processing, but in the long run also to robust adaptive assistants functioning under noisy conditions. To focus the project we will concentrate on human robot assistance in a home environment where a learning humanoid robot will be trained on assisting with small tasks like identifying a human based on sound and vision, approaching a person, delivering small carried items and finding the way back to a charging station. Neural and statistical symbolic structures will be integrated as we have, for example, identified neural fields to be interchangeably used with statistical particle filters (Yan et al. 2011). Emphasis will be put on integrating the implementations and representations into the neural and statistical architecture.

Prof. Dr. Stefan Wermter

Language comprehension is a complex task which involves a range of different representational levels and processing components which mediate between them. In addition to the phonetics of sounds, syllables and lexical items, the range of phenomena language comprehension has to deal with includes at least syntactic and semantic regularities, which establish the meaning of isolated utterances as well as aspects of the discourse structure, which captures the relationships between them. Moreover, suprasegmental (prosodic) properties of the speech signal not only indicate sentence mode and phrasal boundaries, but also help to guide the attention of the hearer to those parts of an utterance, the speaker wants to emphasize. Usually, multi-component and multi-level architectures are adopted to cope with this multitude of different issues. Complexity increases further if multi-modal interaction, possibly with different sensory input channels, is considered. In the simplest case a dialog system is composed out of four different components, speech recognition and synthesis, a backend application, and a dialog model which connects the other components. Even in this case, non-trivial interaction patterns between the components arise if the whole system is expected to work incrementally and to exhibit an immediate response capability. This is particularly important in the context of cognitive assistive technology. The project aims at investigating such interaction patterns under the conditions of recognition uncertainty. In particular, the effect of top-down expectations originating from the verbal or situational context will be examined. They can already be studied in the interplay between a speech recognizer and a dialog model, but more sophisticated architectures including components explicitly dealing with world knowledge and user intentions might also be included here. A dialog model represents the regularities of typical verbal exchanges by means of transitions between dialog states, thus being able to capture essential aspects of discourse structure across the individual verbal contributions of the dialog partners. Moreover, it can be used to generate dynamic, i.e. dialog-state dependent predictions, which are usually highly effective in reducing the hypothesis space of the speech recognition.

Prof. Dr.-Ing. Wolfgang Menzel

The objective of this PhD topic is to investigate fundamental issues on situation awareness in assistive navigation for blind persons to access GPS-denied environments with wearable sensors and assistive display units (i.e. audio and tactile). People with normal vision orient themselves in physical space and navigate from place to place with ease, even in unfamiliar environments. However, these are challenging tasks for people who are blind or have significant visual impairments even with the help of electronic travel aids and vision techniques(Laurent and Christian 2007, Cardin et al. 2007, Ran et al. 2004, Coughlan et al. 2006). While GPS-guided travel aids show much promise in waypoint navigation in outdoor environments, there is still a paucity of technologies to help blind people to independently find doors, rooms, elevators, stairs, bathrooms, and other building amenities in GPS-denied indoor environments. The ability of blind/visually impaired people to access, understand, and explore unfamiliar environments will improve their inclusion and integration into society. The proposal is to be backed up by the previous research experience in computer vision and the interaction of different modalities of sensory and cognitive systems (Mader et al., 2008; Chen et al., 2008; Zhang et al., 2011; Elmogy et al., 2011). The aim of this project is to 1) develop assistive navigation software based on the simultaneous localization and mapping (SLAM) principle, which will be able to detect landmarks by feature extraction (both spatial and temporal, text and iconic, etc), generate and update a navigation map, register landmarks in the map, and associate new measurements with landmarks and detect loop closure to help the blind users obtaining a global perception of the environment and making travel decisions; 2) develop new methods and algorithms to understand surrounding environments (i.e. detect elevators, exits, bathrooms, doors, etc) to handle occlusions, big intra-class variations, small inter-class variations, and different views and scales by associating contextual information.

Prof. Dr. Jianwei Zhang

The use of spatial knowledge is ubiquitous in our daily life. For example, the task of selecting a residence to rent or of planning a route in an unknown territory is not possible without the spatial knowledge of the apartment. In both cases, maps, i.e. external representations, are effective means to solve such problems. External representations (graphs, charts, or diagrams), which are related to but different from maps, are based on graphical elements constituting meaning by use of their spatial properties and spatial relations among them (Kosslyn 1989, Habel & Acartürk 2007, 2011, Acartürk et al. 2008). For visually impaired people, the information presented by traditional, i.e. printed or drawn maps, graphs, etc. are not directly accessible. In order to overcome these limits, appropriate substitutions of such graphical representations are required. Whereas visual perception supports comprehension processes, which switch between global and local aspects of a graphical representation, haptic perception has a more local and in particular a more sequential character. Thus, compared to visual maps one drawback of tactile maps is the restriction of the haptic sense regarding the possibility of simultaneous perception of information (see, Loomis et al., 1991). The additional effort in haptic perception leads, for example in the case of exploring embossed maps, to specific limitations for building up cognitive maps, such as sparse density of information and less sufficient survey knowledge compared to route knowledge. Similar problems occur in exploring physically realized tactile graphs and diagrams. To overcome the problems due to the restriction of the haptic sense in simultaneous perception of information, providing additional information, such as auditory assistance through the auditory channel, has been proved to be helpful(Wang et al. 2009, Lohmann et al. 2011, Lohmann & Habel 2012). This project aims at investigating multimodal comprehension and generation on two layers, (1) the representational layer, e.g. language – graphics, and (2) the perceptual layer, in particular, the concerning sensor channels of vision, audition, and haptics. The main research questions to be investigated in this project consider the interplay between processes in different layers (and sub-layers). The similarity of the two processing structures ─ the one leading to the cognitive representation of the environment and the other to the desired action ─ promises a deeper understanding of sensory and motor cortices. A robotic actor will plan and move while considering a moving human in a dynamic environment.

Prof. Dr. Christopher Habel