University of Hamburg - to research, to teach, to educate and form, to homepage
Computer Vision (CV)

Photo: UHH/Lutsch

Computer Vision

Situated Vision to Perceive Object Shape and Affordances

DFG Project: Situated Vision to Perceive Object Shape and Affordances

The objective of this project is to provide models and methods to detect, recognize, and categorize the 3D shape of everyday objects and their affordances in homes. To tackle these challenges, we propose the Situated Vision paradigm and develop 3D visual perception capabilities from the view of a robot. The Situated Vision approach is inspired by recent work in cognitive science and neuroscience: it fuses qualitative and quantitative cues to extract and group 3D shape elements and relate them to affordance categories. Cognitive mechanisms such as situation-based visual attention and task-oriented visual search focus the processing on relevant scene parts. Perception integrates quantitative and qualitative shape information from multiple 2D and 3D measurements. The analysis of the shapes is used to find instances of semantic 3D concepts, such as "puttable surface", that can be used to find semantic entities and to learn affordance categories. To show the generality of the proposed approach, the system will be tested in three typical home scenarios with varying complexity. Four renowned research teams combine their experience to show that the combination of attention (Uni Bonn), categorization (RWTH Aachen), shape perception (TU Vienna) and learning (IDIAP) will bring about a big step forward in cognitive robotics.

Keywords: Computer vision, cognitive vision, robotics, attention, shape, affordances.

Project Leader:

  • UBO University of Bonn (PD Dr. Simone Frintrop)

Project Partner:

  • Idiap Research Institute (Prof. Dr. Barbara Caputo, now at University of Rome)
  • RWTH_AACHEN RWTH Aachen (Prof. Dr. Bastian Leibe)
  • TUW Technical University of Vienna (Prof. Dr. Markus Vincze)

Project work at Uni Bonn:

  • People: Germán Martín García, Simone Frintrop
  • Topic: Scene exploration and Visual Search
  • Publications:

    • Simone Frintrop, Thomas Werner, and Germán Martín García: Traditional Saliency Reloaded: A Good Old Model in New Shape, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, June 2015 [PDF] [Extended Abstract] [Poster] [Supplementary material] [Code and more details]

    • Simone Frintrop, Thomas Werner, and Germán Martín García: Twin Pyramids for Saliency Computation and the Application to Object Proposal Generation, Workshop WiCV at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, June 2015 [Extended Abstract] [Poster]

    • Thomas Werner, Germán Martín García, and Simone Frintrop: Saliency-Guided Object Candidates Based on Gestalt Principles, accepted for the International Conference on Computer Vision Systems (ICVS), Copenhagen, July 2015

    • Esther Horbert, Germán Martín García, Simone Frintrop, and Bastian Leibe: Sequence-Level Object Candidates Based on Saliency for Generic Object Recognition on Mobile Systems, IEEE International Conference on Robotics and Automation (ICRA 2015), Seattle, May 2015 [PDF]

    • Germán Martín García, Ekaterina Potapova, Thomas Werner, Michael Zillich, Markus Vincze, and Simone Frintrop: Saliency-based Object Discovery on RGB-D Data with a Late-Fusion Approach, IEEE International Conference on Robotics and Automation (ICRA 2015), Seattle, May 2015 [PDF]

    • Germán Martín García, Thomas Werner, and Simone Frintrop: Attentional Scene Exploration and Object Discovery in Camera and RGB-D Data Journal KI - Künstliche Intelligenz, Springer, vol. 29, Issue 1, Feb. 2015. Preprint: [PDF]

    • Germán Martín García, Simone Frintrop, and Armin B. Cremers: Systematicity and Compositionality in Computer Vision In Proc. of KogWis 2014 (Meeting of the German Cognitive Science Society), Sept. 29th - Oct. 2, 2014, in Cognitive Processing, Springer, vol 15, issue 1, Sept. 2014 [PDF]

    • Simone Frintrop, Germán Martín García, and Armin B. Cremers: A Cognitive Approach for Object Discovery, at the International Conference on Pattern Recognition (ICPR), Stockholm, Sweden [PDF]

    • Germán Martín García, Simone Frintrop, and Armin B. Cremers: Attention-Based Detection of Unknown Objects in a Situated Vision Framework KI - Künstliche Intelligenz, Springer, 27 (3), 2013
      The final publication is available at Springer via http://dx.doi.org/10.1007/s13218-013-0256-1

    • Germán Martín García and Simone Frintrop: Computational proto-object detection in 3D data at the European Conference on Visual Perception (ECVP) 2013, Bremen, Germany [Poster]

    • Germán Martín García and Simone Frintrop: A Computational Framework for Attentional 3D Object Detection Proc. of the Annual Meeting of the Cognitive Science Society, July 2013 [PDF]