Visual Attention and Saliency Detection
Visual attention is the selection process in human vision which directs the gaze to the currently most interesting data. These might be salient regions which "pop out" of the image automatically, as in the picture on the left (bottom-up), or cues which are of current interest due to pre-knowledge about the target object or the scene (top-down). |
Computational Visual Attention and Saliency Computation:What is Computational Visual Attention?Computational visual attention deals with the detection of regions of interest in images. This is important for many applications, ranging from automatically creating thumbnails for your photo collection up to controlling the camera of an autonomous service robot.The concept of visual attention comes from one of the basic capabilities of the human visual system: the capability to focus the processing power to relevant parts of the sensory input. Two mechanisms guide the attention of humans: bottom-up and top-down mechanisms. The bottom up system guides the attention to "salient" items: items that seem to "pop out" of an image because they differ from the rest of the image according to some property such as color, orientation, or motion. Top-down attention on the other hand guides the attention to regions that are behaviorally relevant. |
Our Attention and Saliency Systems
Our Attention and Saliency Systems
We have developed several attention and saliency systems during the last years:
- VOCUS: an attention system that combines bottom-up and top-down cues [Frintrop, LNAI 2006]
- BITS: a saliency system based on information theory [Klein/Frintrop ICCV 2011]
- CoDi: an extension of BITS to continuous distributions [Klein/Frintrop DAGM 2012] [Source code CoDi 1.1]
- Simple CoDi: a simplification of CoDi that is faster and obtains more precise saliency maps [Frintrop et al. ICPR 2014]
- VOCUS2: a re-implementation of VOCUS that integrates our findings of the last years to result in an elegant, fast, and precise method for saliency, applicable to photos as well as real-world video sequences from mobile devices [accepted for CVPR 2015]
See more on our webpages on Saliency detection (bottom-up attention) and Visual search (top-down attention).
CVPR paper on saliency computation with the VOCUS2 saliency system
Saliency System VOCUS2
Introduced in:
Traditional Saliency Reloaded: A Good Old Model in New Shape (CVPR 2015)
(Simone Frintrop, Thomas Werner, and Germán Martín García)
[PDF] [Extended Abstract] [Poster] [Supplementary material]
VOCUS2 version 1.1 git repository
C++ Code :
VOCUS2 version 1.1 (Change to 1.0: Different default parameters to match the CVPR 2015 paper)
VOCUS2 version 1.0 (OLD)
(Please email us if you encounter any problems with the code or if you have suggestions or comments)
VOCUS2 is saliency system that computes a saliency map from a given input image or video. The structure of the system is based on the traditional approach of Itti et al. (PAMI 1998): Features are computed in parallel and center-surround contrast is computed by Difference-of-Gaussians on different scales. However, several subtle changes have to be made to achieve state-of-the-art performance for salient object segmentation tasks. The most important change is a Twin Pyramid that contains one Center and one Surround Pyramid, which are subtracted to obtain a Difference-of-Gaussian pyramid. In contrast to subtracting pyramid layers, this approach enables a flexible center-surround ratio. The system is fast, has a simple, coherent structure, and produces pixel-precise saliency maps.
An example result from this paper (left: image, middle: pixel-precise saliency map, right: segment-based saliency map):
You can adapt the parameters to fit different kinds of applications, in the following figure you see some example saliency maps that you can obtain by varying the most important parameters "center size" and "surround size":
Saliency detection (bottom up attention)
|
Visual search (top-down attention)
Visual search (top-down attention)
Introduction to visual attention:
-
Simone Frintrop: Computational Visual Attention, A. A. Salah and T. Gevers, editors, Computer Analysis of Human Behavior, Advances in Pattern Recognition. Springer, 2011 [PDF]
-
Simone Frintrop, Erich Rome and Henrik I. Christensen: Computational Visual Attention Systems and their Cognitive Foundation: A Survey, ACM Transactions on Applied Perception (TAP), Vol. 7, Issue 1, 2010 [PDF]
Top-down attention for robot localization
-
Simone Frintrop and Armin B. Cremers: Top-down Attention Supports Visual Loop Closing, Proc. of European Conference on Mobile Robotics (ECMR 2007), Freiburg, Germany, Sept. 2007. [PDF]
The attention system VOCUS (bottom-up + top-down):
-
Simone Frintrop: VOCUS: A Visual Attention System for Object Detection and Goal-directed Search, Phd thesis, Lecture Notes in Artificial Intelligence (LNAI), Vol. 3899 / 2006, Springer Berlin/Heidelberg. ISBN: 3-540-32759-2 [PDF], (© Springer). (see also here).
-
Simone Frintrop, Gerriet Backer, Erich Rome. Goal-directed Search with a Top-down Modulated Computational Attention System, Proceedings of the Annual Meeting of the German Association for Pattern Recognition (DAGM '05), Wien, Austria, Aug/Sept. 2005, [PDF]
Attentional scene exploration in 3D
Attentional scene exploration in 3D
Germán Martín García, Thomas Werner, and Simone Frintrop: Attentional Scene Exploration and Object Discovery in Camera and RGB-D Data Journal KI - Künstliche Intelligenz, Springer, vol. 29, Issue 1, Feb. 2015
Germán Martín García and Simone Frintrop: Computational proto-object detection in 3D data at the European Conference on Visual Perception (ECVP) 2013, Bremen, Germany [Poster]
Germán Martín García and Simone Frintrop: A Computational Framework for Attentional 3D Object Detection Proc. of the Annual Meeting of the Cognitive Science Society, July 2013 [PDF]
Towards attentive robots
Towards attentive robots
Survey article:
-
Simone Frintrop: Towards Attentive Robots, PALADYN Journal of Behavioral Robotics, Springer, Vol. 2, Issue 2, 2011 Download the article here. If you don't have access, you can download a preprint here.
Applications of attention in robotics
Object Discovery:
-
Germán Martín García and Simone Frintrop: A Computational Framework for Attentional 3D Object Detection Proc. of the Annual Meeting of the Cognitive Science Society, July 2013 [PDF]
Visual object and person tracking:
-
Ali Borji, Simone Frintrop, Dicky Sihite and Laurent Itti Adaptive Object Tracking by Learning Background Context In: Proc. of the Egocentric Vision Workshop at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2012
-
Simone Frintrop: General Object Tracking with a Component-based Target Descriptor, IEEE 2010 International Conference on Robotics and Automation (ICRA '10), Anchorage, Alaska, May 2010 [PDF]
-
Simone Frintrop, Achim Königs, Frank Hoeller and Dirk Schulz: A Component-based Approach to Visual Person Tracking from a Mobile Platform, International Journal of Social Robotics, Special Issue on People Detection and Tracking, Vol. 2, Issue 1, pp. 53-62, Springer Netherlands, 2010 [PDF]
-
Simone Frintrop, Achim Königs, Frank Hoeller and Dirk Schulz: Visual Person Tracking Using a Cognitive Observation Model, Workshop on People Detection and Tracking at the IEEE 2009 International Conference Robotics and Automation (ICRA '09), Kobe, Japan, May 2009 [PDF]
-
Simone Frintrop and Markus Kessel: Most Salient Region Tracking, IEEE 2009 International Conference on Robotics and Automation (ICRA '09), Kobe, Japan, May 2009 [PDF]
-
Simone Frintrop and Markus Kessel: Cognitive Data Association for Visual Person Tracking, Proc. of the 1st IEEE Workshop on Human Detection from Mobile Platforms (HDMP '08) , Pasadena, California, May 2008 [PDF]
Visual robot localization and mapping:
-
Simone Frintrop and Patric Jensfelt: Attentional Landmarks and Active Gaze Control for Visual SLAM, IEEE Transactions on Robotics, Special Issue on Visual SLAM, vol. 24, no. 5, Oct. 2008. Download the article here. If you don't have access, you can download a preprint here.
-
Simone Frintrop and Patric Jensfelt: Active Gaze Control for Attentional Visual SLAM, Proceedings of the IEEE 2008 International Conference Robotics and Automation (ICRA '08), Pasadena, California, May 2008 [PDF]
-
Simone Frintrop, Patric Jensfelt and Henrik Christensen: Simultaneous Robot Localization and Mapping Based on a Visual Attention System, in Attention in Cognitive Systems, Lecture Notes on Artificial Intelligence (LNAI), 4840, Springer-Verlag, 2007 [PDF]
-
Simone Frintrop and Armin B. Cremers: Top-down Attention Supports Visual Loop Closing, Proc. of European Conference on Mobile Robotics (ECMR 2007), Freiburg, Germany, Sept. 2007. [PDF]
-
Simone Frintrop: Visual Robot Localization and Mapping based on Attentional Landmarks, German Conference on Artificial Intelligence (KI), Osnabrück, Germany, Sept. 2007 [PDF]
-
Simone Frintrop, Patric Jensfelt and Henrik Christensen: Attentional Robot Localization and Mapping, at the ICVS Workshop on Computational Attention & Applications (WCAA), Bielefeld, Germany, March 2007 [PDF]
-
Simone Frintrop, Patric Jensfelt, Henrik Christensen: Attentional Landmark Selection for Visual SLAM, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'06), October 2006. [PDF]
-
Simone Frintrop, Patric Jensfelt, Henrik Christensen: Detecting Useful Landmarks for Visual SLAM , Workshop Robotic 3D Environment Cognition, at the International Conference Spatial Cognition, Bremen (Germany), Sept. 2006
Multi-modal attention (saliency on 3D laser scanner data):
-
Simone Frintrop, Erich Rome, Andreas Nüchter, and Hartmut Surmann: A Bimodal Laser-Based Attention System, Journal of Computer Vision and Image Understanding (CVIU), vol. 100, no. 1-2, Oct-Nov 2005, Special Issue on Attention and Performance. [PDF]
-
Simone Frintrop, Andreas Nüchter, Hartmut Surmann, and Joachim Hertzberg: Saliency-based Object Recognition in 3D Data, In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'04), September 2004. [PDF]
- S. Frintrop; A. Nüchter; H. Surmann: Visual Attention for Object Recognition in Spatial 3D Data. In: Proceedings of the 2nd International Workshop on Attention and Performance in Computational Vision (WAPCV 2004) [PDF]
- S. Frintrop; E. Rome; A. Nüchter; H. Surmann Applying Attentional Mechanisms to Bi-modal 3D Laser Data. In: International Workshop on Attention and Performance in Computer Vision (WAPCV 2003) [PDF]
- S. Frintrop; E. Rome; A. Nüchter; H. Surmann An Attentive, Multi-modal Laser "Eye". In: Proc. of 3rd International Conference on Computer Vision Systems (ICVS 2003) [PDF]
Object recognition
-
Simone Frintrop, Andreas Nüchter, Kai Pervölz, Hartmut Surmann, Sara Mitri, Joachim Hertzberg: Attentive Classification, International Journal of Applied Artificial Intelligence in Engineering Systems, Vol. 1, Number 1, June 2009
-
Sara Mitri, Simone Frintrop, Kai Pervölz, Hartmut Surmann, Andreas Nüchter. Robust Object Detection at Regions of Interest with an Application in Ball Recognition, in Proceedings IEEE 2005 International Conference Robotics and Automation (ICRA '05), Barcelona, Spain, April 2005, [PDF]
-
Simone Frintrop, Andreas Nüchter, Hartmut Surmann, and Joachim Hertzberg: Saliency-based Object Recognition in 3D Data, In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'04), September 2004. [PDF]
Publications
- Soubarna Banik, Mikko Lauri, Alois Knoll, Simone Frintrop: Object Localization with Attribute Preference based on Top-Down Attention, International Conference on Computer Vision Systems (ICVS) 2021, [PDF]
- Zhen Deng, Ge Gao, Simone Frintrop, Fuchun Sun, Changshui Zhang, Jianwei Zhang: Attention based visual analysis for fast grasp planning with a multi-fingered robotic hand, in Frontiers in Neurorobotics, 2019. DOI: 10.3389/fnbot.2019.00060, [Paper], [PDF]
- Christian Wilms, Simone Frintrop: AttentionMask: Attentive, Efficient Object Proposal Generation Focusing on Small Objects, Asian Conference on Computer Vision (ACCV), 2018 [webpage], [PDF]
- Ge Gao, Mikko Lauri, Jianwei Zhang, Simone Frintrop: Saliency-guided Adaptive Seeding for Supervoxel Segmentation, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, [PDF], [arXiv], [
- Germán Martín García, Mircea Pavel, and Simone Frintrop: A Computational Framework for Attentional Object Discovery in RGB-D Videos, in "Cognitive Processing" 18(2) (Springer), 2017 [PDF]
- Thomas Werner, Germán Martín García, and Simone Frintrop: Saliency-Guided Object Candidates Based on Gestalt Principles, International Conference on Computer Vision Systems (ICVS), Copenhagen, July 2015 [PDF] (Winner of the best paper award)
- Simone Frintrop, Thomas Werner, and Germán Martín García: Traditional Saliency Reloaded: A Good Old Model in New Shape, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, June 2015 [PDF] [Extended Abstract] [Poster] [Supplementary material] [Code and more details]
- Simone Frintrop, Thomas Werner, and Germán Martín García: Twin Pyramids for Saliency Computation and the Application to Object Proposal Generation, Workshop WiCV at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, June 2015 [Extended Abstract] [Poster]
- Esther Horbert, Germán Martín García, Simone Frintrop, and Bastian Leibe: Sequence-Level Object Candidates Based on Saliency for Generic Object Recognition on Mobile Systems, IEEE International Conference on Robotics and Automation (ICRA 2015), Seattle, May 2015 [PDF] (Finalist (nominated) for the best vision paper award)
- Germán Martín García, Ekaterina Potapova, Thomas Werner, Michael Zillich, Markus Vincze, and Simone Frintrop: Saliency-based Object Discovery on RGB-D Data with a Late-Fusion Approach, IEEE International Conference on Robotics and Automation (ICRA 2015), Seattle, May 2015 [PDF]
- Simone Frintrop, Lina Jackel, and Ulrich Ettinger: A Comparison of Bottom-up and Top-down Salience between Human Vision and a Computer Model of Visual Attention, 18th European Conference on Eye Movements, Vienna, Austria, 2015
- Germán Martín García, Thomas Werner, and Simone Frintrop: Attentional Scene Exploration and Object Discovery in Camera and RGB-D Data Journal KI - Künstliche Intelligenz, Springer, vol. 29, Issue 1, Feb. 2015. Preprint: [PDF]
- Lei Zhu, Dominik A. Klein, Simone Frintrop, Zhiguo Cao, and Armin B. Cremers: A Multisize Superpixel Approach for Salient Object Detection Based on Multivariate Normal Distribution Estimation PDF Saliency maps, In: IEEE Transactions on Image Processing (TIP), Vol. 23(12), December 2014 [PDF] [Saliency Maps]
- Germán Martín García and Simone Frintrop: A Computational Framework for Attentional 3D Object Detection Proc. of the Annual Meeting of the Cognitive Science Society, July 2013 [PDF]
- Germán Martín García, Simone Frintrop, and Armin B. Cremers: Attention-Based Detection of Unknown Objects in a Situated Vision Framework KI - Künstliche Intelligenz, Springer, 27 (3), 2013
The final publication is available at Springer via http://dx.doi.org/10.1007/s13218-013-0256-1 - Lei Zhu, Dominik A. Klein, Simone Frintrop, Zhiguo Cao, and Armin B. Cremers: Multi-Scale Region-Based Saliency Detection Using W2 Distance on N-Dimensional Normal Distributions International Conference on Image Processing (ICIP) September 15-18, 2013, Melbourne, Australia [PDF]
- Dominik A. Klein and Simone Frintrop: Salient Pattern Detection using W2 on Multivariate Normal Distributions Joint Pattern Recognition Symposium of the German Association for Pattern Recognition (DAGM) and the Austrian Association for Pattern Recognition (OAGM) (DAGM-OAGM), August 28-31, 2012, Graz, Austria [PDF] [Source code CoDi Saliency System]
- Dominik A. Klein, Germán Martín García, Simone Frintrop A computational model for saliency detection based on probability distributions European Conference on Visual Perception (ECVP), Sept 2012, Alghero, Italy [Poster]
- Simone Frintrop: Computational Visual Attention, A. A. Salah and T. Gevers, editors, Computer Analysis of Human Behavior, Advances in Pattern Recognition. Springer, 2011 [PDF]
- Simone Frintrop, Erich Rome and Henrik I. Christensen: Computational Visual Attention Systems and their Cognitive Foundation: A Survey, ACM Transactions on Applied Perception (TAP), Vol. 7, Issue 1, 2010 [PDF]
An overview of the work was presented as invited talk at the Symposium on Applied Perception in Graphics and Visualization (APGV), August 27th, 2011 [Slides of talk (pdf)] - Simone Frintrop and Patric Jensfelt: Attentional Landmarks and Active Gaze Control for Visual SLAM, IEEE Transactions on Robotics, Special Issue on Visual SLAM, vol. 24, no. 5, Oct. 2008. Download the article here. If you don't have access, you can download a preprint here.
- Simone Frintrop and Patric Jensfelt: Active Gaze Control for Attentional Visual SLAM, Proceedings of the IEEE 2008 International Conference Robotics and Automation (ICRA '08), Pasadena, California, May 2008 [PDF]
- Simone Frintrop, Patric Jensfelt and Henrik Christensen: Simultaneous Robot Localization and Mapping Based on a Visual Attention System, in Attention in Cognitive Systems, Lecture Notes on Artificial Intelligence (LNAI), 4840, Springer-Verlag, 2007 [PDF]
- Simone Frintrop and Armin B. Cremers: Top-down Attention Supports Visual Loop Closing, Proc. of European Conference on Mobile Robotics (ECMR 2007), Freiburg, Germany, Sept. 2007. [PDF]
- Simone Frintrop: Visual Robot Localization and Mapping based on Attentional Landmarks, German Conference on Artificial Intelligence (KI), Osnabrück, Germany, Sept. 2007 [PDF]
- Simone Frintrop, Patric Jensfelt and Henrik Christensen: Attentional Robot Localization and Mapping, at the ICVS Workshop on Computational Attention & Applications (WCAA), Bielefeld, Germany, March 2007 [PDF]
- Simone Frintrop, Maria Klodt, Erich Rome:
A Real-time Visual Attention System Using Integral Images, in Proc. of the 5th International Conference on Computer Vision Systems (ICVS 2007), Bielefeld, Germany, March 2007. [PDF] - Simone Frintrop, Patric Jensfelt, Henrik Christensen: Attentional Landmark Selection for Visual SLAM, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'06), October 2006. [PDF]
- Simone Frintrop, Patric Jensfelt, Henrik Christensen: Pay Attention When Selecting Features , Proc. of the 18th Int'l Conference on Pattern Recognition (ICPR 2006), Hong Kong, Aug. 2006. [PDF]
- Simone Frintrop: VOCUS: A Visual Attention System for Object Detection and Goal-directed Search, Phd thesis, Lecture Notes in Artificial Intelligence (LNAI), Vol. 3899 / 2006, Springer Berlin/Heidelberg. ISBN: 3-540-32759-2
[PDF], (© Springer). (see also here). - Simone Frintrop, Gerriet Backer, Erich Rome. Goal-directed Search with a Top-down Modulated Computational Attention System, Proceedings of the Annual Meeting of the German Association for Pattern Recognition (DAGM '05), Wien, Austria, Aug/Sept. 2005, [PDF]
- Simone Frintrop, Gerriet Backer, Erich Rome. Selecting what is Important: Training Visual Attention, Proceedings of the 28th German Conference on Artificial Intelligence (KI '05), Koblenz, Germany, Sept. 2005, [PDF]
- Simone Frintrop, Erich Rome, Andreas Nüchter, and Hartmut Surmann: A Bimodal Laser-Based Attention System, Journal of Computer Vision and Image Understanding (CVIU), vol. 100, no. 1-2, Oct-Nov 2005, Special Issue on Attention and Performance. [PDF]
- Simone Frintrop, Erich Rome, and Gerriet Backer: A Goal-directed Visual Attention System. Poster Session of 8th Tübingen Perception Conference (Tübinger Wahrnehmungskonferenz) (TWK 2005). Conference: Tübingen, Germany, Feb 27-28, 2005
- Simone Frintrop, Andreas Nüchter, Hartmut Surmann, and Joachim Hertzberg: Saliency-based Object Recognition in 3D Data, In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'04), September 2004. [PDF]
- Simone Frintrop and Erich Rome: Simulating Visual Attention for Object Recognition, Proceedings of the Workshop on Early Cognitive Vision, Isle of Skye, Scotland, May 2004. [PDF]
- S. Frintrop; A. Nüchter; H. Surmann: Visual Attention for Object Recognition in Spatial 3D Data. In: Proceedings of the 2nd International Workshop on Attention and Performance in Computational Vision (WAPCV 2004), revised post-conference proceedings: Springer LNCS Vol. 3368, Berlin ISBN 3-540-24421-2 L. Paletta, J. Tsotsos, E. Rome, G.W. Humphreys (eds), pp. 168-182, Conference: Prague, Czech Republic, May 15, 2004 [PDF]
- S. Frintrop; E. Rome; A. Nüchter; H. Surmann Applying Attentional Mechanisms to Bi-modal 3D Laser Data. In: International Workshop on Attention and Performance in Computer Vision (WAPCV 2003), L. Paletta, G.W. Humphreys, and R.B. Fisher (eds), pp. 25-30, Conference: Graz, Austria, April 3, 2003 [PDF]