Dr. Jae Hee Lee

Photo: Jae Hee Lee
Postdoctoral Research Associate TRR CML Project
Knowledge Technology
Address
Office
Contact
Homepage: http://jaeheelee.de
About Me
I am a postdoctoral research associate in the Knowledge Technology Group, University of Hamburg. I develop representation learning models for grounded language understanding, with an emphasis on robustness and explainability. My prior work includes symbolic approaches to AI, specifically spatio-temporal reasoning and multiagent systems.
Research Interests
- Multimodal Deep Learning
- Explainable Artificial Intelligence
- Neuro-Symbolic Artificial Intelligence
- Spatio-Temporal Reasoning
Research Experience
- Neurocognitive Models of Crossmodal Language Learning, University of Hamburg, Hamburg, Germany (2020–present)
- Formal Lexically Informed Logics for Searching the Web, Cardiff University, Cardiff, UK (2018–2020)
- Feodor Lynen Research Fellowship by Alexander von Humboldt Foundation, University of Technology Sydney, Sydney, Australia (2016–2017)
- Artificial Intelligence Meets Wireless Sensor Networks, The Australian National University, Canberra, Australia (2015)
- Reasoning about Paths, Shapes, and Configuration, University of Bremen, Bremen, Germany (2009–2014)
Education
- Dr. rer. nat. in Computer Science, University of Bremen, Germany (2009–2013)
- Visiting PhD student, North Carolina State University, USA (2012)
- Visiting PhD student, University at Buffalo, USA (2011)
- Diplom in Mathematics, University of Bremen, Germany (2003–2009)
- DAAD exchange student, Seoul National University, South Korea (2008)
Research Grants and Awards
- Bremen-Cardiff Alliance Collaborative Fund, Cardiff University, UK (2019)
- UTS Blue Sky Research Scheme, University of Technology Sydney, Sydney (2017)
- Feodor Lynen Research Fellowship, Alexander von Humboldt Foundation, Germany (2016–2017)
- Fachbereichspreis Mathematik (diploma thesis award), University of Bremen (2009)
- DAAD-KOSEF Scholarship, DAAD, Germany (2008)
- Third Prize at the Regional Contest Jugend Forscht Bremen (2001)
- Senator-Friedrich-Cordewener-Prize at the Regional Contest Jugend Forscht Bremen (2001)
Recent Teaching Activities
- Lecture on “Transformers and Crossmodal Learning”, University of Hamburg (2023) [Slides]
- Supervisor, Neural Networks Seminar, University of Hamburg (2023)
- Organizer, WISDUM Reading Group, Knowledge Technology, University of Hamburg (2023)
- Supervisor, Bio-inspired Artificial Intelligence Seminar, University of Hamburg (2022)
Thesis Supervision
- Generalization of Transformer-Based Models on Visual Question Answering Tasks, Frederic Voigt, MSc (2023)
- Improving Compositional Generalization By Learning Concepts Individually, Ramtin Nouri, MSc (2023)
- Learning Concepts a Developmental Lifelong Learning Approach to Visual Question Answering, Ramin Farkhondeh, BSc (2022)
- Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks, Jakob Ambsdorf, MSc (2022), now PhD student at University of Copenhagen
- Tackling The Binding Problem And Compositional Generalization In Multimodal Language Learning, Caspar Volquardsen, BSc (2021), ICANN 2022
- Learning Bidirectional Translation Between Robot Actions and Linguistic Descriptions, Markus Heidrich, BSc (2021)
- Using the Reformer for Efficient Summarization, Yannick Wehr, BSc (2020)
- Generalization in Multi-Modal Language Learning from Simulation, Aaron Eisermann, BSc (2020), IJCNN 2021
- Commonsense Validation and Explanation, Christian Rahe, BSc (2020)
Blog Posts
News
- MSc student Björn Plüster, whom I advise, developed a German LLM called LeoLM with LAION and hessian.AI!
- Our paper Internally Rewarded Reinforcement Learning got accepted to ICML 2023.
- I am co-organizing the 2nd International Workshop on Spatio-Temporal Reasoning and Learning.
- Program committee member, EMNLP 2023.
- Program committee member, ECAI 2023.
- Program committee member, ACL 2023.
- Program committee member, IJCAI 2023.
- Received ~10K € research funding for a student project on “explainable visual question answering” that I am advising. Check out our paper!
Selected Papers
Multimodal Deep Learning:
- J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022. [Code]
- O. Özdemir, M. Kerzel, C. Weber, J.H. Lee, M.B. Hafez, P. Bruns, S. Wermter, Learning Bidirectional Action-Language Translation with Limited Supervision and Testing with Incongruent Input, Applied Artificial Intelligence, 2023.
- M. Li, C. Weber, M. Kerzel, J.H. Lee, Z. Zeng, Z. Liu, S. Wermter, Robotic Occlusion Reasoning for Efficient Object Existence Prediction IROS 2021. [Code]
- A. Eisermann, J.H. Lee, C. Weber, S. Wermter, Generalization in Multimodal Language Learning from Simulation, IJCNN 2021.
Explainable Artificial Intelligence:
-
J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, (under review).
-
B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022. [Code]
Neuro-Symbolic Reasoning:
-
J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429 [arXiv Preprint]
-
J. H. Lee et al. Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022. [Code]
Other Machine Learning Topics:
- M. Li, X. Zhao, J.H. Lee, C. Weber, S. Wermter, Internally Rewarded Reinforcement Learning, ICML 2023 [Project page] [Code].
- J.H. Lee, J. Camacho-Collados, L. Espinosa Anke, S. Schockaert, Capturing Word Order in Averaging Based Sentence Embeddings, ECAI 2020. [Code]
- S. Kong, J. Bai, J.H. Lee, D. Chen, A. Allyn, M. Stuart, M. Pinsky, K. Mills, C. Gomes, Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation, IJCAI 2020.
Spatio-Temporal Reasoning:
- F. Dylla, J.H. Lee, T. Mossakowski, T. Schneider, A.V. Delden, J.V.D. Ven, D. Wolter, A Survey of Qualitative Spatial and Temporal Calculi: Algebraic and Computational Properties, ACM Computing Surveys, 2017.
- J.H. Lee, S. Li, Z. Long, M. Sioutis, On Redundancy in Simple Temporal Networks, ECAI 2016.
- D. Wolter, J.H. Lee, Connecting Qualitative Spatial and Temporal Representations by Propositional Closure, IJCAI 2016.
- X. Ge, J.H. Lee, J. Renz, P. Zhang, Trend-Based Prediction of Spatial Change, IJCAI 2016.
- S. Schockaert, J.H. Lee, Qualitative Reasoning About Directions in Semantic Spaces, IJCAI 2015.
- P. Zhang, J.H. Lee, J. Renz, From Raw Sensor Data to Detailed Spatial Knowledge, IJCAI 2015.
- J.H. Lee, The Complexity of Reasoning with Relative Directions, ECAI 2014.
- J.H. Lee, J. Renz, D. Wolter, StarVars: Effective Reasoning About Relative Directions, IJCAI 2013.
- D. Wolter, J.H. Lee, Qualitative reasoning with directional relations, Artificial Intelligence, 2010.
Multiagent Systems:
- S. Kong, J.H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
- S. Kong, J.H. Lee, S. Li, A new distributed algorithm for efficient generalized arc-consistency propagation, Autonomous Agent Multi-Agent Systems 32 (2018).
- S. Kong, J.H. Lee, S. Li, A Deterministic Distributed Algorithm for Reasoning with Connected Row-Convex Constraints, AAMAS 2017.