Dr. Jae Hee Lee

About Me

I am a postdoctoral research associate in the Knowledge Technology Group, University of Hamburg. I develop representation learning models for grounded language understanding, with an emphasis on robustness and explainability. My prior work includes symbolic approaches to AI, specifically spatio-temporal reasoning and multiagent systems.

Research Interests

• Multimodal Deep Learning
• Explainable Artificial Intelligence
• Neuro-Symbolic Artificial Intelligence
• Spatio-Temporal Reasoning

Research Experience

• Neurocognitive Models of Crossmodal Language Learning, University of Hamburg, Hamburg, Germany (2020–present)
• Formal Lexically Informed Logics for Searching the Web, Cardiff University, Cardiff, UK (2018–2020)
• Feodor Lynen Research Fellowship by Alexander von Humboldt Foundation, University of Technology Sydney, Sydney, Australia (2016–2017)
• Artificial Intelligence Meets Wireless Sensor Networks, The Australian National University, Canberra, Australia (2015)
• Reasoning about Paths, Shapes, and Configuration, University of Bremen, Bremen, Germany (2009–2014)

Education

• Dr. rer. nat. in Computer Science, University of Bremen, Germany (2009–2013)
• Visiting PhD student, North Carolina State University, USA (2012)
• Visiting PhD student, University at Buffalo, USA (2011)
• Diplom in Mathematics, University of Bremen, Germany (2003–2009)
• DAAD exchange student, Seoul National University, South Korea (2008)

Research Grants and Awards

• Bremen-Cardiff Alliance Collaborative Fund, Cardiff University, UK (2019)
• UTS Blue Sky Research Scheme, University of Technology Sydney, Sydney (2017)
• Feodor Lynen Research Fellowship, Alexander von Humboldt Foundation, Germany (2016–2017)
• Fachbereichspreis Mathematik (diploma thesis award), University of Bremen (2009)
• DAAD-KOSEF Scholarship, DAAD, Germany (2008)
• Third Prize at the Regional Contest Jugend Forscht Bremen (2001)
• Senator-Friedrich-Cordewener-Prize at the Regional Contest Jugend Forscht Bremen (2001)

Recent Teaching Activities

• Lecture on “Transformers and Crossmodal Learning”, University of Hamburg (2023) [Slides]
• Supervisor, Neural Networks Seminar, University of Hamburg (2023)
• Organizer, WISDUM Reading Group, Knowledge Technology, University of Hamburg (2023)
• Supervisor, Bio-inspired Artificial Intelligence Seminar, University of Hamburg (2022)

Thesis Supervision

• Generalization of Transformer-Based Models on Visual Question Answering Tasks, Frederic Voigt, MSc (2023)
• Improving Compositional Generalization By Learning Concepts Individually, Ramtin Nouri, MSc (2023)
• Learning Concepts a Developmental Lifelong Learning Approach to Visual Question Answering, Ramin Farkhondeh, BSc (2022)
• Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks, Jakob Ambsdorf, MSc (2022), now PhD student at University of Copenhagen
• Tackling The Binding Problem And Compositional Generalization In Multimodal Language Learning, Caspar Volquardsen, BSc (2021), ICANN 2022
• Learning Bidirectional Translation Between Robot Actions and Linguistic Descriptions, Markus Heidrich, BSc (2021)
• Using the Reformer for Efficient Summarization, Yannick Wehr, BSc (2020)
• Generalization in Multi-Modal Language Learning from Simulation, Aaron Eisermann, BSc (2020), IJCNN 2021
• Commonsense Validation and Explanation, Christian Rahe, BSc (2020)

Blog Posts

• Deep Learning Courses and Books for Beginners
• Common LaTeX Mistakes by Newbies

News

• MSc student Björn Plüster, whom I advise, developed a German LLM called LeoLM with LAION and hessian.AI!
• Our paper Internally Rewarded Reinforcement Learning got accepted to ICML 2023.
• I am co-organizing the 2nd International Workshop on Spatio-Temporal Reasoning and Learning.
• Program committee member, EMNLP 2023.
• Program committee member, ECAI 2023.
• Program committee member, ACL 2023.
• Program committee member, IJCAI 2023.
• Received ~10K € research funding for a student project on “explainable visual question answering” that I am advising. Check out our paper!

Selected Papers

Multimodal Deep Learning:

• J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022. [Code]
• O. Özdemir, M. Kerzel, C. Weber, J.H. Lee, M.B. Hafez, P. Bruns, S. Wermter, Learning Bidirectional Action-Language Translation with Limited Supervision and Testing with Incongruent Input, Applied Artificial Intelligence, 2023.
• M. Li, C. Weber, M. Kerzel, J.H. Lee, Z. Zeng, Z. Liu, S. Wermter, Robotic Occlusion Reasoning for Efficient Object Existence Prediction IROS 2021. [Code]
• A. Eisermann, J.H. Lee, C. Weber, S. Wermter, Generalization in Multimodal Language Learning from Simulation, IJCNN 2021.

Explainable Artificial Intelligence:

• J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, (under review).

• B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022. [Code]

Neuro-Symbolic Reasoning:

• J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429 [arXiv Preprint]

• J. H. Lee et al. Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022. [Code]

Other Machine Learning Topics:

• M. Li, X. Zhao, J.H. Lee, C. Weber, S. Wermter, Internally Rewarded Reinforcement Learning, ICML 2023 [Project page] [Code].
• J.H. Lee, J. Camacho-Collados, L. Espinosa Anke, S. Schockaert, Capturing Word Order in Averaging Based Sentence Embeddings, ECAI 2020. [Code]
• S. Kong, J. Bai, J.H. Lee, D. Chen, A. Allyn, M. Stuart, M. Pinsky, K. Mills, C. Gomes, Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation, IJCAI 2020.

Spatio-Temporal Reasoning:

• F. Dylla, J.H. Lee, T. Mossakowski, T. Schneider, A.V. Delden, J.V.D. Ven, D. Wolter, A Survey of Qualitative Spatial and Temporal Calculi: Algebraic and Computational Properties, ACM Computing Surveys, 2017.
• J.H. Lee, S. Li, Z. Long, M. Sioutis, On Redundancy in Simple Temporal Networks, ECAI 2016.
• D. Wolter, J.H. Lee, Connecting Qualitative Spatial and Temporal Representations by Propositional Closure, IJCAI 2016.
• X. Ge, J.H. Lee, J. Renz, P. Zhang, Trend-Based Prediction of Spatial Change, IJCAI 2016.
• S. Schockaert, J.H. Lee, Qualitative Reasoning About Directions in Semantic Spaces, IJCAI 2015.
• P. Zhang, J.H. Lee, J. Renz, From Raw Sensor Data to Detailed Spatial Knowledge, IJCAI 2015.
• J.H. Lee, The Complexity of Reasoning with Relative Directions, ECAI 2014.
• J.H. Lee, J. Renz, D. Wolter, StarVars: Effective Reasoning About Relative Directions, IJCAI 2013.
• D. Wolter, J.H. Lee, Qualitative reasoning with directional relations, Artificial Intelligence, 2010.

Multiagent Systems:

• S. Kong, J.H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
• S. Kong, J.H. Lee, S. Li, A new distributed algorithm for efficient generalized arc-consistency propagation, Autonomous Agent Multi-Agent Systems 32 (2018).
• S. Kong, J.H. Lee, S. Li, A Deterministic Distributed Algorithm for Reasoning with Connected Row-Convex Constraints, AAMAS 2017.