VeriKAS – Verification of learning AI applications in the aviation sector
Pl: Prof. Dr. Stefan Wermter
Associates: Wenhao Lu, Dr. Sven Magg, Dr. Matthias Kerzel
With the tremendous success of data-driven approaches in pattern recognition, natural language processing and decision-making for robotic tasks, the debate on the poor interpretability of the “black-box” model becomes more intense. The problem of non-interpretability impacts their applicability, especially in critical systems, and has therefore become a major research focus. Taking image classification on ImageNet as an example, although a simple ResNet-based classifier can readily achieve a very high accuracy that was unattainable just a few years ago, the complexity within its highly tangled network modules makes it impossible to prove the robustness and performance of its classification given unseen data. Hence, a huge demand arises for tools to structurally inspect the learned “knowledge” of a neural network. Emergent post-hoc visualization techniques, like CAM, LIME and Grad-CAM attempt to significantly improve explainability, by visually making predictions from Convolutional Neural Networks (CNNs) more transparent, though the full CNN-based model itself is usually uninterpretable. The challenge of the VeriKAS project is that the computations of the model can be better interpreted by humans in an appropriate manner.
Deep reinforcement learning (DRL) models also share the same problem and tend to be opaque in their decision-making process. As a result, utilizing them in safety-critical areas like automatic emergency-landing for drones is destined to be challenging. In this project, we propose to, on the one hand, investigate how robust explainability or interpretable DRL models can be realized in a context in which a vision-based DRL agent supports emergency landing of aerial vehicles; on the other hand, to demonstrate the applicability of developed explanation models (visualization tools) to the task of detecting critical objects in an assembly line scenario. One of the expected outcomes is the development of an interpretable DRL model with robust and explainable behaviour while maintaining its performance. Consequently, this can support human confidence in deploying an AI model for high-stake decisions. To this end, embedding reasoning strategies like relational or causal reasoning into DRL is worthy of investigation. Other promising methods include hierarchical DRL with multi-tasking goals and exploration of the reward structure. Aside from this, we aim to co-develop processes that enable future certification of deep learning-based systems. This primarily involves robust visual explanations techniques (e.g. against lighting variance or distractors for images as input), as well as a quality assurance process, for recording training data and launching data-driven methods to downstream tasks, that can improve transparency and ensure traceability.