15 July 2026

Photo: UHH Knowledge Technology

Our group has two papers presented at the ICML'26 conference in Seoul, South Korea. Here is more information about the papers:

Paper 1

Title: The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

Authors: Jeremy Herbst, Stefan Wermter, Jae Hee Lee

Abstract: Mixture-of-Experts (MoE) architectures have become the dominant choice for scaling Large Language Models (LLMs), activating only a subset of parameters per token. While MoE architectures are primarily adopted for computational efficiency, it remains an open question whether their sparsity makes them inherently easier to interpret than dense feed-forward networks (FFNs). We compare MoE experts and dense FFNs using k-sparse probing and find that expert neurons are consistently less polysemantic, with the gap widening as routing becomes sparser. This suggests that sparsity pressures both individual neurons and entire experts toward monosemanticity. Leveraging this finding, we zoom out from the neuron to the expert level as a more effective unit of analysis. We validate this approach by automatically interpreting hundreds of experts. This analysis allows us to resolve the debate on specialization: experts are neither broad domain specialists (e.g., biology) nor simple token-level processors. Instead, they function as fine-grained task experts, specializing in linguistic operations or semantic tasks (e.g., closing brackets in LATEX). Our findings suggest that MoEs are inherently interpretable at the expert level, providing a clearer path toward large-scale model interpretability. Code is available at: this https URL. 

Paper 2

Title: Adaptive Recurrence as Algorithmic Time for Length Generalization in Addition

Authors: Imran Ibrahimli, Stefan Wermter, Jae Hee Lee

Abstract: Length generalization asks whether a model can reuse a learned computation beyond the lengths seen in training. Arithmetic makes this test precise because a model trained on short additions should apply the same digit-wise rule for more steps, rather than fit patterns tied to the training range. Many strong extrapolation results, however, rely on explicit aids that tell the model how digits should be aligned. We ask whether a model can length-generalize without such aids. We train small looped transformers with adaptive halting on a controlled decimal-addition task, without positional encodings or arithmetic-specific side information. The models extrapolate beyond the training range across multiple training regimes. To understand why, we analyze their internal dynamics and find that recurrent depth becomes organized as algorithmic time: later answer positions are resolved at later recurrent phases. These results suggest that adaptive recurrence can turn repeated latent computation into a less hand-engineered route for applying a digit-wise rule beyond the training length.