Mixtures of SubExperts for Large Language Continual Learning
This research, "Mixtures of SubExperts for Large Language Continual Learning," introduces a novel modular and sparse framework for enabling Large Language Models (LLMs) to learn continuously without compromising previously acquired knowledge or incurring excessive parameter growth. The core contribution is the development of Mixtures of SubExperts (MoSEs), which addresses the fundamental stability-plasticity dilemma in lifelong learning for LLMs. Existing methods either suffer from catastrophic forgetting due to shared parameters or scale linearly with task isolation, hindering knowledge transfer. MoSEs tackle this by factorizing model capacity into reusable, compositional primitives.
Authored by Haeyong Kang, Hee Suk Yoon, Dahua Feng, and Chang D. Yoo, this work was submitted to arXiv in the Computer Science AI category. It is intended for software engineers and researchers working on LLM development and deployment, particularly those concerned with lifelong learning, model efficiency, and robustness against forgetting.
Key technical ideas driving MoSEs include: (i) the augmentation of transformer layers with lightweight "SubExperts," which are specialized modules; (ii) a learned sub-routing function that dynamically selects and composes a sparse subset of these SubExperts based on input tasks; and (iii) a structured decomposition of the parameter space, ensuring knowledge localization while maintaining accessibility and promoting reuse. This approach balances stability by isolating knowledge within sparsely activated modules, plasticity through dynamic routing and recombination, and scalability via sublinear effective capacity growth. The routing mechanism specifically enables compositional generalization, allowing new tasks to be represented as combinations of existing SubExperts.
MoSEs represent a significant advancement by establishing a new Pareto frontier in continual learning for LLMs, demonstrating reduced forgetting and improved forward transfer with superior parameter efficiency compared to strong Parameter-Efficient Fine-Tuning (PEFT) baselines on TRACE and SuperNI benchmarks. This work enables the development of foundation models that can indeed learn throughout their lifecycle without hitting a performance saturation point, paving the way for more adaptable and persistent AI systems. The emphasis on modular sparsity and compositional routing offers a promising direction for future LLM architectures designed for evolving data distributions and continuous knowledge acquisition. The provided text is an abstract only.