This work, titled "Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models" by Wenhui Chen, Jianlin Chen, Ziyao Lin, and Chi Man Vong, published on arXiv, addresses a fundamental question in deep learning: how do model capabilities emerge, and is it solely a function of scale? The paper challenges the "Platonic Representation Hypothesis" (PRH), which posits that larger models develop more unified internal representations of reality. Instead, it introduces the "Capability Convergence Hypothesis" (CCH), arguing that under fixed computational budgets per token, representational convergence does not guarantee capability convergence. Capability, the authors posit, converges towards a specific class of architectures: "access-complete hybrids." These are models that possess both a highly compressed, constant-state channel and a scalable, verbatim-indexed channel.
The core problem this research tackles is the disconnect between the observed scaling laws in models, which show improvements with increased parameters, and the underlying mechanisms driving specific, complex reasoning capabilities. The authors identify a gap by suggesting that current scaling paradigms might be inefficient for acquiring certain sophisticated functionalities. They propose that architectural design, specifically how information is accessed and processed, is paramount for achieving true capability gains, not just brute-force scaling.
The intended audience for this paper includes software engineers and researchers working on sequence modeling, artificial intelligence, and theoretical computer science. Those benefiting include developers seeking to build more efficient and capable models, and researchers aiming to understand the fundamental limits and drivers of AI intelligence.
Three key technical ideas underpin this work. Firstly, the "access-complete hybrid" architecture, defined by its dual channels: an $O(1)$-state compressive channel for general processing and a scalable verbatim-index channel for precise information retrieval. Secondly, the identification of three "resource walls" – Shannon, horizon, and circuit walls – that limit capability without the appropriate access structure. These walls represent theoretical barriers related to state compression, input window size, and computational depth respectively. Finally, the use of information-theoretic lower bounds and pre-registered tests to rigorously evaluate their hypotheses, distinguishing between proven results and conjectures. The pre-registered tests on a "Newton's-apple" task and state-tracking problems provide empirical validation for the necessity of specific access structures.
This research opens avenues for designing more efficient and powerful sequence models by focusing on architectural primitives rather than solely on parameter count. It suggests that future advancements in AI might come from a deeper understanding and manipulation of information access structures within neural networks. This could lead to models that achieve superior performance on complex tasks with significantly reduced computational resources, influencing the direction of architectural research and development in the field. The paper's abstract format limits a full discussion of its experimental methodologies and detailed proofs.