Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent
The research, authored by Sriram Balasubramanian and an unnamed co-author from the institution implicitly represented by their arXiv submission, presents HARP (Hypothesis-driven Agentic Retrieval and Probing), a novel training-free approach to neural network interpretability. Published on arXiv in the Computer Science AI and Machine Learning categories, HARP's core contribution is demonstrating that extensive interpretability insights can be extracted solely through retrieval from a pre-compiled database of model activations and their associated contextual metadata, augmented by a suite of activation manipulation tools. This work directly addresses the escalating cost and complexity of current state-of-the-art interpretability methods, such as Sparse Autoencoders (SAEs) and activation oracles, which necessitate computationally intensive training phases. By sidestepping training altogether, HARP offers a significantly more efficient and flexible paradigm for understanding model behavior. The intended audience is primarily software engineers and researchers engaged in model analysis, debugging, and assurance, who stand to benefit from more accessible and potent interpretability solutions.
Key to HARP's efficacy are three critical technical ideas. First, it leverages a vector database populated with neural network activations, critically paired with their textual contexts. This rich pairing enables the agent to ground activation patterns in concrete semantic information. Second, the agent is equipped with a set of manipulation tools, including projection, difference, and averaging operations on activation vectors. These tools allow for the systematic exploration and decomposition of latent space. Third, HARP employs an agentic loop where retrieved samples inform hypothesis generation, which are then validated by constructing linear probes. This iterative, hypothesis-driven process efficiently guides the search for meaningful interpretations.
HARP's results are remarkable: it surpasses training-based methods like activation oracles and SAE-based agents in concept discovery, detection, model steering, and even secret elicitation, all without any training. This suggests that current training-based interpretability techniques may not be extracting insights beyond what is inherently present in their training datasets. HARP's training-free design makes it substantially cheaper and adaptable, allowing new datasets to be indexed on demand. This work has the potential to shift the landscape of interpretability research by motivating the development of benchmarks that explicitly demand insights that go beyond existing training data. It also paves the way for more accessible and scalable model debugging and understanding, particularly as models grow in complexity. The provided content is an abstract only.