Co-Evolving LLM Evaluators and Policies via DynamicRubric
This work, "Co-Evolving LLM Evaluators and Policies via DynamicRubric," by Beining Wang and collaborators from various institutions, published on arXiv, introduces a novel framework for improving large language models (LLMs) through feedback-driven post-training. The core contribution is a method that addresses a critical bottleneck in current evaluator-guided LLM training: the diminishing signal strength of evaluator feedback as the LLM's policy improves. When candidate responses generated by the LLM become too similar in quality, the relative score gaps, which are crucial for guiding policy updates, shrink to the point of being uninformative. This paper theoretically models these score gaps as the directional gradient for policy optimization, quantifying how shifting probability mass between responses directly corresponds to the score difference.
The primary problem solved is the "collapsed relative score gaps" that impede effective LLM fine-tuning. Existing approaches struggle when the LLM's output quality becomes highly uniform, leading to suboptimal policy updates. This research fills the gap by proposing an adaptive evaluation mechanism that dynamically adjusts its criteria. The intended audience includes researchers and engineers working on LLM training, alignment, and reinforcement learning from human feedback. Those benefiting will be developers seeking to improve LLM performance, particularly in tasks requiring nuanced judgment and verifiable reasoning.
Two key technical ideas stand out. First, the theoretical characterization of score gaps as probability allocation signals provides a foundational understanding of why feedback quality matters. Second, the proposed DynamicRubric framework is a response-set-conditioned co-evolutionary system. It generates weighted binary rubric items tailored to each specific set of candidate responses, aggregating these judgments into response-level scores. This dynamic adaptation ensures that the evaluator remains sensitive to subtle quality differences, even as the policy matures. The experiments demonstrate that DynamicRubric, even with smaller models, surpasses baselines using much larger static reward models or rubric generators, and improves performance on reasoning and coding tasks. A notable result is the full deployment of a DynamicRubric-optimized model in WeChat Search, handling millions of daily requests and improving key metrics, underscoring its practical efficacy.
Looking forward, this work enables more robust and efficient LLM alignment by maintaining a strong supervision signal throughout the training process. It suggests a paradigm shift towards dynamic, evolving evaluators that are intrinsically linked to the policies they supervise, rather than static, pre-defined criteria. This could significantly influence the development of more capable and trustworthy LLMs, particularly in complex domains where precise and nuanced evaluation is paramount. The abstract indicates this is the submitted paper content.