A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
A newly published research paper on arXiv cs.AI (June 2026) titled "A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions" introduces a unified mathematical framework for understanding reinforcement learning policy gradients in large language models. Authored by Jianghan Shen, Siqi Luo, Yue Li, and a team of co-researchers, the work deconstructs the standard expected reward objective into its two foundational mathematical components: the trajectory probability and the reward function. By mapping existing algorithms along these two distinct axes—the trajectory side and the reward side—the authors establish a systematic, diagnostic taxonomy of LLM policy optimization.
This framework addresses a critical gap in machine learning literature. Existing surveys typically organize LLM reinforcement learning algorithms chronologically or by application domain, which obscures the underlying mathematical rationale behind algorithmic modifications. For machine learning researchers and reinforcement learning engineers, this paper demystifies the precise mechanics of algorithms ranging from classic REINFORCE and PPO to Group Relative Policy Optimization (GRPO), Agentic RL, and GRPO-OPD. It reveals that every historical development is a targeted mathematical intervention on either the trajectory distribution or the reward formulation to overcome a specific failure mode in previous gradient estimators.
The core technical contribution of this framework lies in tracing all policy optimization variants back to a single shared objective. By placing algorithms along the trajectory and reward axes, the authors demonstrate how modern techniques alter either the policy constraints (the trajectory side) or the credit assignment and reward scaling (the reward side). Crucially, this analytical mapping exposes "compound failures"—complex algorithmic bottlenecks where isolated, single-sided modifications are mathematically insufficient. The authors argue that resolving these boundary cases requires a coupled design that simultaneously restructures both the trajectory generation and the reward signal.
Going forward, this structural mapping provides a principled, extensible starting point for designing the next generation of LLM alignment algorithms. Rather than relying on empirical trial-and-error, researchers can use this coordinate system to diagnose failures in reinforcement learning training pipelines and systematically derive new optimization objectives. This first-principles approach is poised to accelerate the development of more stable and sample-efficient alignment methods, particularly for highly complex environments such as multi-step agentic workflows. Note that this analysis is based on the published abstract of the paper.