Reward Hacking in Rubric-Based Reinforcement Learning
Published in arXiv preprint, 2026
Recommended citation: Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He. "Reward Hacking in Rubric-Based Reinforcement Learning." arXiv preprint arXiv:2605.12474 (2026). https://arxiv.org/abs/2605.12474
Rubric-based rewards are useful for reinforcement learning in open-ended settings, but they can introduce new reward-hacking risks. We study how policy optimization exploits rubric-based verifiers, distinguishing failures caused by weak verification from limitations in rubric design itself. Across medical and science tasks, we find that stronger verifiers reduce but do not eliminate exploitation, and that incomplete rubrics can still drive gains on rubric criteria while degrading broader response quality. We also introduce a verifier-free diagnostic based on policy log-probabilities for tracking when training quality stops improving.