Debate Training Reduces Reward Hacking in RLAIF

Rewarding LLMs is a delicate matter. If you train an LLM to produce correct answers by any means necessary, it turns out it can learn to steal them from Hugging Face. And if you train it to gain approval from another model serving as a judge, it will sooner or later learn to deceive that judge—and the weaker the judge relative to the trained model, the faster this happens, which is precisely the regime most relevant for overseeing systems of growing capability.

We will explain how the AI Safety via Debate technique works (Irving, Christiano, Amodei) and show that it does enable a weak judge, Gemini 2.5 Flash Lite, to reliably reward a stronger model, Gemini 2.5 Flash, without being deceived in the process.

Standard RLAIF quickly overfits to the judge’s mistakes: the reward goes up, while both the judge’s and the model’s accuracy decline. Debate maintains the quality of rewards throughout training and recovers 45% of the performance gap between the pre-training model and one trained with verifiable rewards (an oracle).

We will also discuss the approach’s pitfalls: without constraints on the players, adversarial training degenerates into the critic hacking the judge, and in general, instead of producing good arguments, opponents often resort to shouting.

After that, in an informal setting, we will discuss the broader field and recent news in AI Safety and Alignment. I would be happy to share my own experience and answer your questions.