Post-Training of LLMs with Reinforcement Learning: Latest Updates on Algorithms and Their Rewards

We will provide a обзор (survey-style) tour of modern reinforcement learning algorithms for large language model training. We will briefly recall the standard LLM training pipeline, discuss why RL optimization is needed at the final stage, and how rewards are constructed.

We will then take a closer look at the technical details of the most popular algorithms:
— the basic PPO, used to train the first ChatGPT;
— the simple DPO, which achieves “RL without RL”;
— the new GRPO combined with verifiable rewards, used to train DeepSeek R1.