RLHF in Machine Learning

The Limitation of Supervised Fine-Tuning

Instruction tuning (SFT) teaches a language model to follow instructions by showing it examples of good responses. But SFT has a fundamental limitation: it can only teach the model to imitate the demonstrations it was trained on. It cannot teach the model to distinguish between a mediocre response and an excellent one, or to prefer safety over helpfulness when they conflict.

Consider a question like "How do I pick a lock?" SFT can show the model one correct refusal. But what about the thousand subtle variations of this question? What about cases where the model should provide partial information (e.g., for a locksmith) but refuse in other contexts? SFT treats every response as equally correct -- there's no gradient signal for relative quality.

The Preference Signal

RLHF solves this by introducing a preference signal. Instead of telling the model "this is the right answer," RLHF tells the model "this answer is better than that one." Human annotators compare two model responses to the same prompt and indicate which they prefer. These pairwise comparisons are far easier for humans to provide than writing perfect demonstrations from scratch -- and they capture nuanced quality distinctions that binary correct/incorrect labels miss.

The insight that pairwise comparisons are easier and more reliable than absolute quality ratings comes from the psychology literature on comparative judgment (Thurstone, 1927). The Bradley-Terry model (1952) formalized this into a mathematical framework for converting pairwise preferences into scalar scores -- the same framework that underpins modern reward model training.

The Historical Arc

The intellectual roots of RLHF stretch back to the early 2010s, when DeepMind and OpenAI researchers explored using human feedback to train RL agents for Atari games and robotic tasks. The seminal paper by Christiano et al. (2017) -- "Deep Reinforcement Learning from Human Preferences" -- demonstrated that humans could train RL agents by comparing short video clips of agent behavior, without ever specifying a reward function.

The leap to language models came with Ziegler et al. (2019), who applied RLHF to fine-tune GPT-2 for text summarization. But the technique truly arrived with InstructGPT (Ouyang et al., 2022), which applied RLHF at scale to GPT-3 and showed that the resulting 1.3B model was preferred over the 175B base model. The three-stage pipeline -- SFT, reward model, PPO -- became the standard recipe for LLM alignment.

Anthropic's parallel work on "Training a Helpful and Harmless Assistant with RLHF" (Bai et al., 2022) extended this to multi-objective alignment, training separate reward models for helpfulness and harmlessness. Meta's Llama-2 (Touvron et al., 2023) made the technique accessible to the open-source community with detailed documentation of their RLHF pipeline.

Key Takeaway: RLHF exists because SFT alone cannot capture the nuanced quality spectrum of model outputs. Pairwise human preferences provide a richer, more reliable signal than demonstrations alone, and RL provides the optimization machinery to act on that signal.

The Chef Analogy

Imagine you're training a chef. Supervised fine-tuning is like giving the chef a cookbook of perfect recipes -- they learn to replicate those dishes faithfully. But what happens when a customer orders something not in the cookbook? The chef improvises, sometimes brilliantly, sometimes terribly.

RLHF is like hiring a food critic. The critic tastes pairs of dishes and says "this one is better than that one." Over time, the chef learns not just to follow recipes, but to understand what makes food good -- balance of flavors, presentation, freshness. The critic's preferences become internalized as taste.

The food critic is the reward model. The process of the chef adjusting their cooking based on the critic's scores is PPO optimization. And the rule that says "don't stray too far from the original recipes" is the KL divergence penalty -- without it, the chef might start doing bizarre things that technically score well with the critic but are inedible in practice (this is reward hacking).

Why Pairwise Comparisons Work

Here's a subtle but crucial insight: humans are much better at comparing two things than scoring one thing in isolation. If I show you two summaries of an article, you can quickly tell which is better. But if I ask you to rate a single summary on a 1-10 scale, your rating will be noisy, inconsistent, and poorly calibrated.

RLHF exploits this psychological fact. By collecting pairwise preferences ("A is better than B") rather than absolute ratings, the human feedback is more reliable, more consistent, and requires less annotator expertise. The Bradley-Terry model then converts these relative comparisons into absolute scores that a reward model can learn to predict.

The Three-Stage Pipeline Intuition

Think of the three stages as progressive refinement:

1. SFT teaches the model the language of being helpful (format, tone, structure)
2. Reward Model learns the taste function -- what distinguishes a great response from a good one
3. PPO uses that taste function to polish the model's outputs, pushing them from good to great

Each stage builds on the previous one. You can't do RLHF without SFT first (the model needs a reasonable starting point for RL to improve upon). You can't do PPO without a reward model (you need a differentiable proxy for human judgment). And you can't train a reward model without human preferences (the whole point is to capture what humans want).

Expert Insight: RLHF doesn't teach the model new knowledge or capabilities. Like instruction tuning, it changes how the model expresses its existing capabilities. The difference is that SFT optimizes for imitation ("be like the demonstrations") while RLHF optimizes for preference ("be what humans prefer"). This distinction matters enormously in practice.

Mathematical Framework

Let $\pi_{\text{SFT}}$ denote the policy (language model) after supervised fine-tuning, and let $\pi_\theta$ denote the policy being optimized via RLHF with parameters $\theta$.

Stage 1: Reward Model Training (Bradley-Terry Model)

Given a dataset of human preferences $\mathcal{D} = \{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}_{i=1}^N$ where $x$ is a prompt, $y_w$ is the preferred (chosen) response, and $y_l$ is the rejected response, the reward model $r_\phi(x, y)$ is trained to assign higher scores to preferred responses.

The Bradley-Terry model defines the probability that response $y_w$ is preferred over $y_l$ as:

$P(y_w \succ y_l \mid x) = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$

where $\sigma$ is the sigmoid function. The reward model is trained by minimizing the negative log-likelihood:

$\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) \right]$

This is equivalent to binary cross-entropy where the label is always 1 (the chosen response should score higher). The reward model typically shares the architecture of the base LLM with a scalar value head replacing the language modeling head.

Stage 2: PPO Optimization with KL Penalty

The policy $\pi_\theta$ is optimized to maximize the expected reward while staying close to the reference policy $\pi_{\text{ref}}$ (usually $\pi_{\text{SFT}}$):

$\max_{\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} \left[ r_\phi(x, y) - \beta \cdot D_{\text{KL}}(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x)) \right]$

where $\beta > 0$ controls the strength of the KL divergence penalty. This objective balances two goals:

• Maximize reward: produce outputs the reward model scores highly
• Minimize KL divergence: don't deviate too far from the SFT model

The KL term prevents reward hacking -- exploiting weaknesses in the imperfect reward model to achieve high scores without actually improving quality.

PPO Clipping Objective

PPO (Schulman et al., 2017) optimizes this objective using a clipped surrogate loss. Let $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)}$ be the probability ratio between the current and old policy. The PPO objective is:

$\mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_t \left[ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t) \right]$

where $\hat{A}_t$ is the advantage estimate (computed via a learned value function) and $\epsilon$ (typically 0.2) controls the clipping range. The clipping prevents catastrophically large policy updates -- a critical stability mechanism for language model training.

Scaling Laws for Reward Model Overoptimization

Gao et al. (2022) established that the gold reward (true human preference score) follows a characteristic pattern as optimization proceeds against a proxy reward model:

$R_{\text{gold}}(d) = \alpha \sqrt{d} - \beta d$

where $d = D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ is the KL divergence from the reference policy. The first term represents genuine improvement; the second represents overoptimization. The gold reward peaks at $d^* = \frac{\alpha^2}{4\beta^2}$ and then declines -- this is the quantitative signature of reward hacking.

Formal Property: The RLHF objective can be shown to be equivalent to finding the optimal policy $\pi^*$ in the KL-constrained policy space: $\pi^*(y|x) \propto \pi_{\text{ref}}(y|x) \cdot \exp\left(\frac{1}{\beta} r_\phi(x, y)\right)$. This closed-form solution is the theoretical foundation for DPO, which bypasses the reward model entirely by optimizing this expression directly.

Reinforcement Learning from Human Feedback (RLHF) is the three-stage alignment technique that transformed raw language models into the AI assistants we use today. The pipeline -- SFT, Reward Model, PPO -- was established by OpenAI's InstructGPT and has been adopted by every frontier model builder. The core mechanism is elegantly simple: train a reward model on human preference comparisons using the Bradley-Terry framework, then use PPO to optimize the language model against that reward signal, constrained by a KL divergence penalty that prevents reward hacking.

The practical reality of RLHF is far more complex than the theory. The pipeline requires maintaining four model copies simultaneously (policy, reference, reward, value), demands careful hyperparameter tuning (KL coefficient, clipping range, learning rate scheduling), and is vulnerable to reward hacking, reward model collapse, and capability degradation. The human annotation cost ($50K-$14M for preference data at scale) often exceeds the compute cost, which is why alternatives like DPO and Constitutional AI have gained traction. Gao et al.'s scaling laws for overoptimization provide a principled framework for understanding when to stop training: gold reward peaks at a specific KL divergence and declines thereafter.

Despite its complexity, RLHF remains the gold standard for frontier model alignment. Online PPO-based RLHF consistently outperforms offline alternatives on hard alignment tasks because on-policy exploration provides more informative training signal. For teams building production LLMs -- whether at OpenAI scale or at Indian startups like Krutrim and Sarvam AI -- the choice between full RLHF, DPO, and Constitutional AI depends on the budget, infrastructure, and alignment requirements. Understanding RLHF deeply, including its failure modes and alternatives, is essential for any ML engineer working on LLM alignment in 2026.