SFT vs. PPO vs. DPO: A Guide to LLM Alignment

Large Language Models (LLMs) are trained on enormous amounts of data from the internet during a “pre-training” phase designed to give them general knowledge. They aren’t inherently helpful at this point. They’re essentially “fancy parrots.” Given a sequence of words (a prompt), they are designed to spit out words that, based on their training dataset (the internet), are most likely to appear after your prompt. That means it might generate more questions based on your question in the prompt, mention a tangential fact, etc. This isn’t super helpful for us, but it happens because it’s modeling the distribution of internet text rather than “being a helpful assistant.”

We want to train LLMs to be helpful, friendly, non-harmful, etc. The way we solve this is through Reinforcement Learning from Human Feedback (RLHF). It’s a method for aligning LLMs trained on the internet to be helpful and safe.

Supervised Fine-Tuning

Supervised Fine-Tuning (SFT) is usually the first step in training an LLM to become more helpful. This process involves collecting many quality answers to a diverse range of questions a user might ask. SFT is very similar to the training process in pre-training. The main difference is that the goal is to shape behavior rather than give LLMs general knowledge. Thus, during SFT you train the LLM to predict the response on high-quality instruction -> response pairs rather than the general internet. To learn more about the pre-training process, check out this deep dive where I explain how to train an LLM from scratch!

Proximal Policy Optimization

After SFT, you collect many instruction/response pairs. Each preference example is a triple $(x, y_w, y_l)$ where $x$ is the instruction, $y_l$ is the worse response, and $y_w$ is the “better” response. You train a reward model, $r: \text{instruction, response} \to \mathbb{R}$ , to maximize the Bradley-Terry objective. In other words, the reward model gives a score for each instruction/response (higher means better response). The Bradley-Terry objective then takes an instruction/response pair and the score for each response and computes the probability that $y_w$ is better than $y_l$ : $\sigma(r(x, y_w) - r(x, y_l))$ .

Given the reward model, we can now apply proximal policy optimization. If you are unfamiliar with reinforcement learning, review some of my notes in the Appendix that go over it. Essentially, reinforcement learning provides us tools to learn a policy $\pi$ that maximizes the sum of expected rewards (also known as return) in an environment, and proximal policy optimization is a specific, sample-efficient method of learning such a policy.

To understand how proximal policy optimization can help us, we need to convert the problem of training an LLM into an RL problem. In this case, $\pi$ is the LLM and it is parameterized by the weights $\theta$ . The action space is the set of tokens the LLM can output. Once the response is generated, we can plug it into the reward model to get a final reward (note: in this setup the rewards are sparse, you only get a reward at the end and not after each token is produced).

Given this setup, proximal policy optimization updates the policy (LLM) to choose actions (tokens) that maximize the reward (producing a better, more helpful response). But maximizing reward on its own is dangerous. RL is prone to reward hacking, where the policy drifts into strange outputs that score well under the reward model without actually being good responses. To prevent this, RLHF subtracts a KL-divergence term between the new policy and the SFT policy from the objective:

\pi^* = \underset{\pi}{\max} \; \mathbb{E}[r(x,y)] - \beta KL(\pi \parallel \pi_\text{ref})

Here $\beta > 0$ controls how much we penalize straying from the SFT policy. A higher $\beta$ keeps the new policy closer to the SFT policy we trust.

Direct Preference Optimization

While PPO is effective at teaching LLMs to be more helpful, it also brings in all of the overhead of RL: agent, trajectories, reward model, value functions, etc. This added complexity is often undesirable which is where Direct Preference Optimization (DPO) comes in. It allows you to take a set of preference pairs (instruction/response pair where response $y_w$ is better than response $y_l$ ), and train directly using those pairs with a cross entropy loss.

The important insight that DPO makes is that maximizing rewards while not diverging too far from the reference (SFT) policy has a closed form solution.

First, remember what PPO was trying to maximize:

\pi^* = \underset{\pi}{\max} \mathbb{E}[r(x,y)] - \beta KL(\pi \parallel \pi_\text{ref})

This optimization has a known closed-form solution:

\pi^*(y|x) = \frac{1}{Z(x)} \pi_\text{ref}(y|x)e^{\frac{r(x,y)}{\beta}}

where $Z(x) = \sum_y \pi_\text{ref}(y|x)e^{\frac{r(x,y)}{\beta}}$ is a normalizing constant that makes sure $\pi^*(y|x)$ is a valid probability distribution. Overall, this result shows that you can just re-weight the original SFT policy by the exponentiated reward function to get the optimal policy.

Given this closed form, we can reparameterize to solve for the reward in terms of the policy:

r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} + \beta \log Z(x)

Now we can plug this into our Bradley-Terry objective:

P(y_w \succ y_l) = \sigma(r(x, y_w) - r(x,y_l)) = \sigma \left( \beta \log \frac{\pi(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta \log \frac{\pi(y_l|x)}{\pi_\text{ref}(y_l|x)} \right)

And we can convert the overall goal of maximizing the Bradley-Terry objective into minimizing a supervised loss:

\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta \log \frac{\pi(y_l|x)}{\pi_\text{ref}(y_l|x)} \right)\right]

Now we can just treat this as a supervised fine-tuning problem over preference pairs rather than bringing all the complexities of RL in (and training a reward model). So the natural question is why not use DPO for everything? There are a few reasons:

PPO can explore by generating new responses and seeing how the reward model grades them. DPO is fixed to learning from the preference pairs.
The reward model is reusable and gives a richer signal. The reward model can score any generated response, and can be reused/iterated on.
Perhaps one of the biggest reasons is that well-tuned PPO can still outperform DPO empirically.

Appendix

Reinforcement Learning

The main players in Reinforcement Learning (RL) are an agent and the environment. The environment is completely described at timestep $t$ by a vector $s_t$ . The agent described by policy $\pi$ (often stochastic) performs some action in the environment $a_t$ (discrete or continuous) which takes you to a new state $s_{t+1}$ . The state transition $s_{t+1}$ is determined by the current state and action and can be probabilistic $s_{t+1} \sim P(\cdot | s_t,a_t)$ . You then collect a reward $r_t$ based on the transition $r_t = R(s_t, a_t, s_{t+1})$ . The trajectory $\tau$ is the entire sequence of states and actions that occurred in the world: $\tau = (s_0, a_0, s_1, a_1, ...)$ . An agent’s objective is to maximize the expected sum of rewards collected over a trajectory $R(\tau)$ .

To make the agent’s goal concrete, let’s observe the following. First, we can represent the probability of a trajectory under policy $\pi$ as:

P(\tau | \pi) = \rho_0(s_0) \prod_{t=0}^{T-1} P(s_{t+1}|s_t,a_t) \pi(a_t|s_t)

where $\rho_0(s_0)$ is the probability of $s_0$ occurring from the list of possible start states.

The expected cumulative rewards under policy $\pi$ is:

J(\pi) = \int_\tau P(\tau | \pi) R(\tau)

RL tries to find a policy that maximizes $J$ :

\pi^* = \underset{\pi}{\operatorname{argmax}} J(\pi)

Value, Action-Value, Advantage Functions

The on-policy value function gives you the expected return at state $s$ under policy $\pi$ :

V^\pi(s) = \int_{\tau \sim \pi | s_0 = s} R(\tau)

The on-policy action-value function gives you the expected return at state $s$ if you take action $a$ and then listen to the policy for all future actions:

Q^\pi(s,a) = \int_{\tau \sim \pi | s_0 = s, a_0 = a} R(\tau)

The advantage function tells you how much better an action is compared to other actions on average:

A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)

Standard Policy Iteration

In traditional machine learning, we have algorithms like gradient descent to iteratively improve the weights of a network to converge on a better network (reduce the loss). Policy iteration is a similar concept. We iterate on a policy to converge to a better policy (increase the expected return). This iterative process is done by taking small steps in the gradient of $J$ with respect to $\pi$ :

\begin{aligned} \nabla_\theta J(\pi_\theta) &= \nabla_\theta \int_\tau P(\tau | \pi_\theta) R(\tau) \\ &= \int_\tau \nabla_\theta P(\tau | \pi_\theta) R(\tau) \\ &= \int_\tau P(\tau | \theta) \nabla_\theta \log (P(\tau | \pi_\theta)) R(\tau) \text{ since } \nabla_\theta \log P(\tau | \theta) = \frac{1}{P(\tau | \theta)} \nabla_\theta P(\tau | \theta) \\ &= \underset{\tau \sim \pi_\theta}{\mathbb{E}} \nabla_\theta \log (P(\tau | \pi_\theta)) R(\tau) \\ &= \underset{\tau \sim \pi_\theta}{\mathbb{E}} \nabla_\theta \log (\rho_0(s_0) \prod_{t=0}^{T-1} P(s_{t+1}|s_t,a_t) \pi_\theta(a_t|s_t)) R(\tau) \\ &= \underset{\tau \sim \pi_\theta}{\mathbb{E}} \nabla_\theta (\log (\rho_0(s_0)) + \sum_{t=0}^{T-1} \log(P(s_{t+1}|s_t,a_t) \pi_\theta(a_t|s_t))) R(\tau) \\ &= \underset{\tau \sim \pi_\theta}{\mathbb{E}} \nabla_\theta \sum_{t=0}^{T-1} \log(\pi_\theta(a_t|s_t)) R(\tau) \text{ many terms have no gradient since no dep on } \theta \\ &= \underset{\tau \sim \pi_\theta}{\mathbb{E}} \sum_{t=0}^{T-1} \nabla_\theta \log(\pi_\theta(a_t|s_t)) R(\tau) \end{aligned}

In practice, we collect many sample trajectories $\mathcal{D} = \{\tau_i\}_{i=1 \ldots N}$ , and estimate $\nabla_\theta J(\pi_\theta)$ using the following:

\nabla_\theta J(\pi_\theta) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t=0}^{T-1} \nabla_\theta \log(\pi_\theta(a_t|s_t)) R(\tau_i)

This weights each action by the full return $R(\tau)$ , but we can instead weight it by the advantage $A^\pi(s_t, a_t)$ without changing the gradient of $J$ , which dramatically lowers its variance (see this derivation for why swapping in a baseline and the reward-to-go leaves the gradient unbiased).

Proximal Policy Optimization

While Standard Policy Iteration provides an iterative way to converge on a policy that maximizes expected return, it is very sample inefficient. We must collect a bunch of trajectories to perform just a single policy update. Proximal Policy Optimization is a method that relaxes some of the constraints given by Standard Policy Iteration to improve sample efficiency.

For the sake of completeness, there are two forms of PPO: PPO-Penalty and PPO-Clip. This article focuses on PPO-Clip since it’s the most common form. PPO-Clip relies on a specialized clipping of the objective function to remove incentives for each step of policy iteration to get too far from the old policy. Intuitively, by limiting how far each step changes the old policy, we can use the old policy’s data for longer.

In PPO-Clip, our objective changes from maximizing expected rewards: $\pi^* = \underset{\pi}{\operatorname{argmax}} J(\pi)$ to maximizing a clipped version at each iteration $k$ :

\pi_{\theta_{k+1}} = \underset{\pi}{\operatorname{argmax}} \underset{s, a \sim \pi_{\theta_k}}{\mathbb{E}} [L(s, a, \pi_{\theta_k}, \pi_\theta)]

where

L(s, a, \pi_{\theta_k}, \pi_\theta) = \min \left( \frac{\pi_\theta (a | s)}{\pi_{\theta_k} (a | s)} A^{\pi_{\theta_k}}(s, a), \operatorname{clip}\left( \frac{\pi_\theta (a | s)}{\pi_{\theta_k} (a | s)}, 1 - \epsilon, 1 + \epsilon \right)A^{\pi_{\theta_k}}(s, a) \right)

This is a complicated expression, but notice that when $\pi_\theta = \pi_{\theta_k}$ , this equals the standard policy iteration (using advantages) that we described above. When the policy changes, the objective changes to maximize the advantage within a boundary. The intuition is that as our policy diverges from the policy that data was collected on, we trust our update less and less. To prevent our policy from drifting too far from the original policy, the $\operatorname{clip}$ part of the objective stops giving more advantages after you change the policy too much (effectively flatlining the gradient and removing the incentive for a big policy change).

The $\min$ part of the objective is used to undo the effect of clipping in exactly two cases:

The probability of a worse action increases significantly ( $A < 0$ , $\frac{\pi_\theta (a | s)}{\pi_{\theta_k} (a | s)} > 1 + \epsilon$ )
The probability of a good action decreases significantly ( $A > 0$ , $\frac{\pi_\theta (a | s)}{\pi_{\theta_k} (a | s)} < 1 - \epsilon$ )

In these cases, clipping would hide just how bad the new policy is and make the gradient of our objective $0$ for sufficiently bad updates. The $\min$ helps us restore just how bad the policy is and also restores our gradient, so that way PPO can push the policy in a more optimal direction. Effectively, the $\operatorname{clip}$ caps how good a move can look, but $\min$ ensures a bad move can’t look good because of the $\operatorname{clip}$ .

The reason this algorithm is much more sample efficient is because of the $\operatorname{argmax}$ component. It enables you to use the samples repeatedly to make many updates using the gradient of the objective, as long as you don’t stray too far away from the original policy.