Why Alignment Is Not Fine-Tuning

Fine-tuning teaches a model to produce a kind of text. Alignment teaches a model to prefer certain behavior over other equally plausible behavior.

The first changes the distribution. The second changes the choice within the distribution. This post makes that difference concrete with a policy small enough to print: one prompt, four possible responses, and NumPy. It runs supervised fine-tuning, derives a reward model and DPO from the same preference data, and then shows reward hacking appear on schedule.

The naive version: imitate the demonstrations#

Supervised fine-tuning, or SFT, maximizes the likelihood of demonstration completions. The target is a reference continuation. Success means matching the reference distribution.

import numpy as np
 
def log_softmax(logits):
    z = logits - logits.max(axis=-1, keepdims=True)
    return z - np.log(np.exp(z).sum(axis=-1, keepdims=True))
 
def sft_loss(logits, targets, mask):
    # logits: (batch, seq, vocab); targets: (batch, seq); mask: 1 on completion tokens
    logp = log_softmax(logits)
    b, t = np.indices(targets.shape)
    nll = -logp[b, t, targets]
    return (nll * mask).sum() / mask.sum()

The objective is pure imitation. The model learns to sound like the data because sounding like the data is the only thing being measured.

The smallest policy that shows the problem#

Shrink the language model to a single prompt with four possible responses. The policy is a vector of four logits.

import numpy as np
 
RESPONSES = ["helpful answer", "polite refusal", "harmful answer", "rambling answer"]
ONE_HOT = np.eye(len(RESPONSES))
 
def softmax(z):
    z = z - z.max()
    return np.exp(z) / np.exp(z).sum()
 
# Reference policy after pretraining: the harmful answer is a common continuation.
ref = np.log(np.array([0.30, 0.10, 0.35, 0.25]))
 
def sft(logits, demos, lr=0.5, steps=5):
    # Gradient of mean -log p(y) with respect to the logits is p - onehot(y).
    for _ in range(steps):
        p = softmax(logits)
        logits = logits - lr * np.mean([p - ONE_HOT[y] for y in demos], axis=0)
    return logits
 
pi_sft = sft(ref.copy(), demos=[0, 0, 0, 1])   # three helpful demos, one refusal

SFT raised the demonstrated responses. The harmful answer fell only because probability is normalized: mass moved to the demonstrations and had to come from somewhere. With four responses that side effect is large. With the exponentially many responses of a real language model, it is negligible. The demonstrations never said "not that", and the loss has no term that could.

That is the bottleneck. A demonstration shows what to do in one case and says nothing about what to avoid in all the others.

Deriving the structure: compare, do not copy#

Alignment changes the unit of data. The unit is no longer a continuation. It is a ranked pair of continuations for the same prompt.

# (chosen, rejected) indices into RESPONSES, for the same prompt
pairs = [(0, 2), (1, 2), (0, 3)]   # helpful > harmful, refusal > harmful, helpful > rambling

Pairs express what demonstrations cannot: restraint, refusal, tone, and the negative half of every judgment. Three components follow.

1. A preference signal: the pairs.
2. A mechanism that turns relative rankings into parameter updates.
3. A guard that stops the update from running past the region where the signal is valid.

Remove any one and alignment collapses back into imitation or into noise.

Fix one: learn a reward model, then optimize against it#

The Bradley–Terry model turns a scalar reward $r$ into a preference probability:

Fit $r$ by maximizing the log-likelihood of the observed pairs:

def bradley_terry_loss(r_chosen, r_rejected):
    # -log sigmoid(r_w - r_l), written stably
    return np.mean(np.logaddexp(0.0, -(r_chosen - r_rejected)))
 
def fit_reward(pairs, n, lr=0.5, steps=500):
    r = np.zeros(n)
    for _ in range(steps):
        grad = np.zeros(n)
        for w, l in pairs:
            g = -1.0 / (1.0 + np.exp(r[w] - r[l]))   # d loss / d (r_w - r_l)
            grad[w] += g
            grad[l] -= g
        r -= lr * grad / len(pairs)
    return r

RLHF then maximizes reward under a leash. The policy is penalized for moving away from the reference:

The KL term is the guard. The reward model was trained on responses near the reference policy; it is trusted there and distrusted far away. In large models this objective is optimized with PPO on sampled responses. In the toy it can be solved exactly, which is the next step.

Fix two: remove the reward model#

The KL-regularized objective has a closed-form optimum:

Solve for the reward:

Substitute into Bradley–Terry. The partition function $Z(x)$ is the same for both responses and cancels. What remains is a loss on the policy alone — Direct Preference Optimization:

def dpo(logits, ref_logits, pairs, beta=0.1, lr=1.0, steps=50):
    logp_ref = np.log(softmax(ref_logits))
    for _ in range(steps):
        logp = np.log(softmax(logits))
        grad = np.zeros_like(logits)
        for w, l in pairs:
            margin = beta * ((logp[w] - logp_ref[w]) - (logp[l] - logp_ref[l]))
            dloss = -1.0 / (1.0 + np.exp(margin))          # d(-log sigmoid)/d margin
            # d logp[y]/d logits = onehot(y) - p; the p terms cancel in the difference.
            grad += dloss * beta * (ONE_HOT[w] - ONE_HOT[l])
        logits = logits - lr * grad / len(pairs)
    return logits
 
def kl(p, q):
    return float((p * np.log(p / q)).sum())
 
for steps in (20, 100, 500):
    pi = dpo(pi_sft.copy(), pi_sft, pairs, steps=steps)
    print(steps, softmax(pi).round(2), kl(softmax(pi), softmax(pi_sft)))

Two things happen that SFT could not do. The harmful answer is pushed down directly, because it appears as a loser. And the policy moves only as far as the pairs ask.

A third thing is a warning. The refusal was preferred over the harmful answer, yet its probability fell from 0.12 to 0.05. DPO constrains margins, not absolute probabilities. The refusal's logit rose, but the helpful answer's rose faster, and normalization did the rest. On real data this shows up as the likelihood of chosen responses falling during training. Monitor it.

Bottleneck: the proxy diverges from intent#

Both approaches optimize a model of human judgment, trained on finite ratings. The policy searches a space far larger than the rating set, and optimization pressure finds the places where the model is wrong.

The simplest experiment is best-of-$n$ sampling against a proxy. Give every response a true quality and a length. The proxy reward likes quality and also likes length. Readers like quality and dislike padding.

rng = np.random.default_rng(0)
 
def best_of_n(n, trials=4000):
    quality = rng.normal(size=(trials, n))
    length = rng.normal(size=(trials, n))
    proxy = quality + length                               # what the reward model scores
    true = quality - 0.5 * np.maximum(length, 0) ** 2      # what a reader experiences
    rows, pick = np.arange(trials), proxy.argmax(1)
    return proxy[rows, pick].mean(), true[rows, pick].mean()

The proxy climbs monotonically. The true quality rises, peaks around $n = 16$, and then falls below where it started. More optimization pressure is not more alignment. Past a point it is more exploitation of the proxy's blind spot.

Detecting the divergence#

Rising proxy reward with flat or falling human scores is the signature. Stratify human ratings by proxy score:

def stratified_report(proxy, human, quantiles=(0, 0.5, 0.9, 0.99, 1.0)):
    edges = np.quantile(proxy, quantiles)
    rows = []
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (proxy >= lo) & (proxy <= hi)
        rows.append({
            "proxy_range": (float(lo), float(hi)),
            "n": int(m.sum()),
            "mean_human": float(human[m].mean()),
            "corr": float(np.corrcoef(proxy[m], human[m])[0, 1]) if m.sum() > 2 else None,
        })
    return rows

If mean human score at the top 1% of proxy is lower than at the top 10%, the policy is already past the peak in the table above. Mitigations follow the structure: a larger KL coefficient, early stopping, reward-model ensembles that penalize disagreement, and fresh preference collection under the current policy so the proxy is trained where the policy now lives.

What to measure#

1. Held-out preference accuracy. Does the policy's implicit reward rank unseen pairs correctly?
2. KL from the reference, estimated on samples from the policy. Large KL with flat preference accuracy is drift.
3. Chosen and rejected log-likelihoods separately. Both falling is the DPO warning from the table above.
4. Human score stratified by proxy score. The shape of that curve says whether higher reward still means better.
5. Best-of-$n$ curve. Proxy and human score against $n$. The peak locates the safe optimization budget.
6. Imitation metrics, kept separate. Perplexity and exact match measure fit, not preference. Neither substitutes for the other.

Checklist#

• Use SFT to set the format and the distribution. Do not expect it to teach avoidance.
• Collect ranked pairs for the same prompt. Single continuations cannot express preference.
• Treat every reward, explicit or implicit, as a proxy. Inspect it before trusting the policy.
• Keep the policy on a leash: KL penalty, early stopping, or both.
• Watch the likelihood of chosen responses during DPO, not only the margin.
• Re-collect preferences under the current policy when it drifts.
• Measure the best-of-$n$ peak before spending more optimization.