Reading a Loss Curve

A loss curve is not a score. It is a trace of a process, and like any trace, it can lie in specific ways.

The shape matters more than the final number. A curve that ends high may have been rising for the right reason. A curve that ends low may have collapsed into a degenerate policy. The first job is to learn the vocabulary of shapes.

The simplest model: plot the loss#

The naive model plots a single scalar against a single axis and reads the endpoint. A lower final loss means a better model. A higher final loss means a worse model.

The simplest logging loop written against that model is:

losses = []
for step in range(steps):
    loss = train_batch(batch)
    losses.append(loss)
plot(range(len(losses)), losses)

That loop answers one question: did the average batch loss go down? It cannot answer whether the policy improved, whether exploration collapsed, or whether a bug flattened every quantity at once. A flat line can mean a bug or a hard task. A falling line can mean learning or a degenerate policy exploiting a mis-specified reward.

The improvement starts by deriving the axes. The x-axis determines what a unit means. The y-axis determines what the value measures.

Deriving the axes: steps, updates, returns, and raw data#

Before reading the curve, check what is being plotted.

• Environment steps measure experience. Gradient updates measure optimization. They are not the same, especially in off-policy methods with a replay buffer.
• Training return is what the agent collected while exploring. Evaluation return is what a deterministic or near-deterministic policy achieves. The second is the number that matters.
• Episodes are misleading when episode lengths vary. A high return may come from a longer episode, not a better policy.
• Smoothed curves hide variance. Always keep the raw curve nearby.

A plot without units is a decoration. A plot with the wrong units is an argument.

The logging call must therefore record the axes separately. Mixing environment steps and gradient updates on a single x-axis makes the slope unreadable:

log = []
 
def record(env_steps, grad_updates, eval_return, train_return, entropy):
    log.append({
        "env_steps": env_steps,
        "grad_updates": grad_updates,
        "eval_return": eval_return,
        "train_return": train_return,
        "entropy": entropy,
    })

A loss curve alone is not enough. The quantities that explain it must sit on the same x-axis:

• evaluation return, with a fixed evaluation protocol;
• policy entropy;
• KL divergence from the reference policy, if there is one;
• critic loss and Q-value statistics;
• gradient norm;
• action distribution statistics;
• episode length;
• replay buffer size and age.

Plot them together. The cause is often visible in a different quantity than the one being watched.

Shape 1: flat from the start#

The curve does not move. The cause is usually structural.

• The learning rate is zero, or the optimizer is not stepping.
• The loss is detached from the computation graph.
• The actions are not being passed to the environment correctly.
• The observations are constant, or the reward is always zero.
• The policy has collapsed to a single action, and the entropy bonus is too small to escape.

The fastest test is a sanity check on a tiny problem. If the agent cannot solve a one-state bandit, the bug is not in the algorithm's convergence properties.

Shape 2: slow drift#

The curve rises, but too slowly to be useful. Common causes:

• The learning rate is too small.
• The observation or reward scale is wrong.
• Exploration is too low, so the replay buffer is narrow.
• The target network is updated too slowly, or the discount factor is too high for the horizon.
• The batch size is too small, so the gradient is noisy.

Slow drift is not always a bug. Some tasks are hard. The question is whether the curve is consistent with the compute budget. If a simple baseline learns faster, the problem is probably in the implementation.

Shape 3: spike and recovery#

The curve jumps, then returns. A single spike is often harmless: an outlier batch, a large gradient, a rare transition. A series of spikes is a warning.

• Gradient clipping may be missing or too loose.
• The learning rate may be at the edge of stability.
• The target network may be updating too often.
• The replay buffer may contain stale or badly normalized data.

If the curve recovers to the same trend, the spike is noise. If the trend changes after the spike, the spike was an event.

Shape 4: divergence#

The curve goes up and does not come back. In RL, divergence often looks like a policy that becomes more confident and more wrong.

• The critic is overestimating, and the actor is exploiting the error.
• The KL penalty is too weak, and the policy drifts away from the data.
• The reward scale is too large for the value function.
• The entropy coefficient is too low, and the policy collapses.
• The replay buffer is normalized with statistics that no longer match the data.

Divergence is a diagnosis, not a verdict. The fix depends on which quantity is exploding. Log the gradient norm, the Q-values, the entropy, and the KL term separately.

Shape 5: plateau#

The curve rises, then flattens below the target. Causes:

• The policy has found a local optimum.
• Exploration has collapsed.
• The network capacity is too small.
• The reward is sparse, and the agent never reaches the rewarding state.
• The evaluation is deterministic, and the training policy is stochastic; the gap is exploration.

A plateau is often a sign that the algorithm is working and the task is hard. It is also a sign that the current objective is not the one expected.

Shape 6: train/evaluation gap#

Training return keeps rising; evaluation return stalls or falls. That pattern is the RL version of overfitting, but the mechanism is different.

• The policy is exploiting the training environment's quirks.
• The evaluation is deterministic and the policy is too brittle.
• The environment has a distribution shift between training and evaluation.
• The replay buffer is stale, and the policy has moved away from the data.

A gap is not automatically bad. It is a signal to look at the evaluation protocol.

Step 1: smooth without hiding variance#

Bottleneck: raw RL curves are noisy. A reader cannot distinguish a trend from noise at a single seed.

Fix: an exponential moving average makes the trend readable. Two cautions apply. First, the first few points are biased toward the initial value; warm up or discard them. Second, smoothing can hide the spike that matters. Keep the raw curve.

def ema(values, beta=0.9):
    out = []
    last = values[0]
    for value in values:
        last = beta * last + (1 - beta) * value
        out.append(last)
    return out

The EMA is a reading aid, not evidence. A smoothed curve without the raw data beside it is a claim without support.

Step 2: replace one seed with a distribution#

Bottleneck: a single curve is an anecdote. Reinforcement learning is noisy, and seeds can change the conclusion.

Fix: report at least three seeds. Plot the median and the interquartile range, or plot every seed with low opacity. A method that wins on one seed and loses on two is not a method; it is a coincidence.

import numpy as np
 
def summarize(runs: list[list[float]]) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    arr = np.asarray(runs)  # shape: seeds x steps
    return np.median(arr, axis=0), np.percentile(arr, 25, axis=0), np.percentile(arr, 75, axis=0)

The same applies to hyperparameters. A sweep that selects the best seed is not a sweep; it is a search for a story. Use a validation protocol, or report the distribution.

Worked example: the shared bug#

Consider a codebase in which SAC, TD3, and DDPG all fail on a simple continuous-control task, and all three curves look indistinguishable from random. The tempting move is to tune each algorithm separately. The better hypothesis is that they share a bug.

A plausible shared cause sits in the common base class: observations are normalized when they are stored in the replay buffer, using statistics from a handful of early samples. With a tiny estimated variance, almost everything is clipped to the edge of the range. Those samples stay in the buffer for the whole run, so the critic trains on a mixture of distributions that disagree about what each state looks like.

The fix is to store raw observations and normalize when sampling, using the statistics of the current buffer.

# Store raw; normalize at sampling time from buffer statistics.
def sample_normalized(buffer, batch_size):
    buffer_obs = buffer.observations()          # raw
    mu = buffer_obs.mean(axis=0)
    std = buffer_obs.std(axis=0) + 1e-8
    idx = np.random.choice(len(buffer_obs), batch_size)
    return (buffer_obs[idx] - mu) / std

With the fix in place, all three curves should move together. If they do, the bug was shared.

The lesson is not "normalization is hard." The lesson is that three identical failures are one failure with three names. When several methods fail in the same way, look at what they share before tuning what they do not.

What to measure#

• Eval return on a fixed protocol. The primary quantity. Training return is secondary.
• Entropy. Collapse to a single action signals premature convergence.
• Gradient norm. Explosion precedes divergence.
• Q-value mean and variance. Systematic overestimation precedes actor exploitation.
• KL term. Drift from the reference policy signals a weak penalty.
• Buffer age. Stale data explains train/evaluation gaps.
• Median and IQR across seeds. The distribution is the result; the single curve is a sample.

A checklist#

• Check the axes before the shape. Steps, updates, and returns measure different things.
• Plot raw and smoothed together. Smoothing is an aid, never a replacement.
• Report at least three seeds. Median and interquartile range beat a single winner.
• Log the explaining quantities on the same x-axis. Entropy, KL, gradient norm, and Q-statistics localize the cause.
• Treat identical failures across algorithms as one shared bug before tuning any of them.
• Sanity-check on a tiny problem. A one-state bandit isolates implementation from convergence.
• Distinguish a spike from an event. Recovery to the same trend is noise; a changed trend is a diagnosis.

A loss curve is a conversation between the code and the person reading it. A flat line says "nothing is happening." A spike says "something happened once." A divergence says "the feedback loop is unstable." A plateau says "the easy progress is over."

Learn the vocabulary, and the curve will tell the reader where to look. It will not tell the answer. That part remains manual.