What Is a Loss Function?

A loss function is a scalar summary of everything wrong with a prediction.

Training descends that scalar. The shape of the scalar decides the path, the destination, and the behavior at the margin.

The simplest version#

The smallest learning problem contains only a constant prediction. Data arrives as pairs of inputs and targets. The model ignores the input and predicts a single number. The loss measures the average mismatch between that number and the targets.

import numpy as np
 
def mse_loss(pred, target):
    return np.mean((pred - target) ** 2)
 
def predict_constant(targets):
    # The optimal constant under squared error is the mean.
    return np.mean(targets)
 
y = np.array([1.0, 2.0, 3.0, 100.0])
c = predict_constant(y)
print(c)                     # 26.5
print(mse_loss(c, y))        # mean squared error around the mean

The code contains the entire idea. A prediction is scored. The score is averaged over examples. Optimization changes parameters to reduce the score. No gradient is needed yet, because a constant can be solved in closed form.

The example already reveals the central commitment. A single outlier moves the optimal constant from 2 to 26.5. The loss defines success before any algorithm runs.

Deriving the structure underneath#

A parametric model maps inputs to predictions through weights. A loss maps predictions and targets to a scalar. Training combines the two by minimizing the scalar over the training set.

# y_hat = model(x; w) composed with loss L(y_hat, y)
# Gradient descent:  w <- w - lr * dL/dw
 
def losses(pred, target):
    mse = np.mean((pred - target) ** 2)
    mae = np.mean(np.abs(pred - target))
    return mse, mae
 
def numerical_gradient(fn, w, eps=1e-5):
    grad = np.zeros_like(w)
    f0 = fn(w)
    for i in range(len(w)):
        w2 = w.copy()
        w2[i] += eps
        grad[i] = (fn(w2) - f0) / eps
    return grad

Three structural facts follow from that composition.

First, differentiation requires a scalar. A gradient is a vector of partial derivatives of a single objective. Without compression to one number, no single descent direction exists. Compression is not an implementation detail. Compression is the requirement that makes gradient-based learning well-defined.

Second, differentiation requires a smooth-enough surface. A loss with flat regions provides no signal. A loss with discontinuities provides no usable derivative. A loss with exploding slopes destabilizes updates. The usable loss sits between those failures.

Third, weighting is inevitable. Uniform averaging over examples assumes every example matters equally. Reweighting changes the effective dataset. Every loss therefore encodes a judgment about which errors matter most, made before training begins.

Naive loss one: mean squared error#

Mean squared error penalizes the squared distance between prediction and target. Regression uses the form because the target is continuous and distance carries natural meaning.

def mse(pred, target):
    return np.mean((pred - target) ** 2)
 
def mse_grad(pred, target):
    # d/dpred of mean((pred - target)^2)
    return 2.0 * (pred - target) / len(pred)
 
# Linear regression closed form minimizes this loss directly.
def fit_linear_mse(X, y):
    X = np.column_stack([X, np.ones(len(X))])
    w, *_ = np.linalg.lstsq(X, y, rcond=None)
    return w

The geometry is attractive. The loss is convex for linear models, has a unique closed-form solution, and assigns large errors disproportionate weight. The square means a mistake of 10 counts one hundred times as much as a mistake of 1.

The bottleneck appears at the margin. Outliers pull harder than typical cases. A few extreme points dominate the gradient, and the model chases them at the expense of the bulk of the data. A second bottleneck appears under uncertainty. When outcomes are multimodal, squared error prefers the conditional mean. The prediction in the middle may correspond to no outcome that ever occurs.

Fix one: change the penalty shape#

Mean absolute error replaces the square with an absolute value. Large errors gain weight linearly rather than quadratically. The model becomes more robust to outliers and less sensitive to the tail.

def mae(pred, target):
    return np.mean(np.abs(pred - target))
 
def huber(pred, target, delta=1.0):
    err = pred - target
    abs_err = np.abs(err)
    quadratic = 0.5 * err ** 2
    linear = delta * (abs_err - 0.5 * delta)
    return np.mean(np.where(abs_err <= delta, quadratic, linear))
 
def huber_grad(pred, target, delta=1.0):
    err = pred - target
    grad = np.clip(err, -delta, delta)
    return grad / len(pred)

Huber loss interpolates between the two regimes. Near zero the penalty is quadratic and smooth. Far from zero the penalty is linear and bounded in slope. The single parameter controls where the transition occurs.

The trade is explicit. Robustness improves. Strong gradients at large distances disappear, which can slow learning when the initial predictions are far from the targets. The loss that resists outliers also resists correction.

Naive loss two: classification with squared error#

Squared error can be applied to classification by encoding labels as numbers, but the encoding creates a false geometry. Class 2 is not located between class 1 and class 3. Distance between labels means nothing.

Classification needs a probabilistic loss. The model outputs a distribution over classes. The target is the correct class. The loss scores how much probability mass the distribution placed on the correct choice.

def softmax(logits):
    z = logits - np.max(logits, axis=1, keepdims=True)
    e = np.exp(z)
    return e / np.sum(e, axis=1, keepdims=True)
 
def cross_entropy(logits, labels):
    probs = softmax(logits)
    n = len(labels)
    # -log p(correct class)
    return -np.mean(np.log(probs[np.arange(n), labels] + 1e-12))
 
def cross_entropy_grad(logits, labels):
    probs = softmax(logits)
    n = len(labels)
    grad = probs.copy()
    grad[np.arange(n), labels] -= 1.0
    return grad / n

The gradient has a clean form. The update pushes probability toward the correct class and away from every competing class in proportion to current probability. Confidence matters as much as the top choice. A correct prediction at 99 percent confidence loses far less than a correct prediction at 51 percent.

The bottleneck is sharpness. Cross-entropy punishes confident errors heavily. A wrong prediction at 99 percent confidence incurs a large loss. That sharpness suits clean classification and creates fragility under label noise. Noisy labels force the model to spend capacity memorizing errors.

Fix two: temper the sharpness#

Label smoothing replaces the hard target distribution with a mixture of the hard target and a uniform distribution. No class receives zero probability and no class receives full probability. The loss stops demanding unlimited confidence.

def cross_entropy_smoothed(logits, labels, eps=0.1, num_classes=None):
    probs = softmax(logits)
    n = len(labels)
    k = num_classes or probs.shape[1]
    # Target distribution: (1-eps) on correct class, eps/(k-1) elsewhere
    target = np.full_like(probs, eps / (k - 1))
    target[np.arange(n), labels] = 1.0 - eps + eps / k
    return -np.mean(np.sum(target * np.log(probs + 1e-12), axis=1))
 
def focal_loss(logits, labels, gamma=2.0):
    probs = softmax(logits)
    n = len(labels)
    p_correct = probs[np.arange(n), labels]
    ce = -np.log(p_correct + 1e-12)
    return np.mean(((1.0 - p_correct) ** gamma) * ce)

Focal loss attacks the same bottleneck from a different direction. Easy examples receive exponentially smaller weight. Hard examples dominate the gradient. Class imbalance, where easy majority examples drown rare hard examples, becomes manageable.

Both fixes trade sharpness for control. Smoothing prevents overconfidence. Focal weighting prevents easy examples from dominating. Neither changes the underlying commitment to probabilistic fit. Both change the margin behavior.

A different objective: contrastive loss#

Representation learning needs a loss that shapes geometry rather than prediction. Positive pairs pull together. Negative pairs push apart. Similarity becomes the learned structure.

def contrastive_loss(z, labels, temperature=0.07):
    # z: (n, d) normalized embeddings, labels: (n,)
    n = z.shape[0]
    sim = (z @ z.T) / temperature
    # Mask self-similarity
    sim = sim - np.eye(n) * 1e9
    # For each anchor, positives share the label
    positives = (labels[:, None] == labels[None, :]) & ~np.eye(n, dtype=bool)
    # Log-softmax denominator over all other examples
    log_denom = np.log(np.sum(np.exp(sim), axis=1))
    log_num_pos = np.sum(np.where(positives, sim, -1e9), axis=1)
    # Simplified version: contrast anchor against all others
    loss = -np.mean(np.log(np.exp(sim[np.arange(n), 0]) + 1e-12))
    return loss
 
def triplet_loss(anchor, positive, negative, margin=1.0):
    dist_ap = np.sum((anchor - positive) ** 2, axis=1)
    dist_an = np.sum((anchor - negative) ** 2, axis=1)
    return np.mean(np.maximum(dist_ap - dist_an + margin, 0.0))

The loss itself defines no similarity. Pair construction defines similarity. Positive sampling decides what counts as the same. Negative sampling decides what counts as different. Temperature, margin, and batch composition tune the geometry further. Small changes in sampling produce large changes in the learned structure.

The bottleneck is batch dependence. Contrastive losses need many negatives per batch to learn useful separations. Small batches starve the denominator. Hard negatives improve the structure dramatically and destabilize training when sampled too early.

What to measure#

Loss values from different objectives cannot be compared directly. A lower cross-entropy does not imply a better model than a lower squared error. Each number lives inside its own objective.

Three measurements separate progress from illusion. First, track the training loss and the validation loss on the same objective. A widening gap signals overfitting. Second, track a task metric that shares no formula with the loss. Accuracy for classification, ranking quality for retrieval, calibration error for probabilities. Third, inspect the margin behavior. Histogram the per-example loss and observe whether progress comes from fixing easy examples or hard ones.

def per_example_cross_entropy(logits, labels):
    probs = softmax(logits)
    n = len(labels)
    return -np.log(probs[np.arange(n), labels] + 1e-12)
 
def loss_report(logits, labels):
    losses = per_example_cross_entropy(logits, labels)
    return {
        "mean": np.mean(losses),
        "median": np.median(losses),
        "p90": np.percentile(losses, 90),
        "frac_confident_errors": np.mean(losses > 5.0),
    }

Checklist#

• Name the target type: continuous, categorical, or relational. The type selects the loss family.
• State the commitment: distance, probability, or geometry. The loss enforces exactly one.
• Check the outlier profile. Squared losses chase extremes. Robust losses resist them.
• Check the calibration implication. Cross-entropy trains probabilities. Squared error trains conditional means.
• Inspect pair construction for contrastive losses. Sampling is the specification.
• Report a task metric alongside the loss. The scalar shows progress on the chosen measure. The metric shows whether the measure deserves trust.