The Sparse Checkpoint

A checkpoint is a contract between the model that was trained and the system that runs it. The contract specifies the number of parameters, the precision of each parameter, the file size, the memory footprint, and the compute required for inference. Changing any of those changes the contract, and may change the model.

Compression — quantization, pruning, distillation — changes the model in exchange for a smaller contract. The exchange is often worthwhile. It is rarely free.

This note starts with an uncompressed checkpoint, derives the size arithmetic from first principles, then applies each compression type step by step with the measure that decides whether the trade is safe.

The naïve version: ship the training checkpoint#

The simplest contract is the training file itself: full-precision weights, saved exactly as they left the optimizer. A 7-billion-parameter model at 32-bit float precision occupies

before any overhead for buffers, metadata, or the inference runtime. Inference reads every weight for every token generated, so latency and memory traffic both scale with this size. On a server with a large-memory GPU the file loads. On anything smaller, it does not.

def checkpoint_bytes(num_params, bits_per_param):
    return num_params * bits_per_param / 8
 
for bits in (32, 16, 8, 4):
    size_gb = checkpoint_bytes(7_000_000_000, bits) / 1e9
    print(f"{bits}-bit: {size_gb:.1f} GB")

The output is linear in the number of bits. Halving precision halves the file, halves the memory footprint, and roughly halves the bytes moved per token. The arithmetic is exact; the question is what the halving does to the model's answers.

First principles: what a weight actually is#

A weight is one floating-point number. The number of possible values a $b$-bit representation can hold is $2^b$, and the spacing between representable values grows as the range grows. Quantization chooses a mapping from the full-precision range to a smaller grid of integer codes:

def quantize_scalar(value, bits, vmin, vmax):
    levels = 2 ** bits - 1
    scale = (vmax - vmin) / levels
    code = round((value - vmin) / scale)
    code = max(0, min(levels, code))
    return code, vmin + code * scale
 
def quantize_tensor(weights, bits):
    vmin = min(weights)
    vmax = max(weights)
    codes = []
    reconstructed = []
    for w in weights:
        code, approx = quantize_scalar(w, bits, vmin, vmax)
        codes.append(code)
        reconstructed.append(approx)
    return codes, reconstructed
 
weights = [0.125, -0.9, 3.14, 0.0, 1.5]
codes, rebuilt = quantize_tensor(weights, 4)
print(codes, rebuilt)

Each weight is rounded to the nearest representable value. The rounding error is bounded by half the grid spacing, so its size depends on the range — and the range of a trained layer is dominated by a few outlier weights. The naive version below pays for that domination.

Bottleneck one: outliers stretch the grid#

The scalar quantizer above chooses vmin and vmax from the extreme values in the tensor. A single weight with magnitude 40 makes the grid spacing 80/15 ≈ 5.3 at 4 bits, while the bulk of the weights live in $[-0.5, 0.5]$ and round to two or three codes. Most of the precision is spent on the outliers; most of the model's signal is destroyed by the resulting coarse grid.

The fix splits the scale per output channel, so each channel carries its own minimum, maximum, and spacing:

def quantize_per_channel(matrix, bits):
    """matrix: (out_channels, in_features). One scale per row."""
    codes = []
    scales = []
    for row in matrix:
        vmin = min(row)
        vmax = max(row)
        levels = 2 ** bits - 1
        scale = (vmax - vmin) / levels
        codes.append([max(0, min(levels, round((w - vmin) / scale)))
                      for w in row])
        scales.append((vmin, scale))
    return codes, scales
 
def dequantize_row(codes_row, vmin, scale):
    return [vmin + c * scale for c in codes_row]

Per-channel scales confine each outlier to its own channel. The storage overhead is two floats per channel, negligible against the per-weight savings. Accuracy improves sharply at 4 bits; the cluster of ordinary weights gets a grid proportional to its own range rather than to the extreme value.

Bottleneck two: activations move at runtime#

Weights are static; activations are recomputed every forward pass. Dynamic quantization computes activation scales on the fly, which trades a small amount of compute for a tighter grid:

def dynamic_quantize_activation(vector, bits=8):
    vmin = min(vector)
    vmax = max(vector)
    levels = 2 ** bits - 1
    scale = (vmax - vmin) / levels
    codes = [max(0, min(levels, round((v - vmin) / scale))) for v in vector]
    return codes, (vmin, scale)
 
def matvec_int8(weights_codes, weight_scales, activation, act_scale):
    """Dequantize on the fly; exact packing is a hardware detail."""
    vmin, scale = act_scale
    out = []
    for row_codes, (wvmin, wscale) in zip(weights_codes, weight_scales):
        acc = 0.0
        for c, a in zip(row_codes, activation):
            w = wvmin + c * wscale
            acc += w * a
        out.append(acc)
    return out

Quantizing weights alone and leaving activations in full precision is already a large win on memory; quantizing activations too cuts the arithmetic cost, because integer multiply-accumulate runs faster than floating-point on the same silicon. At 8 bits the accuracy loss is small for most models. At 4 bits it becomes task-dependent: a model robust to rounding on one benchmark may be fragile on another.

The error compounds across layers. A rounding perturbation at layer $\ell$ shifts the input to layer $\ell+1$, whose own rounding perturbs it further. The total deviation grows with depth, and the deepest layers contribute the largest share. Measuring the error at a single layer understates the system error by design.

def layer_error(model, example_input, bits):
    """Difference in final output between full and quantized forward passes."""
    full = model_forward(model, example_input, bits=None)
    quant = model_forward(model, example_input, bits=bits)
    max_diff = max(abs(a - b) for a, b in zip(full, quant))
    mean_diff = sum(abs(a - b) for a, b in zip(full, quant)) / len(full)
    return max_diff, mean_diff

Bottleneck three: the bytes still move#

Quantization shrinks the file, but a pruned model shrinks the number of weights the arithmetic touches at all. Unstructured pruning sets individual weights to zero:

def prune_unstructured(weights, sparsity):
    """magnitude pruning: keep the largest (1 - sparsity) fraction"""
    threshold = sorted(abs(w) for w in weights)[
        int(len(weights) * sparsity)]
    return [w if abs(w) >= threshold else 0.0 for w in weights]
 
def sparsity_of(weights):
    zeros = sum(1 for w in weights if w == 0.0)
    return zeros / len(weights)

Small weights contribute little to the output in aggregate, and removing them leaves the large ones to carry the function. The sparsity-accuracy relationship is steep at the edges: a model that is 50% sparse may lose little accuracy, while a model 90% sparse may be unusable. The exact elbow depends on how redundant the trained representation is.

Structured pruning removes whole channels, heads, or neurons instead of individual weights, which the hardware can exploit without sparse-matrix machinery:

def prune_channels(matrix, sparsity):
    """Keep the channels with the largest L2 norm."""
    norms = [sum(x * x for x in row) ** 0.5 for row in matrix]
    keep = sorted(range(len(matrix)), key=lambda i: -norms[i])[
        : int(len(matrix) * (1 - sparsity))]
    return [matrix[i] for i in sorted(keep)]
 
def pruned_forward(matrix, x):
    """Dense forward pass over the surviving channels only."""
    return [sum(w * v for w, v in zip(row, x)) for row in matrix]

Unstructured sparsity needs a sparse storage format and hardware or library support to realise the savings; the zeros still occupy memory in a dense layout. Channel pruning drops rows outright, and the dense matvec above runs at the reduced size.

Pruning and quantization compose. A model can be both pruned and quantized, and the savings multiply. So do the errors: the pruned weights are the ones the quantizer rounds worst, and the quantized residuals shift the channel norms that pruning ranked.

def compress(weights, sparsity, bits):
    pruned = prune_unstructured(weights, sparsity)
    codes, scales = quantize_per_channel(pruned, bits)
    return codes, scales
 
# prune then quantize: test each combination separately
for sparsity, bits in [(0.0, 8), (0.5, 8), (0.0, 4), (0.5, 4)]:
    codes, scales = compress(base_weights, sparsity, bits)
    acc = evaluate(dequantize(codes, scales))
    print(f"sparsity={sparsity}, bits={bits}, accuracy={acc:.4f}")

Bottleneck four: the compressed model is not the compressed weights#

Distillation takes a different route. Instead of shrinking the trained weights, a smaller student is trained to imitate the teacher's outputs:

import math
 
def soft_cross_entropy(student_logits, teacher_logits, temperature):
    """KL divergence between temperature-softened distributions."""
    def softmax(logits, T):
        m = max(x / T for x in logits)
        exps = [math.exp(x / T - m) for x in logits]
        total = sum(exps)
        return [e / total for e in exps]
    p = softmax(teacher_logits, temperature)
    q = softmax(student_logits, temperature)
    return -sum(pi * math.log(qi + 1e-12) for pi, qi in zip(p, q))
 
def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0):
    soft = soft_cross_entropy(student_logits, teacher_logits, temperature)
    hard = -math.log(max(1e-12, softmax_list(student_logits)[labels]))
    return temperature * temperature * soft + hard

The temperature divides the logits before the softmax, exposing the relative ordering of non-target classes — the "dark knowledge" that a hard label erases. The student receives both the teacher's soft distribution and the true label. Distillation is the most effective strategy when the teacher is available and the student architecture can represent the function; it is also the most expensive, requiring a full training run rather than a post-processing step.

The student is not a compressed version of the teacher. It is a new model that learned from the teacher's outputs, with its own parameters and its own error profile.

The evaluation problem#

A compressed model is a different model. It has different errors, and it may fail on inputs the original handled. The benchmark that validated the original does not validate the compressed version. The error profile changes in specific ways:

def compare_models(original, compressed, dataset):
    diffs = []
    for example in dataset["examples"]:
        a = original.predict(example["input"])
        b = compressed.predict(example["input"])
        diffs.append((example, a, b, a == b))
    agree = sum(1 for _, _, _, ok in diffs if ok) / len(diffs)
    disagreements = [d for d in diffs if not d[3]]
    return agree, disagreements

• Distribution shift. Quantization error is larger on out-of-distribution inputs. The model is more fragile to the inputs it was not tuned on.
• Tail behaviour. Rare cases are more sensitive to rounding and pruning. Average accuracy may hold while tail accuracy drops.
• Calibration. A quantized model is often overconfident on the cases it gets wrong. The confidence does not reflect the increased error rate.

The right evaluation is a comparison, not a single number. Measure the original and the compressed version on the same benchmark, report the difference, and report where the difference is largest:

def tail_accuracy(results, percentile=1.0):
    """Accuracy on the hardest percentile of examples."""
    ordered = sorted(results, key=lambda r: r["margin"])
    tail = ordered[: max(1, int(len(ordered) * percentile / 100))]
    return sum(1 for r in tail if r["correct"]) / len(tail)
 
def calibration_table(results, bins=10):
    """Mean confidence versus mean accuracy per confidence bin."""
    table = []
    for i in range(bins):
        lo, hi = i / bins, (i + 1) / bins
        group = [r for r in results if lo <= r["confidence"] < hi]
        if group:
            table.append((lo, hi, len(group),
                          sum(r["confidence"] for r in group) / len(group),
                          sum(1 for r in group if r["correct"]) / len(group)))
    return table

The deployment decision#

Compression trades accuracy for practicality. The right amount depends on the constraint:

• Browser or phone: 4-bit quantization is usually necessary. The accuracy loss is the price of fitting at all.
• Server with a GPU: 8-bit is often sufficient. The loss is small.
• Server with a CPU: 8-bit or 16-bit, plus pruning, may be required for latency.

The decision is not about the model. It is about the system the model runs in. A model that is accurate but cannot be deployed is not useful. A model that is deployable but inaccurate is not useful either. The contract is the constraint, and the compression strategy must satisfy it.

What to measure#

The following checks settle whether a compressed checkpoint can ship:

def memory_budget(checkpoint, device_bytes):
    return checkpoint["file_bytes"] + checkpoint["runtime_bytes"] <= device_bytes
 
def latency_check(model, inputs, budget_ms):
    import time
    for x in inputs:
        start = time.perf_counter()
        model.predict(x)
        elapsed = (time.perf_counter() - start) * 1000
        if elapsed > budget_ms:
            return False, x, elapsed
    return True, None, None

The checklist:

• Report agreement with the original, not only accuracy. A compressed model can score well while disagreeing with the original on exactly the inputs where the original was correct.
• Measure the tail, not the mean. Slice the evaluation by percentile of confidence and by rarity of the input; the tail is where rounding and pruning do their damage.
• Check calibration after compression. Overconfidence on wrong answers is the first symptom that the error profile has moved.
• Test on out-of-distribution inputs. Compression error grows where the original model was already weakest, and in-distribution numbers will not show it.
• Verify the memory budget end to end: file bytes plus runtime allocations plus the KV cache at maximum context. A checkpoint that fits the device on paper can still OOM at decode time.
• Keep the original checkpoint beside the compressed one. Deleting the teacher removes the only reference that can quantify the trade.

A compressed model is not the original model made smaller. It is a model that approximates the original, with its own error profile. Treat it as a separate artifact. Evaluate it separately. Deploy it separately. Shrink the contract only when the new contract is acceptable.