The Bitter Lesson, in World Models

Sutton's bitter lesson states that general methods win, computation grows, and hand-built knowledge becomes a liability.

World models test the lesson against a domain that once seemed to belong to hand-built physics. This post builds one tiny world three ways — a hand-written simulator, a learned model, and a hybrid — and measures where each one wins. Every number below comes from the code in the post, which runs in a few seconds with NumPy.

The naive version: encode the physics by hand#

A hand-built world model encodes dynamics explicitly. Given the current state, a set of rules predicts the next state. Physics engines, game rules, and planning simulators share the same structure.

import itertools
import numpy as np
 
DT, G, DRAG = 0.05, 9.8, 0.08
 
def step_hand(s, a):
    # Hand-built simulator: Newton plus gravity, semi-implicit Euler. No drag.
    x, y, vx, vy = s
    vx, vy = vx + a[0] * DT, vy + (a[1] - G) * DT
    return np.array([x + vx * DT, y + vy * DT, vx, vy])

The code is exact inside its assumptions. Integration is deterministic. The rules are inspectable. A plan made inside the simulator matches the simulator's own future perfectly.

The world, however, is not the simulator. Here the real dynamics include quadratic air drag that the engineer did not write down:

def step_true(s, a):
    # The world the engineer did not fully model: quadratic air drag.
    x, y, vx, vy = s
    speed = np.hypot(vx, vy)
    vx = vx + (a[0] - DRAG * speed * vx) * DT
    vy = vy + (a[1] - G - DRAG * speed * vy) * DT
    return np.array([x + vx * DT, y + vy * DT, vx, vy])

Measuring a world model: error against horizon#

A world model is used for planning, so the right test is open loop. Start the model and the world in the same state, feed both the same action sequence, and record how far apart they drift after $t$ steps.

def horizon_error(model, H=40, trials=200, seed=1):
    # Mean position error after t steps, for t = 1..H, open loop.
    rng = np.random.default_rng(seed)
    errs = np.zeros(H)
    for _ in range(trials):
        s_true = s_model = np.array([0.0, 0.0, *rng.uniform(-10, 10, 2)])
        for t in range(H):
            a = rng.uniform(-5, 5, 2)
            s_true, s_model = step_true(s_true, a), model(s_model, a)
            errs[t] += np.linalg.norm(s_true[:2] - s_model[:2])
    return errs / trials

For the hand-built simulator:

One-step error is tiny. Forty-step error is nearly ten units. The simulator is not wrong by much at any instant; it is wrong in the same direction at every instant, and the errors add up.

The bottleneck is coverage. Drag, contact, friction, fluids, and deformable objects each demand new rules, and each rule demands an engineer. Human effort grows with coverage, and coverage never reaches the deployment distribution.

Deriving the structure underneath#

Rich Sutton's 2019 essay "The Bitter Lesson" argues that, over the long run, performance comes from general methods that scale with computation — search and learning — rather than from human-encoded domain knowledge. The essay's examples are computer chess, computer Go, speech recognition, and computer vision. In each, methods built on human understanding of the domain led at first and were overtaken once compute grew.

The lesson is not that knowledge is useless. The lesson is that knowledge encoded by hand rarely survives contact with scale.

A learned world model, in the line that runs from "World Models" (Ha and Schmidhuber, 2018) through the Dreamer agents, has three parts:

• an encoder that compresses an observation into a latent state,
• a dynamics model that predicts the next latent from the current latent and an action,
• a decoder or reward head that turns latents back into something checkable.

In this toy, the state is already low-dimensional, so the encoder is the identity and the experiment isolates the dynamics model. That is where the bitter lesson lives.

Step 1: replace the rules with a generic function class#

Bottleneck: the hand-built step function contains exactly the physics someone thought to write.

Fix: fit a generic function class to observed transitions. Polynomials in velocity and action contain no gravity, no drag, and no Newton — only capacity.

def features(s, a, degree):
    # Generic polynomial in (vx, vy, ax, ay). No physics inside.
    v = np.concatenate([s[2:], a])
    out = [1.0]
    for d in range(1, degree + 1):
        out += [np.prod(c) for c in itertools.combinations_with_replacement(v, d)]
    return np.array(out)
 
def collect(n, seed):
    # n random one-step transitions sampled from the real world.
    rng = np.random.default_rng(seed)
    S = np.column_stack([np.zeros((n, 2)), rng.uniform(-10, 10, (n, 2))])
    A = rng.uniform(-5, 5, (n, 2))
    S2 = np.array([step_true(s, a) for s, a in zip(S, A)])
    return S, A, S2
 
def fit(S, A, S2, degree, base=None, lam=1e-6):
    # Ridge regression on the residual: next_state - prior(state, action).
    prior = base if base else (lambda s, a: s)
    X = np.array([features(s, a, degree) for s, a in zip(S, A)])
    Y = S2 - np.array([prior(s, a) for s, a in zip(S, A)])
    W = np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ Y)
    return lambda s, a: prior(s, a) + features(s, a, degree) @ W

The features drop position because the dynamics are translation invariant. That is one small piece of structure, and it is worth noticing: even the "general" method carries a choice of inputs.

Step 2: scale data and capacity together#

Bottleneck: a learned model is only as good as its data and its function class. The bitter lesson predicts that both should grow together.

Fix: sweep both and measure the 40-step error.

for degree in (2, 3, 4):
    for n in (30, 1000, 10000):
        S, A, S2 = collect(n, seed=n)
        err = horizon_error(fit(S, A, S2, degree))
        print(degree, n, err[0], err[9], err[39])

Three patterns appear, and each is a sentence from the essay made concrete.

1. Thirty transitions beat the engineer. A quadratic with no physics in it already has a quarter of the hand-built model's 40-step error.
2. Data alone saturates. The degree-2 model stops improving between 1,000 and 10,000 transitions. Its function class cannot represent $\|v\|\,v$, so more data cannot help.
3. Capacity without data diverges. A degree-3 model fit to 30 transitions has more parameters than it can pin down. Its rollouts blow up to infinity within ten steps.

Scale is a pair: more parameters need more data, and more data needs more parameters. Growing either alone stalls.

Step 3: measure compounding, not one-step fit#

Bottleneck: one-step error hides the failure that matters. The degree-3 model at 30 samples has a one-step error of 0.57 — bad, but finite — and then diverges.

The arithmetic of compounding is unforgiving. If each step is "right" with probability $p$, a rollout of length $H$ is right with probability roughly

Errors in continuous state behave worse than this, because a model that drifts off the training distribution starts making predictions about states it has never seen. A useful summary is the growth ratio:

def growth(errs, h_short=1, h_long=40):
    # How much the error multiplies between short and long horizons.
    return errs[h_long - 1] / max(errs[h_short - 1], 1e-12)

For the hand-built model the ratio is about 670. For the degree-4 model with 1,000 transitions it is about 220, and the absolute error is eighty times smaller. A model with low one-step error and a steep growth ratio produces confident long-horizon plans inside a fiction.

Note what the comparison does not say. The hand-built simulator also compounds error. Exact rules do not prevent drift once the rules disagree with the world; the sim-to-real gap is that disagreement, compounded.

Step 4: put the physics back as a prior#

Bottleneck: at small data, the learned model is unreliable, and the hand-built model is biased. Neither is good enough.

Fix: learn the residual. Use the simulator as the prior and let the regression model only the difference between simulator and world. The same fit function does this when given a base:

S, A, S2 = collect(30, seed=30)
pure = horizon_error(fit(S, A, S2, degree=3))
hybrid = horizon_error(fit(S, A, S2, degree=3, base=step_hand))
print(pure[39], hybrid[39])   # diverges, then about 2.5

With 30 samples, the residual model is stable where the pure model is not, and it is four times better than the simulator alone. With 1,000 samples the two are indistinguishable: the learned part has absorbed everything the prior contributed. Structure buys sample efficiency; it does not raise the ceiling.

Step 5: verify imagined trajectories#

Bottleneck: a planner searches for action sequences the model rates highly, and those are exactly the sequences where the model is most likely wrong.

Fix: check invariants the physics guarantees, and reject plans that violate them. With drag and no thrust, mechanical energy cannot increase:

def energy(s):
    return 0.5 * (s[2] ** 2 + s[3] ** 2) + G * s[1]
 
def verify_rollout(model, s0, H=40, tol=1e-6):
    # Coast with zero thrust; a passive system with drag must lose energy.
    s, a = s0, np.zeros(2)
    for t in range(H):
        s_next = model(s, a)
        if energy(s_next) > energy(s) + tol:
            return t        # first step where the model invents energy
        s = s_next
    return None

A learned model has no reason to respect an invariant it was never told about. The check costs one line per step and catches a class of hallucinated futures before a controller acts on them.

Where the lesson holds, and where it breaks#

The lesson holds where data is abundant and the dynamics are reachable by a general function class: video, game traces, large robotics logs. There, one architecture shared across domains improves with every increase in data and compute, while per-domain simulators restart from zero each time.

The lesson bends where data is scarce, horizons are long, or precision is non-negotiable. Safety-critical control, rare events, and counterfactual planning ask for behavior in states the data barely covers. In that regime the toy's Step 4 is the honest answer: structure as a prior, learning as a correction, invariants as a check.

What to measure#

1. One-step error. Local fit. Necessary, never sufficient.
2. Error at the planning horizon. The number that predicts whether a plan transfers.
3. Growth ratio. Long-horizon error divided by one-step error. A steep ratio flags drift even when both numbers look small.
4. Error against data size, per model size. The scaling table above, for the real system. A plateau means the function class is the bottleneck; divergence means the data is.
5. Invariant violations per rollout. Energy, mass, contact, joint limits — whatever the domain guarantees.
6. Planning success. A good predictor is not automatically a good planner; measure the plan in the real environment.

Checklist#

• State the data regime first. Abundant data favors learned models. Scarce data favors structure as a prior.
• Scale data and capacity together. Either one alone plateaus or diverges.
• Evaluate open loop at the horizon the planner uses, not one step ahead.
• Learn residuals on top of a simulator when samples are expensive.
• Verify imagined trajectories against invariants before executing plans.
• Re-run the scaling sweep when the data grows. The crossover point moves.