The Bitter Lesson

The bitter lesson, stated by Rich Sutton, is that the best AI methods are the general ones that scale with compute, and clever hand-designed systems lose to them over time.

The lesson is bitter because it runs against research intuition. It says the feature engineering, the search heuristics, and the clever representations that look like progress are the things that will be displaced. The argument is empirical, drawn from seventy years of results, and it has been right often enough to demand an answer.

The simplest version#

The simplest machine is a lookup table written by a person. Chess supplies the canonical example: rating a position by material counts and piece-square tables, compiled from expert knowledge.

# Material values in centipawns
VALUE = {"p": 100, "n": 320, "b": 330, "r": 500, "q": 900, "k": 20000}
 
def handcrafted_eval(board):
    score = 0
    for piece, square in board.pieces():
        v = VALUE[piece.fig]
        if piece.color == "w":
            score += v
        else:
            score -= v
    return score  # higher is better for White

The code is readable, debuggable, and finite. Its ceiling is visible in the source: every case the designer anticipated is a table entry, and every unanticipated case is a missing entry. The method contains no way to improve with more computation, because computation is not where the knowledge lives. The knowledge lives in the designer.

The underlying structure: what scales and what does not#

Two families of methods differ on exactly that axis. A hand-designed method stores knowledge in fixed structure: features, heuristics, rules. A general method stores knowledge in adjustable structure: parameters fitted by search or by gradient descent.

import numpy as np
import torch.nn as nn
 
class LearnedEval(nn.Module):
    """A general method: the evaluation function is fitted, not specified."""
    def __init__(self, board_features):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(board_features, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1),
        )
 
    def forward(self, features):
        return self.net(features).squeeze(-1)
 
# Training loop: more compute -> better fit, no designer intervention.
def train(model, positions, targets, steps=1000, lr=1e-3):
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    for _ in range(steps):
        pred = model(positions)
        loss = ((pred - targets) ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model

The architectural difference is a ceiling difference. A handcrafted heuristic is only as good as the cases its designer anticipated. A general learner has no ceiling of that kind. When the resource grows — more compute, more data, more training time — the general method improves, and the handcrafted one does not. The resource curve is exponential, so the gap compounds. That is the mechanism behind the pattern, and it is also the prediction: whatever wins in the short term, the method that absorbs the resource wins in the long term.

Chess: from pieces to self-play#

The chess history runs in three stages. The feature-engineering stage is the handcrafted evaluation above: experts supply the knowledge. The compute stage keeps a simple evaluation and spends resources on search instead. Shannon's insight was that deeper search with a simpler evaluation beats a deeper evaluation with shallower search. Deep Blue won by adding search and speed. The knowledge moved out of the evaluation and into the calculation.

import time
 
def negamax(board, depth):
    if depth == 0 or board.is_checkmate():
        return board.outcome()  # heuristic: simple, fast to evaluate
    best = -float("inf")
    for move in board.legal_moves():
        board.push(move)
        best = max(best, -negamax(board, depth - 1))
        board.pop()
    return best
 
def choose(board, budget_seconds=1.0):
    start = time.monotonic()
    best_move, depth = None, 1
    while time.monotonic() - start < budget_seconds:
        for move in board.legal_moves():
            board.push(move)
            value = -negamax(board, depth - 1)
            board.pop()
            if best_move is None or value > best_value:
                best_move, best_value = move, value
        depth += 1
    return best_move  # deepest search that fits the budget

The third stage removes the designer entirely. Reinforcement learning from self-play generated AlphaZero's chess: the evaluation and the policy were learned from games the system played against itself, with no handcrafted evaluation function at all. The method that won was the one with no designer knowledge in it. The pattern is that the fixed, human-designed representation stopped paying off once the learner had enough computation to find its own.

Vision: features learned instead of engineered#

Computer vision ran the same cycle. The feature-engineering era used SIFT features, HOG descriptors, and boosted ensembles — handcrafted front ends, each with a designer ceiling. Convolutional neural networks replaced them: the feature hierarchy is learned from data by gradient descent, so more compute and more images improve the features directly.

import torch.nn as nn
 
def handcrafted_frontend(image):
    # A designer-specified statistic. Fixed at compile time; cannot learn.
    return compute_histograms_of_oriented_gradients(image)
 
class LearnedFrontend(nn.Module):
    """The same slot, filled by learning: filters are parameters."""
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Conv2d(3, 32, 5), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 5), nn.ReLU(), nn.MaxPool2d(2),
            nn.AdaptiveAvgPool2d(1),
        )
 
    def forward(self, image):
        return self.layers(image).flatten(1)

The lesson was not that domain knowledge was useless. It was that learned representations, given enough data and compute, outperformed the best handcrafted features. The handcrafted front end had stopped improving; the learned one kept tracking the resource curve.

Language: statistics, then translation, then transformers#

Language processing repeated the cycle on a longer fuse. Statistical machine translation with handcrafted resources gave way to neural machine translation, which gave way to the Transformer, which gave way to large language models. Each step replaced a more bespoke, more handcrafted approach with a more general one trained on more data at more scale.

def expected_loss(compute, scale_exponent, constant_overhead):
    # A general method: loss keeps falling as compute grows.
    return constant_overhead * compute ** (-scale_exponent)
 
def expected_loss_handcrafted(compute, ceiling, designer_hours):
    # A handcrafted method: improvement requires human effort, not FLOPs.
    return ceiling + 1.0 / max(designer_hours, 1)
 
# compute grows exponentially over decades; designer_hours grows linearly.
# The crossover is structural, not a matter of tuning.
for flops in [1e15, 1e18, 1e21, 1e24]:
    print(flops, expected_loss(flops, 0.05, 20.0), expected_loss_handcrafted(flops, 2.0, 1000))

The attempts to reintroduce symbolic structure — knowledge bases, rule systems, curated ontologies — repeatedly beat the general method in the short term and lost in the long term. In the short term the handcrafted system can win, and often does. The lesson is that the win does not survive the next scaling step.

Speech: the same cycle in a different modality#

Speech recognition supplies the cleanest third case. The handcrafted era derived phoneme features from signal processing knowledge — mel filter banks, linear prediction coefficients — and combined them with expert-tuned acoustic models.

def handcrafted_frontend(waveform):
    # Signal-processing features: fixed transforms, no learning.
    # The designer chose the filterbank; the filterbank cannot improve.
    mfcc = mel_filterbank(waveform, bands=40)
    return mfcc
 
class LearnedSpeech(nn.Module):
    """The acoustic model and the features are fitted together."""
    def __init__(self):
        super().__init__()
        self.frontend = nn.Conv1d(1, 64, 10, stride=2)
        self.backbone = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=64, nhead=4), num_layers=4)
        self.head = nn.Linear(64, 256)  # phoneme targets
 
    def forward(self, waveform):
        return self.head(self.backbone(self.frontend(waveform)))

End-to-end neural models absorbed both the front end and the acoustic model into one fitted function. More audio and more compute improved both. The signal-processing features remain in use as preprocessing — data, not architecture — which is the lesson's nuance in miniature: domain knowledge survives where it functions as an input, and is displaced where it functions as a fixed decision procedure.

Bottleneck: the frontier of "general" moves#

The lesson does not imply that domain knowledge is worthless, that engineering does not matter, or that the next breakthrough will be another scale-up. It does not imply that transformers are the final architecture or that self-supervised pretraining is the final paradigm. It implies that whatever the next breakthrough is, it will be general within its domain and will improve with more compute and data.

It also does not imply that the "general method" is the same method forever. Search and learning were general in 1960; deep learning is general in 2020. The frontier of what counts as general moves, and the lesson moves with it. Each era's general method looked like the previous era's clever hack, right up until the resource curve made the hack the loser.

What to measure#

The bitter lesson is a claim about resource scaling, so the measurement is a scaling claim. The question is never whether a method works; everything works at some scale. The question is whether the method's error falls as a function of compute and data, or whether it is capped by designer effort.

The discipline the lesson asks for is a preference for methods that get better when the world gets more expensive to process, and a skepticism about wins that depend on the designer having foreseen the test cases. The general method wins because it bets on the resource curve. The bet has been safe for seventy years, and the lesson is the record of its safety.

A checklist for evaluating a method against the lesson:

• Plot error against compute. A general method shows a falling curve; a handcrafted method shows a plateau set by designer effort. The plateau is the diagnosis.
• Ask where the knowledge lives. If the method's edge comes from entries a person wrote, the edge is a ceiling. If it comes from parameters fitted to data, the edge compounds.
• Distinguish short-term and long-term wins. A handcrafted win at today's compute is evidence about today's compute only. Test the same comparison at the next doubling.
• Measure data scaling separately from compute scaling. Some methods scale in one and not the other, and the resource that matters is the one the field is actually growing.
• Check what the frontier of "general" currently means in the domain. The lesson moves with the frontier: methods general within their era are the ones that survive.
• Do not read the lesson as "features are worthless." Read it as "features installed by hand are loans against future compute, and the loan comes due."
• Keep domain knowledge where it functions as data rather than as architecture. Preprocessing, evaluation design, and curricula absorb resources; fixed heuristics do not.
• Report generalization, not just training performance. Benchmarks declare which measure is authoritative, and a method that wins a benchmark by construction has not escaped the designer's ceiling — it has hidden it.