The Reasoning Model Budget

A reasoning model is a model that generates an intermediate sequence before producing its answer. The intermediate sequence — often called a chain of thought — is the mechanism. The model receives more compute, in the form of more tokens, to work out the answer.

The structure follows the gold-standard shape: a naive direct-answer model, the autoregressive structure underneath, a chain-of-thought trace built step by step, the budget arithmetic, and the measurement questions that separate real thinking from decorative thinking.

The naive model: answer directly#

The simplest language model answers a question in one forward pass per token with no intermediate reasoning. A math question, a logic puzzle, or a multi-step derivation must be solved inside the network depth.

def answer_directly(question: str, model) -> str:
    # One-shot generation: the answer tokens are produced
    # conditioned only on the question.
    completion = ""
    context = question
    for step in range(64):
        token = model.next_token(context)
        completion += token
        context += token
        if model.is_stop(token):
            break
    return completion

The naive version fails on multi-step problems. A chain of three arithmetic operations has to be compressed into the fixed depth of the transformer. The model can memorize patterns, but composition beyond a few steps exceeds the depth. Accuracy collapses on problems that require scratch work.

The bottleneck is depth. A transformer computes each token with a fixed number of layers. Intermediate results have nowhere to live except the hidden state. Chain-of-thought externalizes the scratch work into the context.

The autoregressive structure: tokens as compute#

An autoregressive model generates one token at a time, conditioned on everything before it. Each generated token costs a full forward pass: the sequence passes through every layer, and the output distribution selects the next token.

def generate(question: str, model, max_tokens: int = 128) -> list[str]:
    tokens: list[str] = []
    context = question
    for _ in range(max_tokens):
        tok = model.next_token(context)
        tokens.append(tok)
        context += tok
        if model.is_stop(tok):
            break
    return tokens

Each token buys two things: another pass through the network, and another slot in the context where intermediate results persist. Chain-of-thought is additional depth purchased with sequence length. The tokens before the answer are not an explanation. They are the computation.

def scratchpad_answer(question: str, model):
    # Scratchpad: generate intermediate tokens first,
    # then the final answer conditioned on them.
    scratch = generate(question + "\nThink step by step.", model, 512)
    answer = generate(question + "\n" + "".join(scratch) + "\nAnswer:", model, 64)
    return scratch, answer

The scratchpad converts a depth problem into a length problem. Longer scratchpads allow deeper composition. The tradeoff moves from architecture to budget.

The budget arithmetic#

A reasoning model introduces a resource: the number of tokens the model may spend thinking. Parameters stay fixed. Training data stays fixed. The reasoning budget varies by caller, by API setting, or by deployment.

def cost_of_budget(budget_tokens: int, per_token: float) -> float:
    return budget_tokens * per_token
 
for budget in (256, 1024, 4096, 16384):
    print(budget, "tokens ->", cost_of_budget(budget, 0.00001), "cost units")

Accuracy scales with budget up to a point. A model allowed ten times as many thinking tokens answers harder questions more accurately. The improvement is not linear and it saturates. The direction is consistent. The curve has a slope and a ceiling.

where $B$ is the reasoning budget in tokens. Latency and cost grow linearly with $B$. Accuracy grows sublinearly. The operating point therefore balances score against time and money.

The bottleneck is that the budget must be set before the answer exists. A trivial question wastes a large budget. A hard question starves under a small budget. Adaptive budgets attempt to match spending to difficulty, which adds a prediction problem on top of the reasoning problem.

def allocate_budget(question: str, model, lo=512, hi=8192) -> int:
    difficulty = model.estimate_difficulty(question)  # scalar in [0, 1]
    return int(lo + difficulty * (hi - lo))

Adaptive allocation helps. Estimation errors hurt: under-budgeted hard problems fail early, over-budgeted easy problems pay for idle tokens.

What longer thinking does not fix#

Knowledge gaps. If the model lacks a fact, longer reasoning elaborates the absence. A hallucinated fact gains consistency and confidence with each step. The chain becomes more coherent and the premise stays false. Errors become harder to spot because the reasoning is internally smooth.

Underspecified problems. If a question admits multiple interpretations, a reasoning model selects one and elaborates it. The model does not ask for clarification. It commits. Chain length measures effort, not whether the interpretation was right.

Mislabeled or adversarial objectives. More compute on a noisy objective optimizes the wrong function more efficiently. A model with a larger budget finds shortcuts more reliably. The shortcut exists in the objective, not in the reasoning length.

Verbosity traps. A model trained to produce reasoning may emit tokens that look like reasoning but play no causal role. The answer may be fixed by the first few tokens while the remainder decorates the trajectory. Verifying causality requires intervention: truncating or corrupting the trace and observing whether the answer changes.

def causal_check(scratch, answer, model, question):
    # Corrupt one scratch step and regenerate the answer.
    broken = list(scratch)
    broken[0] = "WRONG STEP"
    answer2 = generate(question + "\n" + "".join(broken) + "\nAnswer:", model, 64)
    return answer2 != answer

A trace whose corruption leaves the answer unchanged is decorative. A trace whose corruption changes the answer is load-bearing. The test runs repeatedly across positions for a full causal profile.

The evaluation problem#

A benchmark with a fixed reasoning budget measures one point on a curve. Changing the budget changes the score. Two models evaluated at different budgets are not comparable. Selecting a favorable budget is equivalent to selecting a favorable slice.

def accuracy_curve(questions, model, budgets):
    points = []
    for budget in budgets:
        correct = 0
        for q in questions:
            _scratch, answer = scratchpad_answer(q, model)
            if grade(answer, q) == correct_answer(q):
                correct += 1
        points.append((budget, correct / len(questions)))
    return points

The right evaluation reports a curve: accuracy against budget, against tokens, or against wall-clock time. The x-axis must be named. Token counts and time counts diverge when batching or hardware parallelism changes.

Production adds a second drift: the API budget may change after deployment. A system built around model behavior at one budget behaves differently at another. Logging the budget alongside each response makes the behavior reproducible.

Self-correction: the next step#

The next improvement after longer thinking is self-correction: generate, critique, revise. The critique pass turns additional tokens into error repair rather than only forward search.

def revise(question, model, rounds=2):
    draft = "".join(scratchpad_answer(question, model)[1])
    for _ in range(rounds):
        critique = "".join(generate(question + "\nDraft: " + draft + "\nFind errors:", model, 256))
        draft = "".join(generate(
            question + "\nDraft: " + draft + "\nCritique: " + critique + "\nRevise:", model, 512))
    return draft

Self-correction costs more tokens and needs a reliable critic. A weak critic corrupts correct answers. A good critic improves accuracy at equal budget. The budget arithmetic still governs: every revision step consumes tokens that could fund a longer first pass.

What to measure#

The primary curve is accuracy against thinking tokens. The secondary curves are accuracy against latency and accuracy against cost. All three curves share the same y-axis and differ only in the x-axis.

Causal checks are the second measurement. Corruption tests and truncation tests reveal whether the trace drives the answer. Decorative reasoning wastes budget and misleads evaluation.

Calibration is the third measurement. Confidence should track correctness. Longer chains raise verbal confidence mechanically. Measuring the gap between stated confidence and empirical accuracy exposes the inflation.

def calibration_gap(claims):
    # claims: list of (stated_confidence, correct_bool)
    bins = {}
    for conf, ok in claims:
        b = round(conf * 10) / 10
        bins.setdefault(b, [0, 0])
        bins[b][0] += 1
        bins[b][1] += int(ok)
    gaps = {b: counts[1] / counts[0] - b for b, counts in bins.items()}
    return gaps

A checklist#

• Report accuracy as a curve against thinking tokens, latency, and cost.
• Name the x-axis in every chart. Tokens and time are different budgets.
• Run corruption and truncation tests to verify the trace causes the answer.
• Measure calibration separately from accuracy. Confidence grows with length by construction.
• Log the deployed budget with every response. Budget drift changes behavior.
• Compare self-correction against longer first passes at equal token spend.
• Keep knowledge, problem formulation, and benchmark quality as separate concerns. Budget does not fix them.

Compute is not capability. The budget is a resource that can be spent on a problem. Spending it well is a skill. Spending it is not the same as having the answer.