The Context Window Is a Budget

A context window is a budget constraint. It is the number of tokens a model can see at once, and everything the model reasons about must fit inside it. Long-context models have enlarged the budget. They have not abolished it, and the arithmetic still determines what tasks are possible.

The simplest model: a fixed token allowance#

The naive picture treats the window as a container. Fill it with instructions, documents, and history, and the model reads the whole container.

def naive_pack(instructions: str, documents: list[str],
               history: list[str], tokenizer, window: int) -> bool:
    total = (
        len(tokenizer.encode(instructions))
        + sum(len(tokenizer.encode(d)) for d in documents)
        + sum(len(tokenizer.encode(h)) for h in history)
    )
    return total <= window
 
# The check passes or fails. There is no partial credit.

The bottleneck is that the boundary is binary. A passage that fits by one token is visible; a passage that exceeds the limit by one token is dropped entirely. Two refinements follow: allocate the budget deliberately instead of treating it as one pool, and account for output tokens, which share the same budget as input.

The arithmetic: tokens are the currency#

Let $C$ be the context window in tokens. A model with $C = 128{,}000$ can see roughly 128,000 tokens of input, output, and everything in between. The relevant comparison is not between models. It is between the budget and the task.

C = 128_000
 
def tokens_for_words(words: int, words_per_token: float = 0.75) -> int:
    # Rule of thumb for English: ~0.75 words per token
    return int(words / words_per_token)
 
novel_words = 100_000
print(f"novel: {tokens_for_words(novel_words):,} tokens vs budget {C:,}")
 
# A decade of documents at 10M words
decade_tokens = tokens_for_words(10_000_000)
print(f"decade archive: {decade_tokens:,} tokens (ratio {decade_tokens / C:.0f}x budget)")

A novel is around 100,000 words. In English, that is about 130,000 tokens. It fits in a 128,000-token window, barely, with room for a few instructions. A codebase is larger. A year's worth of documents is much larger. A video, represented as frames, is larger still.

The budget is not the same for every language. Urdu in Arabic script and Nepali in Devanagari cost more tokens per word than English. The same window holds fewer words. The budget is in tokens, and tokens are not words.

def effective_words(window: int, words_per_token: float) -> int:
    return int(window * words_per_token)
 
for lang, wpt in [("en", 0.75), ("ne", 0.40), ("ur", 0.45)]:
    print(f"{lang}: window holds ≈ {effective_words(C, wpt):,} words")

Attention cost is also quadratic in sequence length — $O(C^2)$. Doubling the window quadruples the compute. Budget and price rise at different rates, which is why enlarging the window is the last strategy, not the first.

Bottleneck 1: one pool wastes the budget#

The naive container spends tokens in input order. Instructions, documents, and history compete with no priority, and output space is forgotten until generation truncates mid-sentence.

def allocate(window: int, reserves: dict[str, int]) -> dict[str, int]:
    total = sum(reserves.values())
    assert total <= window, f"over-committed: {total} > {window}"
    return reserves
 
budget = allocate(128_000, {
    "instructions": 2_000,
    "retrieved_docs": 100_000,
    "conversation": 20_000,
    "output": 6_000,
})
for part, cost in budget.items():
    print(f"{part}: {cost / 1280:.0f}% of window")

The fix is explicit allocation: reserve output space first, reserve instructions second, and treat the remainder as the retrieval budget. Blocks that exceed their allocation are evicted by policy, not by accident.

Three strategies for the same budget#

Make the model bigger-context. Increase $C$. This works until the task outgrows it. The price is quadratic compute and memory, plus a per-request cost that grows with every token actually sent.

def attention_cost_ratio(old_len: int, new_len: int) -> float:
    return (new_len / old_len) ** 2
 
print(f"doubling the window costs {attention_cost_ratio(64_000, 128_000):.0f}x attention compute")
print("The budget grows. Its price grows faster than its size.")

Retrieve, do not fit. Retrieval-augmented generation keeps a large corpus outside the window and brings in the relevant pieces. The budget is spent on the retrieved context, not the whole corpus.

def retrieval_prompt(query: str, chunks: list[str], tokenizer, window: int) -> str:
    ranked = sorted(chunks, key=lambda c: len(c))  # placeholder ranking
    selected, used = [], 0
    for chunk in ranked:
        cost = len(tokenizer.encode(chunk))
        if used + cost > window * 0.8:
            break
        selected.append(chunk)
        used += cost
    return f"Question: {query}\n\n" + "\n\n".join(selected)
 
# When retrieval fails, the model cannot see what it needs,
# no matter how large the window is. The bottleneck moved
# from window size to retriever accuracy.

Compress. Summarization, long-context compression, and hierarchical memory reduce the text presented to the model. The compression is lossy by design. For a question about a specific detail, a summary is not a substitute for the source.

def compression_ratio(original_tokens: int, summary_tokens: int) -> float:
    return original_tokens / max(summary_tokens, 1)
 
print(f"130k-token novel -> 1k-token summary: {compression_ratio(130_000, 1_000):.0f}x compression")
print("A 130:1 ratio preserves gist and loses evidence.")

Each approach moves the bottleneck. None removes it.

Attention is not memory#

A large context window does not mean the model uses everything in it equally. Attention distributes weight, and the distribution is not uniform. Empirically, information in the middle of a long context is used less than information at the start and the end. The effect appears across architectures.

def needle_accuracy(correct_at_position: list[float]) -> float:
    # positions 0..1 (start -> end); compare early, middle, late
    n = len(correct_at_position)
    early = sum(correct_at_position[: n // 4]) / max(n // 4, 1)
    middle = sum(correct_at_position[n // 4 : 3 * n // 4]) / max(n // 2, 1)
    late = sum(correct_at_position[3 * n // 4 :]) / max(n // 4, 1)
    return early, middle, late
 
# Illustrative pattern seen in needle-in-haystack evaluations:
print("early 0.95 | middle 0.70 | late 0.94")
print("Middle recall dips even though the text is inside the window.")

"Needle in a haystack" evaluations measure whether a model can find a fact in a long document. They do not measure whether the model reasons correctly about the whole document. The haystack problem is retrieval. The reasoning problem is harder.

Effective context is smaller than advertised. A window of 128,000 tokens with poor middle-attention is not 128,000 tokens of usable input. The number that matters for a task is the length at which the model still answers correctly, not the length it accepts.

def usable_window(advertised: int, middle_recall: float) -> int:
    # Discount the middle half by its observed recall loss
    early_late = advertised // 2
    middle_usable = int((advertised // 2) * middle_recall)
    return early_late + middle_usable
 
print(f"advertised 128,000 -> usable ≈ {usable_window(128_000, 0.70):,} at 70% middle recall")

The budget determines the task class#

A fixed budget implies a fixed class of tasks.

TASKS = {
    "small": ["question answering", "short-form generation", "classification"],
    "medium": ["document understanding", "multi-turn conversation", "code editing with context"],
    "large": ["long-document analysis", "whole-file repository work", "extended reasoning"],
    "beyond": ["anything requiring the whole corpus in view"],
}
for bucket, tasks in TASKS.items():
    print(bucket, "->", ", ".join(tasks))

Retrieval extends the class without enlarging the budget, at the cost of dependence on the retriever. Compression extends it at the cost of information loss. Enlarging the window extends it at the cost of compute. The tradeoffs are not theoretical. They decide which applications are possible at which price.

strategies = {
    "enlarge window": "pay quadratic compute per request",
    "retrieve": "pay index + retriever error",
    "compress": "pay information loss",
    "hierarchical memory": "pay complexity + stale summaries",
}
for name, cost in strategies.items():
    print(f"{name}: {cost}")

What to measure#

Four measurements turn the budget from a slogan into a number.

def budget_report(window: int, prompt_tokens: int, output_reserve: int) -> dict:
    used = prompt_tokens + output_reserve
    return {
        "headroom": window - used,
        "headroom_pct": (window - used) / window,
        "fits": used <= window,
    }
 
print(budget_report(128_000, prompt_tokens=100_000, output_reserve=6_000))

• Headroom: window minus prompt minus output reserve, in tokens and percent.
• Effective window: accuracy as a function of position, early versus middle versus late.
• Retriever recall: fraction of answer-bearing chunks actually retrieved.
• Cost per answered question: tokens per request multiplied by request volume.

A checklist for a context-budgeted system:

• Count tokens with the production tokenizer, not characters or bytes.
• Reserve output space before filling the prompt.
• Allocate sections explicitly: instructions, retrieval, history.
• Measure middle-context recall on real documents.
• Measure retriever recall separately from generator quality.
• Verify the language adjustment: lower-fertility languages consume more budget.

Context windows will keep growing. Retrieval quality will keep improving. Compression will keep getting smarter. The budget will not go away, because the budget is not an implementation detail. It is the constraint that defines what the model can do in one pass. The right question is not how large the window is. It is how the task's information requirement compares to the budget, and which strategy — fit, retrieve, or compress — is the best fit for the information that matters.