The Token Budget for Code

Code is concise to the eye and expensive to the tokenizer.

A file that occupies a few hundred lines on screen becomes thousands of tokens in the window, and the budget that covers a chapter of English covers a handful of functions. The gap is systematic. It comes from whitespace, punctuation, identifiers, and repetition, each of which charges rent in tokens.

The simplest measurement: characters per token

The naive estimate assumes all text costs the same. Count characters, divide by a constant, and the estimate follows.

naive_char_estimate.py

def naive_tokens(text: str) -> int:
    return len(text) // 4  # common rough rule for English
 
code = "def f(x):\n    return x + 1\n"
prose = "The function adds one.\n"
 
print(f"code:  {naive_tokens(code)} estimated tokens for {len(code)} chars")
print(f"prose: {naive_tokens(prose)} estimated tokens for {len(prose)} chars")
print("The estimate treats both as equally dense. They are not.")

The bottleneck is that code is denser per character. Natural language benefits from frequent multi-character units: common words become single tokens, and the average stays near one token per word. Code behaves differently. The character set is small but combinations are nearly unbounded. Keywords, operators, brackets, and identifiers fragment into smaller pieces, and each piece draws from the same fixed quota.

How a tokenizer sees code

Byte-pair encoding merges frequent adjacent pairs. In prose, the most frequent pairs are letter sequences inside common words. In code, the most frequent pairs include braces, dots, underscores, and indent runs.

code_tokenizer_walk.py

# Illustrative tokenization of one Python line
line = "    return self.value + count"
 
# No third-party tokenizer needed to see the structure:
# leading spaces, "return", "self", "value", "+", "count" are separate units.
units = ["    ", "return", " self", ".", "value", " +", " count"]
print(f"{len(units)} units for {len(line)} chars: {units}")
print("Punctuation and indentation are first-class token consumers.")

The result is a density penalty: the more precise the notation, the higher the price per line. Three components dominate the bill.

Whitespace is not free

Indentation encodes structure. In Python, a tab or a stretch of spaces decides block membership. In every language, newlines and leading spaces mark depth, alignment, and separation.

whitespace_cost.py

def whitespace_ratio(code: str) -> float:
    if not code:
        return 0.0
    ws = sum(1 for ch in code if ch in " \t\n")
    return ws / len(code)
 
depths = [
    "x = 1\n",
    "if x:\n    y = 2\n",
    "def f(x):\n    if x:\n        return x\n",
]
for snippet in depths:
    print(f"{whitespace_ratio(snippet):.0%} whitespace in {repr(snippet[:24])}")

Byte-pair tokenizers do merge repeating whitespace into longer units, but the merge remains partial. Deep nesting repeats the same indent cost at every level, and mixed tabs and spaces fragment further. Whitespace carries meaning, so the tokenizer preserves it. Preservation costs tokens, and the cost scales with formatting discipline.

Bottleneck 1: identifiers repeat without compression

Code reuses names constantly. A project-specific identifier splits into subwords at every occurrence, and the split repeats across files.

identifier_cost.py

# A long, domain-specific name splits into many fragments
identifier = "validate_international_account_record"
 
# At 4 chars/token (rough), the name costs ~10 tokens per occurrence.
# Ten files that each mention it twice pay 200 tokens for one concept.
occurrences = 20
print(f"{len(identifier)}-char identifier, ~{len(identifier) // 4} tokens each")
print(f"{occurrences} occurrences ≈ {occurrences * (len(identifier) // 4)} tokens for one name")

Repetition helps fluency and hurts budget. The model sees the same shape over and over, and the window fills with echoes of structure rather than substance. A prose passage varies its words; code repeats its identifiers, operators, and delimiters.

Bottleneck 2: punctuation charges per occurrence

Punctuation behaves like identifiers. Braces, brackets, dots, underscores, and arrows recur in fixed patterns, and each recurrence pays its own token fare.

punctuation_cost.py

def punctuation_ratio(code: str) -> float:
    if not code:
        return 0.0
    punct = sum(1 for ch in code if ch in "(){}[]<>=+-*/%!&|:;,.?@#")
    return punct / len(code)
 
lines = [
    "value = compute(items[0], key=lambda x: x.field)",
    "The function computes items by key.",
]
for line in lines:
    print(f"{punctuation_ratio(line):.0%} punctuation: {line[:40]}")

The punctuation in code is load-bearing syntax, not decoration. It cannot be dropped without changing meaning, so it cannot be compressed away without changing the program.

Fix (a): measure with the production tokenizer before designing

The naive character estimate misleads planning. The fix is to count tokens with the actual tokenizer that will serve the model.

measure_first.py

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("gpt2")
 
def report(path: str, text: str) -> None:
    ids = tok.encode(text)
    lines = text.count("\n") + 1
    print(f"{path}: {len(ids)} tokens, {len(text)} chars, "
          f"{len(ids) / lines:.1f} tokens/line, {len(text) / max(len(ids), 1):.1f} chars/token")
 
report("prose.py", "The function adds one to the input value.\n")
report("code.py", "def f(x):\n    return x + 1\n")

Chars-per-token is the key ratio. English prose commonly lands near four. Dense code lands lower. The ratio is computable on any file in seconds, and it decides whether the file fits the window before any model is called.

Bottleneck 3: file-level context scales linearly, the window does not

A local edit depends on imports, types, tests, and callers in surrounding files. Each additional file competes for the same token quota.

file_level_budget.py

def budget_for_files(files: dict[str, str], tokenizer, window: int) -> dict:
    costs = {name: len(tokenizer.encode(src)) for name, src in files.items()}
    total = sum(costs.values())
    return {
        "per_file_tokens": costs,
        "total": total,
        "fits": total <= window,
        "files_dropped": [n for n in files if costs[n] > window] if total > window else [],
    }
 
files = {
    "main.py": "def main():\n    pass\n" * 50,
    "utils.py": "def helper():\n    pass\n" * 50,
    "tests.py": "def test_x():\n    pass\n" * 50,
}
print(budget_for_files(files, __import__("transformers").AutoTokenizer.from_pretrained("gpt2"), 8_000))

Relevant files push out irrelevant ones, and the ranking itself consumes effort and introduces error. Retrieval, summarization, and chunking appear as budgeting strategies rather than conveniences. Every strategy trades fidelity for coverage.

Fix (b): spend the budget on structure, not surface

Once the cost is measured, the allocation strategy follows. Three techniques cover most cases.

spend_on_structure.py

# 1. Strip comments and blank lines before sending: they carry no semantics
def strip_noncode(code: str) -> str:
    return "\n".join(l for l in code.splitlines() if l.strip() and not l.strip().startswith("#"))
 
# 2. Send signatures and types first, bodies on demand
def signatures_only(code: str) -> list[str]:
    return [l for l in code.splitlines() if l.startswith(("def ", "class "))]
 
# 3. Chunk by function, not by fixed character count
def chunk_by_function(code: str) -> list[str]:
    chunks, current = [], []
    for line in code.splitlines():
        if line.startswith("def ") and current:
            chunks.append("\n".join(current))
            current = []
        current.append(line)
    if current:
        chunks.append("\n".join(current))
    return chunks
 
src = "def f(x):\n    # comment\n    return x\n\ndef g(y):\n    return y\n"
print("stripped:", repr(strip_noncode(src)))
print("signatures:", signatures_only(src))
print("functions:", len(chunk_by_function(src)), "chunks")

Stripping non-code removes tokens that carry no semantics. Signatures-first ordering puts the interface in the window and bodies on demand. Function-level chunking keeps a unit intact instead of splitting a body across two chunks.

chunking_vs_fixed.py

def chunk_fixed(text: str, size: int) -> list[str]:
    return [text[i:i + size] for i in range(0, len(text), size)]
 
def keeps_units_together(chunks: list[str]) -> bool:
    # A body split across chunks fails this check
    return all("return" in c for c in chunks) if chunks else True
 
src = "def f(x):\n    return x\n\ndef g(y):\n    return y\n"
print("fixed chunks:", chunk_fixed(src, 12))
print("function chunks:", chunk_by_function(src))

Fixed-size chunking can split a function body across chunks, breaking the unit a retriever is meant to preserve. Function boundaries keep semantics aligned with chunks.

What this does to code assistants

A code assistant reasons over text that is already fragmented. Attention spends capacity on syntax, spelling, and layout before semantics enters. That pressure creates predictable failure modes: long functions truncate the tail, cross-file calls drop out of view, and edits target the visible portion while the unseen portion drifts.

failure_modes.py

def assistant_risk(file_tokens: int, window: int) -> str:
    if file_tokens <= window * 0.5:
        return "low risk: full file visible"
    if file_tokens <= window:
        return "medium risk: tail truncated under growth"
    return "high risk: cross-file context missing"
 
for file_t, win in [(2_000, 8_000), (6_000, 8_000), (15_000, 8_000)]:
    print(f"file {file_t:,} tokens / window {win:,}: {assistant_risk(file_t, win)}")

The assistant remains reliable inside the window and brittle outside it. The boundary is economic before it is intellectual.

What to measure

Four measurements decide code-context viability.

code_budget_suite.py

def suite(files: dict[str, str], tok) -> dict:
    total_tokens, total_chars = 0, 0
    for name, src in files.items():
        ids = tok.encode(src)
        total_tokens += len(ids)
        total_chars += len(src)
    return {
        "chars_per_token": total_chars / max(total_tokens, 1),
        "tokens_total": total_tokens,
        "whitespace_share": 0.0,  # compute with whitespace_ratio() per repo
    }
 
print("Run per repository, then per language, then per style.")

Chars per token per language: the density number, measured with the production tokenizer.
Whitespace share: the formatting-discipline cost.
Identifier fragmentation: average subwords per unique identifier.
Tokens per file versus window: the fit decision, per file, before any prompt is built.

A checklist for code-context design:

Measure tokens per file with the serving tokenizer before promising file-level context.
Strip comments, docstrings, and blank lines from prompt content.
Send signatures first and bodies on demand.
Chunk by function or class boundary, never by fixed character count.
Rank files by relevance before truncation, not after.
Track cross-file call visibility explicitly; it is the first casualty.

The token budget decides which code exists for the model. That decision shapes every answer that follows.

Code is concise to the eye and expensive to the tokenizer.

The simplest measurement: characters per token

The naive estimate assumes all text costs the same. Count characters, divide by a constant, and the estimate follows.

naive_char_estimate.py

def naive_tokens(text: str) -> int:
    return len(text) // 4  # common rough rule for English
 
code = "def f(x):\n    return x + 1\n"
prose = "The function adds one.\n"
 
print(f"code:  {naive_tokens(code)} estimated tokens for {len(code)} chars")
print(f"prose: {naive_tokens(prose)} estimated tokens for {len(prose)} chars")
print("The estimate treats both as equally dense. They are not.")

How a tokenizer sees code

code_tokenizer_walk.py

# Illustrative tokenization of one Python line
line = "    return self.value + count"
 
# No third-party tokenizer needed to see the structure:
# leading spaces, "return", "self", "value", "+", "count" are separate units.
units = ["    ", "return", " self", ".", "value", " +", " count"]
print(f"{len(units)} units for {len(line)} chars: {units}")
print("Punctuation and indentation are first-class token consumers.")

The result is a density penalty: the more precise the notation, the higher the price per line. Three components dominate the bill.

Whitespace is not free

Indentation encodes structure. In Python, a tab or a stretch of spaces decides block membership. In every language, newlines and leading spaces mark depth, alignment, and separation.

whitespace_cost.py

def whitespace_ratio(code: str) -> float:
    if not code:
        return 0.0
    ws = sum(1 for ch in code if ch in " \t\n")
    return ws / len(code)
 
depths = [
    "x = 1\n",
    "if x:\n    y = 2\n",
    "def f(x):\n    if x:\n        return x\n",
]
for snippet in depths:
    print(f"{whitespace_ratio(snippet):.0%} whitespace in {repr(snippet[:24])}")

Bottleneck 1: identifiers repeat without compression

Code reuses names constantly. A project-specific identifier splits into subwords at every occurrence, and the split repeats across files.

identifier_cost.py

# A long, domain-specific name splits into many fragments
identifier = "validate_international_account_record"
 
# At 4 chars/token (rough), the name costs ~10 tokens per occurrence.
# Ten files that each mention it twice pay 200 tokens for one concept.
occurrences = 20
print(f"{len(identifier)}-char identifier, ~{len(identifier) // 4} tokens each")
print(f"{occurrences} occurrences ≈ {occurrences * (len(identifier) // 4)} tokens for one name")

Bottleneck 2: punctuation charges per occurrence

Punctuation behaves like identifiers. Braces, brackets, dots, underscores, and arrows recur in fixed patterns, and each recurrence pays its own token fare.

punctuation_cost.py

def punctuation_ratio(code: str) -> float:
    if not code:
        return 0.0
    punct = sum(1 for ch in code if ch in "(){}[]<>=+-*/%!&|:;,.?@#")
    return punct / len(code)
 
lines = [
    "value = compute(items[0], key=lambda x: x.field)",
    "The function computes items by key.",
]
for line in lines:
    print(f"{punctuation_ratio(line):.0%} punctuation: {line[:40]}")

The punctuation in code is load-bearing syntax, not decoration. It cannot be dropped without changing meaning, so it cannot be compressed away without changing the program.

Fix (a): measure with the production tokenizer before designing

The naive character estimate misleads planning. The fix is to count tokens with the actual tokenizer that will serve the model.

measure_first.py

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("gpt2")
 
def report(path: str, text: str) -> None:
    ids = tok.encode(text)
    lines = text.count("\n") + 1
    print(f"{path}: {len(ids)} tokens, {len(text)} chars, "
          f"{len(ids) / lines:.1f} tokens/line, {len(text) / max(len(ids), 1):.1f} chars/token")
 
report("prose.py", "The function adds one to the input value.\n")
report("code.py", "def f(x):\n    return x + 1\n")

Bottleneck 3: file-level context scales linearly, the window does not

A local edit depends on imports, types, tests, and callers in surrounding files. Each additional file competes for the same token quota.

file_level_budget.py

def budget_for_files(files: dict[str, str], tokenizer, window: int) -> dict:
    costs = {name: len(tokenizer.encode(src)) for name, src in files.items()}
    total = sum(costs.values())
    return {
        "per_file_tokens": costs,
        "total": total,
        "fits": total <= window,
        "files_dropped": [n for n in files if costs[n] > window] if total > window else [],
    }
 
files = {
    "main.py": "def main():\n    pass\n" * 50,
    "utils.py": "def helper():\n    pass\n" * 50,
    "tests.py": "def test_x():\n    pass\n" * 50,
}
print(budget_for_files(files, __import__("transformers").AutoTokenizer.from_pretrained("gpt2"), 8_000))

Fix (b): spend the budget on structure, not surface

Once the cost is measured, the allocation strategy follows. Three techniques cover most cases.

spend_on_structure.py

# 1. Strip comments and blank lines before sending: they carry no semantics
def strip_noncode(code: str) -> str:
    return "\n".join(l for l in code.splitlines() if l.strip() and not l.strip().startswith("#"))
 
# 2. Send signatures and types first, bodies on demand
def signatures_only(code: str) -> list[str]:
    return [l for l in code.splitlines() if l.startswith(("def ", "class "))]
 
# 3. Chunk by function, not by fixed character count
def chunk_by_function(code: str) -> list[str]:
    chunks, current = [], []
    for line in code.splitlines():
        if line.startswith("def ") and current:
            chunks.append("\n".join(current))
            current = []
        current.append(line)
    if current:
        chunks.append("\n".join(current))
    return chunks
 
src = "def f(x):\n    # comment\n    return x\n\ndef g(y):\n    return y\n"
print("stripped:", repr(strip_noncode(src)))
print("signatures:", signatures_only(src))
print("functions:", len(chunk_by_function(src)), "chunks")

chunking_vs_fixed.py

def chunk_fixed(text: str, size: int) -> list[str]:
    return [text[i:i + size] for i in range(0, len(text), size)]
 
def keeps_units_together(chunks: list[str]) -> bool:
    # A body split across chunks fails this check
    return all("return" in c for c in chunks) if chunks else True
 
src = "def f(x):\n    return x\n\ndef g(y):\n    return y\n"
print("fixed chunks:", chunk_fixed(src, 12))
print("function chunks:", chunk_by_function(src))

Fixed-size chunking can split a function body across chunks, breaking the unit a retriever is meant to preserve. Function boundaries keep semantics aligned with chunks.

What this does to code assistants

failure_modes.py

def assistant_risk(file_tokens: int, window: int) -> str:
    if file_tokens <= window * 0.5:
        return "low risk: full file visible"
    if file_tokens <= window:
        return "medium risk: tail truncated under growth"
    return "high risk: cross-file context missing"
 
for file_t, win in [(2_000, 8_000), (6_000, 8_000), (15_000, 8_000)]:
    print(f"file {file_t:,} tokens / window {win:,}: {assistant_risk(file_t, win)}")

The assistant remains reliable inside the window and brittle outside it. The boundary is economic before it is intellectual.

What to measure

Four measurements decide code-context viability.

code_budget_suite.py

def suite(files: dict[str, str], tok) -> dict:
    total_tokens, total_chars = 0, 0
    for name, src in files.items():
        ids = tok.encode(src)
        total_tokens += len(ids)
        total_chars += len(src)
    return {
        "chars_per_token": total_chars / max(total_tokens, 1),
        "tokens_total": total_tokens,
        "whitespace_share": 0.0,  # compute with whitespace_ratio() per repo
    }
 
print("Run per repository, then per language, then per style.")

Chars per token per language: the density number, measured with the production tokenizer.
Whitespace share: the formatting-discipline cost.
Identifier fragmentation: average subwords per unique identifier.
Tokens per file versus window: the fit decision, per file, before any prompt is built.

A checklist for code-context design:

Measure tokens per file with the serving tokenizer before promising file-level context.
Strip comments, docstrings, and blank lines from prompt content.
Send signatures first and bodies on demand.
Chunk by function or class boundary, never by fixed character count.
Rank files by relevance before truncation, not after.
Track cross-file call visibility explicitly; it is the first casualty.

The token budget decides which code exists for the model. That decision shapes every answer that follows.

The simplest measurement: characters per token#

How a tokenizer sees code#

Whitespace is not free#

Bottleneck 1: identifiers repeat without compression#

Bottleneck 2: punctuation charges per occurrence#

Fix (a): measure with the production tokenizer before designing#

Bottleneck 3: file-level context scales linearly, the window does not#

Fix (b): spend the budget on structure, not surface#

What this does to code assistants#

What to measure#

The simplest measurement: characters per token#

How a tokenizer sees code#

Whitespace is not free#

Bottleneck 1: identifiers repeat without compression#

Bottleneck 2: punctuation charges per occurrence#

Fix (a): measure with the production tokenizer before designing#

Bottleneck 3: file-level context scales linearly, the window does not#

Fix (b): spend the budget on structure, not surface#

What this does to code assistants#

What to measure#

The simplest measurement: characters per token

How a tokenizer sees code

Whitespace is not free

Bottleneck 1: identifiers repeat without compression

Bottleneck 2: punctuation charges per occurrence

Fix (a): measure with the production tokenizer before designing

Bottleneck 3: file-level context scales linearly, the window does not

Fix (b): spend the budget on structure, not surface

What this does to code assistants

What to measure

The simplest measurement: characters per token

How a tokenizer sees code

Whitespace is not free

Bottleneck 1: identifiers repeat without compression

Bottleneck 2: punctuation charges per occurrence

Fix (a): measure with the production tokenizer before designing

Bottleneck 3: file-level context scales linearly, the window does not

Fix (b): spend the budget on structure, not surface

What this does to code assistants

What to measure