Hallucination in Medical VLMs

A medical vision-language model that hallucinates does not hedge. It answers in the same register as a correct answer, with the same confidence, about a finding that is not in the image. The failure is not dishonesty. It is a mismatch between two things the model was trained to produce: plausible text and grounded text. Plausibility is easy to learn. Grounding is expensive.

The simplest useful version of the problem is a single prompt over a single image. Everything else — encoders, decoders, calibration tricks, verifiers — follows from where that minimal version breaks.

The smallest useful model#

A vision-language model maps an image $x$ and a question $q$ to a text answer $y$:

The factorization looks innocent. It is the source of the failure. Text generation is autoregressive: each token is sampled conditioned on all previous tokens plus the visual tokens. If the visual conditioning is weak, the text prior — what a radiology report usually says next — dominates.

A minimal pipeline makes the weakness explicit:

import torch
 
def naive_answer(model, image, question):
    # model.encode: image -> visual tokens
    # model.lm: visual tokens + question -> next token distribution
    visual = model.encode(image)          # shape: [1, K, D]
    prompt = model.format(question)
    tokens = model.tokenize(prompt)
 
    output_ids = model.generate(
        visual_tokens=visual,
        input_ids=tokens,
        max_new_tokens=256,
        do_sample=False,                  # greedy decoding
    )
    return model.detokenize(output_ids)

Four failure shapes appear immediately from this structure:

• Presence. A finding is reported that no radiologist would mark. "There is a 4 mm nodule in the right upper lobe." There is no nodule. The prior completes "There is a" with a common finding.
• Absence. A real finding is missed and the study is called normal. The mirror error, equally dangerous.
• Location. Something real is placed in the wrong place, wrong lung, or wrong image in the series. The token sequence is correct; the pointer is not.
• Count and size. Two lesions are reported when there is one, or a size is estimated from a magnification the image does not contain.

These are not rare edge cases. They are the natural failure modes of a model that has learned the shape of a radiology report more thoroughly than the geometry of the scan.

What the hardware of the model actually is#

Three components decide whether a hallucination is produced, and each has a bottleneck.

The language prior. A VLM is an LLM with an image encoder attached. The LLM has seen millions of reports. Given a chest X-ray and the prefix "There is a", the most probable continuations are phrases that occur in reports. Visual tokens nudge the distribution; they do not override it. When the image is ambiguous, the prior wins. The strength of that prior grows with model size and with instruction tuning that rewards a single definite answer.

The image encoder. Many VLMs use a vision tower trained for general image–text alignment. A chest X-ray has fine-grained structure — a 4 mm nodule, a thin pneumothorax line — that a general-purpose encoder has never been asked to preserve. Patch-based encoders downsample aggressively. A 224×224 input with 14×14 patches yields 256 tokens; each token summarizes a region the size of the finding. The information is lost before the LLM sees it. No decoding trick can recover detail that was never encoded.

The training objective. Most instruction-tuning data contains a question with a definite answer. The gradient rewards a short, certain output. "I cannot tell from this image" is rarely the target. The model therefore learns that abstention is an error rather than a valid response. Calibration is not trained; confidence is.

A useful decomposition: hallucination happens when the prior is strong, the visual evidence is weak, and the objective penalizes abstention. Fixing only one of the three leaves the other two intact.

Bottleneck one: the encoder is too coarse#

The first fix is a domain-specific visual encoder trained on radiology images with contrastive or masked objectives. The improvement is real but partial. A better encoder feeding a strong language prior still hallucinates on ambiguous cases, because ambiguous means the visual signal genuinely underdetermines the answer.

The honest version of the fix is higher-resolution encoding with explicit region structure, not just a larger tower:

import torch
from PIL import Image
 
def encode_at_resolution(encoder, pil_image, tile=448):
    # Naive: resize once to 224, lose the nodule.
    naive = encoder.encode(pil_image.resize((224, 224)))
 
    # Improved: tile the image, encode tiles + global view.
    w, h = pil_image.size
    tiles = []
    for y in range(0, h, tile):
        for x in range(0, w, tile):
            crop = pil_image.crop((x, y, x + tile, y + tile))
            tiles.append(crop.resize((224, 224)))
    tile_tokens = encoder.encode_batch(tiles)     # [n_tiles, K, D]
    global_tokens = encoder.encode(pil_image.resize((448, 448)))
    visual = torch.cat([global_tokens, tile_tokens.reshape(-1, tile_tokens.shape[1], tile_tokens.shape[2])], dim=1)
    return visual

Resolution alone does not fix grounding. A model can now see the nodule and still report one that is absent. That transition — from losing information to using it incorrectly — defines the next bottleneck.

Bottleneck two: text is not attached to a location#

Image-grounded training attaches text to locations. Region-level annotations such as "the opacity in the left lower zone" force the model to bind a phrase to coordinates before it can use the phrase in a sentence. This is expensive to collect and is the correct direction, because it targets the mechanism rather than the symptom.

Constrained decoding is the cheaper approximation. The model must emit a pointing coordinate alongside the text. A model that must point will sometimes point wrongly, but the pointing is checkable, and a checked wrong point is not a hallucination — it is a localizable error.

import json
import torch
 
def grounded_answer(model, image, question):
    visual = model.encode(image)
    prompt = (
        "Find the region described. Respond ONLY as JSON: "
        "{" '"finding"' ': <str>, " + '"x_min"' ": <int>, " + '"y_min"' ": <int>, "
        + '"x_max"' ": <int>, " + '"y_max"' ": <int>, " + '"visible"' ": <bool>} "
        f"Question: {question}"
    )
    raw = model.generate_from(visual, prompt, max_new_tokens=128)
    parsed = json.loads(raw)
    if not parsed.get("visible", True):
        return {"finding": "not_visible", "bbox": None, "abstain": True}
    return parsed

The bottleneck this exposes is calibration. A model trained always to produce a bounding box will produce a bounding box for an absent finding. The visible flag only helps if abstention is rewarded during training. Without that reward, constrained decoding moves the hallucination from the sentence to the rectangle.

Bottleneck three: the answer is never verified#

Self-consistency checks — asking the same question under paraphrase, or asking the model to describe the image before answering — produce a weak detector. If paraphrased answers disagree, that is a signal. It is not a fix, because the same prior inflates both answers.

Separate verification uses a different inductive bias. A second model, or a classical algorithm, checks the first model's finding against the image. The verifier does not share the generator's prior, so it does not share its hallucinations.

import torch
 
def verify_finding(verifier, image, finding_text, bbox):
    # verifier: independent detector / second VLM / classical pipeline
    # Returns a score in [0, 1] for "is this finding present in this region?"
    crop = crop_to_bbox(image, bbox) if bbox else image
    score = verifier.score(crop, finding_text)
    return {"score": score, "confirmed": score > 0.5}
 
def report_with_verification(model, verifier, image, question, threshold=0.5):
    answer = grounded_answer(model, image, question)
    if answer.get("abstain"):
        return answer
    check = verify_finding(verifier, image, answer["finding"], answer.get("bbox"))
    if not check["confirmed"]:
        return {"finding": answer["finding"], "verified": False, "verdict": "unconfirmed", "score": check["score"]}
    return {"finding": answer["finding"], "verified": True, "bbox": answer.get("bbox"), "score": check["score"]}

Verification is unglamorous and outperforms most single-model tricks. The cost is latency and a second failure surface. The verifier can be wrong, especially on small findings. That cost is acceptable when the alternative is an unverified claim entering a report.

Bottleneck four: the benchmark cannot see the failure#

A benchmark with one reference answer per image cannot distinguish a grounded model from a confident one. Both produce the same string. The difference only appears when the image changes and the answer should change with it. Evaluation therefore has to be interventional.

import torch
 
def consistency_score(model, image, question):
    # The same question, the same image, perturbed. A grounded model
    # tracks the image; a prior-driven model repeats its answer.
    base = model.generate(model.encode(image), question)
    mirrored = model.generate(model.encode(image.transpose()), question)
    noised = model.generate(model.encode(add_noise(image)), question)
    return {
        "base": base,
        "tracks_mirror": mirrored != base,   # should change if laterality is asked
        "tracks_noise": noised != base,      # should stay if noise is irrelevant
    }
 
def hallucination_rate(model, cases, threshold=0.5):
    # cases: list of (image, question, has_finding, bbox)
    false_positives = 0
    false_negatives = 0
    for image, question, has_finding, bbox in cases:
        out = grounded_answer(model, image, question)
        predicted_present = out.get("bbox") is not None
        if predicted_present and not has_finding:
            false_positives += 1
        if not predicted_present and has_finding:
            false_negatives += 1
    n = len(cases)
    return {"fp_rate": false_positives / n, "fn_rate": false_positives and false_positives / n or false_negatives / n}

Negative cases are essential. A presence-heavy benchmark rewards a model that always says "present." Human review of failures is also essential; automated metrics on open-ended medical text are weak proxies. Read the outputs.

What to measure#

A VLM is not a radiologist. It has no liability, no access to history, and no understanding of what it would mean to be wrong. Used as a draft, a second reader, or a triage signal — with a human required to confirm — the error rate can be acceptable. Used as an autonomous reporter, it is not.

A measurement checklist:

• Report false-positive and false-negative rates separately, on a case mix that includes true negatives. A single accuracy number hides the dangerous direction.
• Run interventional probes: remove the lesion, mirror the scan, add noise, and record whether the answer changes in the right direction.
• Report pointing accuracy, not just text accuracy. A correct sentence with a wrong box is a hallucination with coordinates.
• Report abstention behavior on uninterpretable inputs. A model that never abstains has not been calibrated.
• Log verifier agreement separately from generator confidence. High confidence with low verification is the hallucination signature.

The right question is not whether the model hallucinates. In this setting, it does. The right question is what mechanism catches the hallucination, and who is responsible for operating it. Until that answer is precise, the safe deployment is the one with a human in the loop and a verifier on the side.