How Much Does an Image Cost? Measuring Image Token Cost on a 27B Multimodal Model

You capped the pixel budget on your vision model to save tokens, your latency dropped, and three weeks later someone noticed the document-QA accuracy had quietly fallen off a cliff. This is the most common self-inflicted wound in multimodal serving, and it happens because teams treat resolution as a cost slider when it is a correctness knob. The image token cost you pay buys legibility: below a threshold the model literally cannot read the small text in a form or a chart, and no amount of prompting recovers it.

The analytical version of this argument lives in Multimodal LLMs in Production: What Native Vision Actually Costs, which makes the case that resolution decides accuracy and not just price. This tutorial is the measurement behind that claim. We are going to fix a model, sweep image resolution across several tiers, and report four numbers at each tier: per-image token count, prefill time, KV cache growth, and DocVQA accuracy. Everything runs on a single 24GB card, because the model we use, Qwen3.8-27B, ships open weights (Apache 2.0) and fits one consumer GPU in a 4-bit build. At the end you will have a budget table and a resolution-selection rule for your own workload, and you will have produced a concrete before/after accuracy number rather than an eyeball impression.

What one image actually costs in tokens

Qwen's vision models do not send an image as "one token." They tile it. The processor divides the image into 16-pixel patches, then merges each 2x2 block of patches into a single visual token, so one visual token covers a 32x32 pixel region. The count follows directly:

visual_tokens ≈ resized_pixels / (32 * 32)  =  resized_pixels / 1024

Check this against the model you actually load, because it changed between generations. Qwen2-VL and Qwen2.5-VL used 14-pixel patches (28x28 per token, pixels / 784), and a lot of copy-pasted serving code still carries that constant. Qwen3-VL and Qwen3.8-27B ship patch_size: 16 and merge_size: 2 in their preprocessor_config.json. Reuse the old 784 unit on the new model and a cap you meant as N tokens actually buys about 0.77N, while every token estimate you compute runs about 30% high.

The processor exposes two bounds, size["shortest_edge"] and size["longest_edge"], that clamp the resized area before tiling. Despite the names, both are pixel areas, not edge lengths (older Qwen code calls them min_pixels and max_pixels). Both are expressed in the same 32x32 unit, so a longest_edge of 1024 * 32 * 32 means "spend at most 1,024 visual tokens on this image." That makes the resolution decision a token-budget decision in disguise, and the mapping is exact rather than empirical:

The default upper bound is much higher (longest_edge: 16777216, which is 16,384 tokens, roughly a full high-resolution document page), which is exactly why an uncapped multimodal request can blow through a context window that felt generous when you were thinking in text. A single dense page at the default cap can cost more tokens than the entire text prompt around it. That displaced context is real: every visual token you spend is a token you cannot spend on retrieved passages or conversation history, the same scarcity the effective context length discussion warns about from the text side.

The point of this tutorial is that the right token budget is not the maximum and not the minimum. It is the knee of the accuracy curve, and you have to measure to find it.

Setup

One 24GB GPU is enough. The only hard requirement is a transformers release new enough to load Qwen3.8-27B, which shipped 2026-08-14 with a config.json written by 5.8, so treat 5.8 as the floor (5.x itself needs PyTorch 2.6 or newer). Record the exact versions pip resolves, because they belong next to every latency number you report. The first load downloads the BF16 checkpoint (about 56GB) and quantizes it to 4-bit in memory.

python -m venv .venv && source .venv/bin/activate
pip install "torch>=2.6" \
            torchvision \
            "transformers>=5.8" \
            "bitsandbytes>=0.45" \
            "accelerate>=1.1" \
            "datasets>=3.0" \
            "python-Levenshtein>=0.25"

Load the model and processor once. We load the language model in 4-bit so the whole thing sits under 24GB, and keep the small vision tower (about 0.4B parameters) in BF16, because low-bit numerics hit exactly the fine glyph detail this tutorial measures.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig

MODEL_ID = "Qwen/Qwen3.8-27B"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    llm_int8_skip_modules=["model.visual", "lm_head"],  # vision tower stays BF16
)

model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    quantization_config=bnb,
    device_map="auto",
    dtype=torch.bfloat16,
)
model.eval()

# One processor for every tier; the token budget is passed per call.
processor = AutoProcessor.from_pretrained(MODEL_ID)
ip = processor.image_processor
PX_PER_TOKEN = (ip.patch_size * ip.merge_size) ** 2  # 16 * 2 = 32, so 1024 px per token
print(ip.patch_size, ip.merge_size, ip.size)  # expect 16, 2, shortest_edge=65536, longest_edge=16777216

Print that line before you trust any arithmetic in this tutorial. If it says 14, you loaded an older Qwen-VL generation and the unit is 784, not 1024.

For the data, use DocVQA, the standard document visual-question-answering benchmark, scored with ANLS (Average Normalized Levenshtein Similarity). DocVQA is the right instrument here because it is dominated by small printed text, so it is maximally sensitive to resolution: it is exactly the task that punishes a too-low token budget. For reference points, the Qwen technical reports give 96.4 ANLS for Qwen2.5-VL-72B and 97.1 for Qwen3-VL-235B-A22B-Instruct on the DocVQA test set. Qwen3.8-27B's model card does not report DocVQA, and a 4-bit run on a validation slice is not the same measurement anyway, so treat the mid-90s as the ceiling this model family reaches at full resolution and your own largest-budget run as the baseline that counts. The question this tutorial answers is how far below that baseline you fall as you shrink the budget.

from datasets import load_dataset

# The validation split (5,349 questions) ships gold answers; the test split does not.
# lmms-lab/DocVQA now redirects to lmms-lab-encoder/DocVQA; either name loads.
ds = load_dataset("lmms-lab-encoder/DocVQA", "DocVQA", split="validation")
ds = ds.select(range(200))  # raise to the full split for a publishable number

Measure tokens, prefill, and KV at each tier

The core harness builds the chat input at a given longest_edge budget, counts the visual tokens the processor actually emitted, times the answer, and computes the KV bytes the prompt will hold. It reads the KV geometry from the model config rather than hardcoding it, so the arithmetic stays correct if you swap models. Two settings in it are load-bearing for accuracy. Thinking is switched off (enable_thinking=False): Qwen3.8-27B thinks by default at its highest reasoning_effort, and even at low the thinking block would eat a 32-token answer budget before any answer appears. And the question carries the standard DocVQA instruction to answer with a single word or phrase, because ANLS scores a full sentence around the right answer as a miss.

import time

def kv_bytes_per_token(cfg):
    # The model config is composite (vision + text); KV lives in the text tower.
    cfg = cfg.get_text_config()
    # Hybrid stack: only full-attention layers keep a per-token KV cache.
    # The linear-attention layers hold a fixed-size state that does not grow.
    layer_types = getattr(cfg, "layer_types", None)
    n_layers = (sum(t == "full_attention" for t in layer_types)
                if layer_types else cfg.num_hidden_layers)
    # GQA-aware: KV is sized by num_key_value_heads, not num_attention_heads.
    head_dim = getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)
    n_kv = getattr(cfg, "num_key_value_heads", cfg.num_attention_heads)
    bytes_per_elem = 2  # bf16 KV cache
    return 2 * n_layers * n_kv * head_dim * bytes_per_elem  # K and V

def budget_size(max_tokens):
    # Pixel-area bounds in the processor's own unit (32x32 px per visual token).
    # shortest_edge is the floor (4 tokens); longest_edge is the budget under test.
    return {"shortest_edge": 4 * PX_PER_TOKEN, "longest_edge": max_tokens * PX_PER_TOKEN}

def measure(example, max_tokens):
    messages = [{
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": example["question"]
             + "\nAnswer the question using a single word or phrase."},
        ],
    }]
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True,
        enable_thinking=False,  # extractive QA: no thinking block in the answer
    )
    inputs = processor(
        text=[text], images=[example["image"].convert("RGB")],
        images_kwargs={"size": budget_size(max_tokens)},
        padding=True, return_tensors="pt",
    ).to(model.device)

    # Visual tokens actually emitted: product of the grid, divided by the 2x2 merge.
    grid = inputs["image_grid_thw"][0]
    merge = processor.image_processor.merge_size
    visual_tokens = int(grid.prod().item() // (merge * merge))
    total_tokens = int(inputs["input_ids"].shape[1])

    torch.cuda.synchronize()
    t0 = time.perf_counter()
    with torch.inference_mode():
        out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
    torch.cuda.synchronize()
    prefill_ms = (time.perf_counter() - t0) * 1000

    answer = processor.batch_decode(
        out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True,
    )[0].strip()

    kv_mb = total_tokens * kv_bytes_per_token(model.config) / 1e6
    return {
        "max_tokens": max_tokens,
        "visual_tokens": visual_tokens,
        "total_tokens": total_tokens,
        "prefill_ms": round(prefill_ms, 1),
        "kv_mb": round(kv_mb, 1),
        "answer": answer,
    }

Two honest notes on the measurements. The prefill timing above wraps a short generate, so it is time-to-answer for a reply of up to 32 tokens, not a pure prefill microbenchmark; it is the number a user actually waits, which is the one worth reporting. And every latency figure is meaningless without its context, so record the fields local-inference benchmarks always omit: quantization format (nf4 here), backend (transformers + bitsandbytes, plus whether the linear-attention kernels loaded or fell back to the slower PyTorch path), and driver or GPU. A latency number without those is not reproducible.

The KV column, by contrast, you can check by hand before running anything. Qwen3.8-27B is a hybrid: of its 64 layers, 48 are linear attention with a fixed-size state, and only 16 are full attention with a per-token cache, each with 4 KV heads of dimension 256. That gives 2 (K and V) x 16 x 4 x 256 x 2 bytes = 64 KiB per token in BF16, so the visual tokens alone cost:

Those numbers are derived from config.json, not measured, and they are a quarter of what a naive 64-layer estimate would give. That is why the harness counts only full_attention layers.

Measure accuracy at each tier

Now attach the benchmark. ANLS is the DocVQA metric: normalized edit-distance similarity between prediction and each gold answer, thresholded so that near-misses below 0.5 similarity score zero.

import Levenshtein

def anls(pred, golds, tau=0.5):
    pred = pred.lower().strip()
    best = 0.0
    for g in golds:
        g = g.lower().strip()
        if not g and not pred:
            best = max(best, 1.0); continue
        dist = Levenshtein.distance(pred, g)
        sim = 1 - dist / max(len(pred), len(g), 1)
        best = max(best, sim if sim >= tau else 0.0)
    return best

def sweep(dataset, tiers=(256, 512, 1024, 2048)):
    results = {t: {"anls": [], "cost": None} for t in tiers}
    for t in tiers:
        rows = []
        score = 0.0
        for ex in dataset:
            r = measure(ex, t)
            golds = ex["answers"] if isinstance(ex["answers"], list) else [ex["answers"]]
            score += anls(r["answer"], golds)
            rows.append(r)
        n = len(dataset)
        results[t]["anls"] = round(100 * score / n, 1)
        # Median visual tokens (deterministic per image); mean latency and KV.
        results[t]["cost"] = {
            "visual_tokens": int(sorted(x["visual_tokens"] for x in rows)[len(rows)//2]),
            "prefill_ms": round(sum(x["prefill_ms"] for x in rows) / n, 1),
            "kv_mb": round(sum(x["kv_mb"] for x in rows) / n, 1),
        }
    return results

report = sweep(ds)
for t, v in report.items():
    print(f"budget={t:>4} tok | ANLS={v['anls']:>5} | "
          f"visual_tok={v['cost']['visual_tokens']:>5} | "
          f"prefill={v['cost']['prefill_ms']:>7}ms | KV={v['cost']['kv_mb']:>6}MB")

This produces the before/after number the tutorial exists to generate: DocVQA ANLS at the smallest budget versus the largest, on the same model and the same slice, with the exact token and latency cost of each printed alongside. The token and KV columns are deterministic (the token count follows from the image and the budget), so the only thing your hardware changes is the millisecond column, not the shape of the accuracy curve.

Read the curve: the knee, not the peak

No measured curve is printed here on purpose: the numbers are yours to produce, on your card, with your settings attached. What the published DocVQA results and the documented sensitivity of these models to the pixel cap do tell you is the shape to expect, in three regions:

• Starved (very low budget). At 256 tokens a 1700x2200 letter-size scan is resized to 448x576 (252 tokens), and small print stops resolving, and ANLS should land well below your full-budget baseline because the text the question asks about is no longer legible. This is the region where "we saved tokens" silently became "we broke document QA."
• The climb. ANLS rises steeply as the budget grows and the text becomes readable. This is the region where every extra token buys accuracy, and it is short.
• The plateau. Past the point where the text is legible, more resolution stops helping. ANLS approaches its ceiling and flattens while token count, prefill, and KV keep rising linearly. Spending here is pure cost.

The engineering decision is to serve at the knee, the smallest budget on the plateau, not the peak. The peak wastes tokens for accuracy you already have; anything left of the knee trades away accuracy you needed. Because the cost columns are exact and the accuracy columns are measured, the knee is a number you can defend to whoever owns the latency budget, rather than a guess.

One caveat that keeps this honest: the knee is task-specific. DocVQA is deliberately the hardest case, dominated by small text, so its knee sits at a high budget. A task of natural photos with a single salient object will plateau far earlier, and using DocVQA's knee there would overpay. Re-run the sweep on a slice of your own traffic; the harness does not change, only the dataset does. And report the accuracy consequence at every tier, never cost alone, because a resolution table that shows only tokens and milliseconds is the exact artifact that leads a team to cap longest_edge and lose accuracy they did not know they were spending.

Three ways this measurement goes wrong

Before you trust your own curve, rule out the three failures that turn a clean sweep into a misleading one.

1. You changed two things at once. If you vary longest_edge while the answer-extraction prompt or the decoding settings also drift, you cannot attribute the accuracy change to resolution. Hold the prompt, do_sample=False, and the thinking setting fixed across every tier; the only variable under test is the token budget.
2. shortest_edge upscaled your small images. The floor is a floor: an image below shortest_edge gets scaled up. The harness sets it to 4 tokens, but the shipped default is 65,536 pixels (64 tokens), so with default settings a small input can cost more than you labeled it, and the visual-token column will show it. Always read the emitted visual_tokens rather than assuming the budget was spent, which is why the harness reports the measured count and not the cap.
3. You compared accuracy across quant formats. A 4-bit and a BF16 run of the same model are two different measurements, and low-bit numerics hit exactly the fine detail (small glyphs, thin strokes) that DocVQA depends on, which is why the harness keeps the vision tower in BF16 even though the language model is nf4. Pick one precision for the whole sweep, state it next to every number, and do not read a resolution curve produced under nf4 as if it were the BF16 model's curve.

Serving the decision

Once you have the knee, pin it. In transformers, pass the size bounds to the processor as above, or bake them into preprocessor_config.json. If you serve with vLLM, set them server-wide with --mm-processor-kwargs '{"size": {"shortest_edge": 4096, "longest_edge": 1048576}}' (a 1,024-token cap); since v0.31 vLLM rejects per-request mm_processor_kwargs unless the server runs with --trust-request-mm-kwargs, so a server-side cap is also the one clients cannot override. If you run locally through llama.cpp with a GGUF build (covered in Running LLMs Locally), set the cap with --image-max-tokens and keep the projector on GPU with --mmproj-offload. Check the emitted token count on each stack rather than assuming the cap means the same thing everywhere; the knee you measured is a token count, and that is what has to match.

The KV column is the part that surprises people at scale. Because visual tokens are real KV-cache residents, a batch of high-resolution image requests grows the cache exactly like a batch of long text prompts, and the KV math from KV-Cache Engineering applies unchanged: KV bytes scale linearly with total tokens, and visual tokens are total tokens. At 64 KiB per token, sixteen concurrent pages at the default cap hold about 17GB of KV, more than the card has left after the weights; at a 1,024-token knee the same sixteen hold about 1GB. Serving at the knee rather than the peak is not only a latency win, it is a memory-capacity win that decides how many concurrent image requests fit on the card.

Key Takeaways

1. Image token cost is deterministic, not mysterious: a Qwen3-generation visual token covers a 32x32 pixel region (16px patches, 2x2 merge), so tokens ≈ pixels / 1024, and longest_edge is a token budget in disguise. Qwen2.5-VL's 28x28 / 784 rule does not carry over. A dense page at the default cap can cost more tokens than the entire text prompt around it.
2. Resolution is a correctness knob, not a cost slider. DocVQA, dominated by small text, collapses below the legibility threshold and no prompting recovers it. Larger Qwen-VL models report 96 to 97 ANLS at full resolution; your own full-budget run is the baseline, and a starved budget should land far below it.
3. Measure four numbers per tier: visual tokens, time-to-answer, KV growth, and benchmark accuracy. Tokens and KV are deterministic (64 KiB per token on Qwen3.8-27B, counting only its 16 full-attention layers); only the millisecond column depends on your hardware.
4. Always report latency with its context: quantization format, backend, and GPU or driver, plus the thinking and reasoning_effort settings. A latency figure without those is not reproducible.
5. Serve at the knee, not the peak. The knee is the smallest token budget on the accuracy plateau. Left of it you lose accuracy you needed; right of it you pay tokens for accuracy you already have.
6. The knee is task-specific. DocVQA sits at a high budget because it is the hardest case; natural-image tasks plateau earlier. Re-run the same harness on a slice of your own traffic before pinning a production budget.
7. Visual tokens are KV-cache residents. A batch of high-resolution images grows the cache like a batch of long prompts, so serving at the knee is a memory-capacity decision that sets how many concurrent image requests fit on one 24GB card.