Vision Encoder Disaggregation: When Your Encoder Should Be Its Own Service

You are serving a vision-language model and your time-to-first-token is far worse than the same GPU gives you on text. The profiler says the language model is barely the problem. Requests carrying images stall in the vision encoder before a single token of prefill runs, and because the encoder shares the GPU and the scheduler with prefill and decode, a burst of image traffic stalls the text-only requests queued behind it too. The structural fix that every major serving stack now ships is vision encoder disaggregation: running the encoder as its own instance, separate from the prefill and decode work of the language model.

vLLM calls it EPD (encode, prefill, decode), SGLang ships it under the same name, and NVIDIA Dynamo orchestrates it across vLLM, SGLang and TensorRT-LLM backends. It is easy to read as a flag you turn on for free throughput. It is an architecture decision: a multimodal model stops being one service you scale as a unit and becomes up to three services with three cost profiles, three autoscaling signals and three failure domains. The vendor numbers attached to it are real, and the best of them were measured on exactly the traffic shape where the answer is obviously yes. The same posts also contain the configurations where it loses. This piece is about telling the two apart before you split anything.

Why vision encode is a third phase, not a term inside prefill

Prefill and decode were disaggregated because they have different bottlenecks. Prefill is compute-bound and processes the whole prompt in one parallel pass. Decode is memory-bandwidth-bound and emits one token at a time while streaming the KV cache. Run them on one instance and one phase's profile sets the hardware while the other pays for it.

Vision encode fails the same test, and it fails it in a direction neither of the other phases shares. A vision transformer turns pixels into a block of embedding vectors before the language model can attend to anything, and that work has its own profile:

Two cells in the encode row do most of the work. It produces no KV cache: everything the KV-cache literature tells you to optimize (paging, quantized cache, prefix reuse) is irrelevant to it. And its output is a pure function of the image, the preprocessing and the encoder weights, so two users asking different questions about the same document page need exactly one encode. Fold the encoder into prefill and both properties disappear into a blended TTFT number: the scheduler cannot tell image work from text work, cannot batch or scale them separately, and has no cache object to reuse.

The per-image price is set by resolution. Current Qwen vision models (Qwen3-VL, Qwen3.5, Qwen3.8) cut images into 16-pixel patches and merge each 2x2 block into one visual token, so one token covers a 32x32 pixel region and a 1,024-token image is roughly a 1-megapixel input. The processor's default ceiling is 16,384 visual tokens per image. That budget is the first lever, and it is a correctness lever before it is a cost lever: the resolution-sweep tutorial and the analysis in Multimodal LLMs in Production both land on serving at the knee of the accuracy curve. Disaggregation changes where those tokens are produced. It does nothing to reduce how many there are.

What actually ships, as of vLLM v0.32

The mechanism is the same across stacks. The encoder runs in its own instance, and an encoder-cache (EC) connector moves its output to whichever instance runs the language model. In vLLM the connector has a scheduler-side role, which checks whether an image's embeddings already exist and schedules a load, and a worker-side role, which moves them into the consumer's memory; the code lives under vllm/distributed/ec_transfer. Two layouts matter:

• E then PD (1E+1PD): a dedicated encoder feeds a combined prefill-and-decode instance. This is where the isolation win is and where most fleets should start.
• E then P then D (1E+1P+1D): encoder, prefill and decode are all separate. Prefill runs one step, then hands its KV to decode exactly as in text-only P/D disaggregation, while embeddings arrive from the side.

The timeline matters, because most of what you will read about this is a release-note snapshot. vLLM first shipped native encoder disaggregation in v0.11.1 (November 2025, PR #25233). v0.28.0 (August 2026) brought it to Model Runner V2 with GPU-side image preprocessing, encoder CUDA graphs, and KV consumers that may omit multimodal embeddings because the EC connector already carried them. The networked transports that early write-ups described as nightly-only shipped in v0.30.0 (September 2026): a P2P NIXL path through the CPU EC connector and ECMooncakeConnector over the Mooncake Transfer Engine. v0.31.0 (October 2026) added a dynamic EPD proxy, per-encoder image batching, cross-encoder output reuse through a shared Mooncake store, an --mm-encoder-only flag that skips loading language-model shards on encoder instances, and EC connector metrics. v0.32.0 (October 18) added no new EPD features and hardened the existing path instead: EC connector transfers are now sized by the encoder's measured output width, and shared memory is released when NIXL setup fails. v0.33.0 is in release candidates as of October 31. The shared-filesystem ECExampleConnector used in the example scripts is still the demo path, not the production one.

SGLang has had EPD since January 2026 (--encoder-only encoder servers, --language-only prefill, ZMQ or Mooncake transfer), Dynamo (v1.6, October 2026) ships launch recipes for E/PD and E/P/D on vLLM plus encoder workers on TensorRT-LLM, and llm-d v1.0.0 (October 20) folded E/PD and E/P/D into its standard disaggregated-serving recipe. Two honest caveats survive all of that. vLLM's own EPD tracker is still open, and model support is per model (a model has to declare which fields its embeddings carry before an EPD pair can serve it), so "vLLM supports EPD" and "EPD works for the checkpoint you run" are separate claims. And Dynamo's full E/P/D launcher still defaults to LLaVA 1.5 for functional testing. Confirm your model on your build before you design a capacity plan around it.

The speedups are real, and they describe one traffic shape

There are now three independent sets of published numbers, and reading them side by side is more useful than any one of them.

vLLM (December 2025): on 4x A100 with Qwen3-VL-4B, one encoder plus three PD instances against four data-parallel replicas. With four images per request and short text, goodput doubled (6 to 12 QPS). With long text, gains ran 2.25x to 2.75x. With a single image and short text, goodput moved from 23 to 24 QPS, which is noise.

NVIDIA Dynamo (September 2026): on GB200 with Qwen3.5-122B-A10B in NVFP4, up to 5x faster TTFT and up to 7x faster end-to-end latency under heavy image load. Under a 50:50 mix of text and image traffic, text-request TTFT fell 42% (92.3 to 53.3 ms) because text no longer waited behind vision encodes. Then the same post runs the ablations that matter more than the headline. As output length grows from 128 to 2,048 tokens, the end-to-end gain narrows from 20.3% to 5.2%, and colocated EPD goes slightly negative (-2.5%) at five images and 2,048 output tokens. Across model size, the vision tower's share of parameters falls from 7.2% at Qwen3.5-4B to 1.7% at Qwen3.5-27B, and colocated EPD's goodput relative to aggregated serving falls with it: 2.62x, 1.50x, and 0.65x. On a dense 27B, splitting the encoder made serving worse.

SGLang and llm-d (2026): SGLang reported 6 to 8x lower latency than colocation at 1 QPS on multi-image requests with Qwen3-VL-235B-A22B, and says plainly that for image-light traffic the embedding transfer can cost more than the offload saves. The llm-d team reported 2.4 to 2.8x throughput and 69 to 80% lower TTFT on Kimi-VL with four extra Intel B60 GPUs as encoders, and notes in its own caveats that the comparison does not use the same total accelerators: it measures what four added encoder GPUs buy, not what splitting the same hardware buys. That is the honest-baseline question every one of these numbers has to answer, and that post answers it in its own text.

Put together, the pattern is consistent enough to predict your own result without running anyone's benchmark. Encoder disaggregation wins when vision encoding is a large share of request time: many or high-resolution images, short-to-medium outputs, a small or quantized or sparse language model whose active compute does not dwarf the encoder, and a mixed text-and-image stream where isolation matters. It loses when decode dominates, when images are few and small, or when the language model is large and dense. A headline multiplier is a claim about a measurement setup. Copy the setup before you quote the number.

What you sign up for: three services

Suppose your traffic passes that test. Three operational consequences follow that a fused deployment never had to think about.

Three autoscaling groups, three signals

The encoder pool scales on images per second and visual tokens per second. Prefill scales on input tokens per second, now including embedding blocks it receives rather than computes. Decode scales on concurrent sequences and KV capacity. These move independently: a wave of document uploads should add encoders without touching decode, and a wave of long chat generations should add decode without touching the encoder. Wire all three to one blended metric and you have paid for disaggregation while keeping a monolith's scaling behaviour. The ratio between tiers is workload-specific and drifts, so it is a dashboard, not a constant. The vision encoder is also small: Qwen3.8-27B's vision tower is 27 layers at width 1,152, roughly 0.4B parameters against a 27.8B language model, so a full same-class GPU per encoder is usually wasted. NVIDIA's own recommendation on homogeneous clusters is a colocated encoder (separate encoder workers sharing the PD GPU with their own queue and batching) and a separate encoder tier only when you have a cheaper GPU class to put it on.

Encode stops being a latency term and becomes a queue

On a fused instance, encode is additive latency inside TTFT. Disaggregated, it is a queue in front of prefill plus a network hop, so the failure mode changes from "TTFT crept up" to "requests are waiting for an encoder slot." That is a better failure mode, because you can see it and scale the saturated tier, but only if your latency instrumentation breaks out encoder queue wait as its own component. Otherwise you will blame the language model for a starved encoder pool.

The hop is smaller than people assume, until it is not. An image embedding for Qwen3.8-27B is visual tokens times the 5,120-wide projection output: at a 1,024-token budget that is about 10.5 MB in BF16, around 4 ms over the 20 Gbps link NVIDIA measured. At the default 16,384-token ceiling it is about 168 MB per image, closer to 70 ms. That is the same resolution budget showing up a third time, now as network traffic. Note also what disaggregation cannot fix: in NVIDIA's breakdown, 43% of TTFT was spent before the ViT even started, in media download, decoding and preprocessing. That is a CPU and serving-frontend problem, and no encoder topology touches it.

The encoded image becomes a cache object with a correctness question

Cross-request reuse is one of the three benefits vLLM's own docs list for the split, and it turns the encoded image into shared state with a hit rate, an eviction policy and a validity rule.

Hit rate is a property of your traffic. A fixed product catalog, a document re-queried across a session, or a reference chart injected into every prompt will cache beautifully. A stream of one-shot user uploads will cache nothing and pay transfer overhead for no return. Measure the reuse rate before you count it as a benefit.

Eviction is a capacity decision, because at 10 to 170 MB per image the cache fills fast.

Validity is the one that ships as a silent bug. An encode is a function of the image bytes, the preprocessing configuration (resolution budget, normalization, tiling) and the encoder weights. Change any of the last two and every cached entry is stale even though the image bytes are identical. vLLM v0.31 added a guard that rejects encoder-cache hits whose embedding count does not match, which catches a resolution change that alters the token count. It cannot catch a fine-tuned or point-released encoder that produces embeddings of the same shape. The other stacks are finding the same class of bug in their own keys: SGLang added re-encoding after an embedding-cache mismatch in v0.5.19, and Dynamo v1.6.0 fixed an image cache key that did not preserve the case of the image URL path. A stale hit returns a confident answer about the image from the wrong model, with no error and no latency signal. Key the cache on image hash, preprocessing config and encoder version, and make a model deploy or a resolution change an automatic cache flush.

The honest baseline: what to try before you split

Most of the benefits people attribute to disaggregation have cheaper versions that do not add failure domains.

• Cap the resolution budget. Fewer visual tokens shrink encoder compute, prefill, KV and transfer at once. Find the knee on your own data first.
• Turn on the encoder cache in a single instance. vLLM's EC connector supports an ec_both role in which one instance both produces and consumes cached embeddings, and Dynamo layers a CPU-side LRU embedding cache on top of it. Repeated images skip the encoder entirely, with no second service.
• Data-parallel the vision tower on one node. vLLM's ViT data-parallel plus LLM tensor-parallel mode cuts multimodal TTFT on a multi-GPU box without standing up separate instances.
• Colocate before you disaggregate. A colocated encoder worker gets separate queueing and batching on the same GPU, which is most of the isolation win without a new hardware tier.
• Move media decoding off the worker. If a large share of your TTFT is spent before the ViT starts, fix the frontend first.

The concrete version of the single-node case: an open multimodal checkpoint like Qwen3.8-27B (dense 27.8B, text, image and video in, Apache 2.0, about 17 GB in a 4-bit build on one 24 GB card) has a vision tower at roughly the 1.7% parameter share where NVIDIA measured colocated EPD at 0.65x of aggregated goodput on the same architecture. On one card it wants one process with a capped resolution and a warm encoder cache. Disaggregation is designed for the fleet where a smaller, sparser or quantized model serves heavy, mixed, repeated image traffic across many machines.

Should you disaggregate the encoder?

Run this in order and stop at the first clear answer.

1. Measure the encoder's share of request time on a real traffic slice: media decode, ViT forward, prefill, decode, each logged separately. If ViT forward is a small fraction of TTFT (NVIDIA's ablations suggest single digits is not enough on a large dense model), stop. You do not have the problem this solves.
2. Measure output length. If median generation is long, latency lives in decode and no encoder topology will move end-to-end numbers. TTFT may still improve; decide whether TTFT is the SLO you are actually missing.
3. Check for a mixed stream. If a meaningful share of requests are text-only and they miss TTFT during image bursts, the isolation win is real on its own.
4. Estimate image reuse. Sample a day of image hashes. High reuse argues for the encoder cache, which you can get in a single instance first.
5. Try the baseline levers above (resolution cap, single-instance encoder cache, ViT data parallel, colocation) and re-measure. Many fleets stop here.
6. Only if steps 1 and 3 still clear, and you are multi-node, adopt E then PD. Confirm the EC connector works for your exact checkpoint on your vLLM or SGLang build, use a production transport (NIXL or Mooncake, not the shared-filesystem example), and alert on encoder queue depth and EC connector errors.
7. Pin cache invalidation to your deploy process and keep the fused topology as a rollback, A/B'd on live traffic with the same total accelerators before you cut over.

The reason to understand this even if you never turn it on is that it reframes multimodal serving. For two years the default mental model was a language model that happens to read images. Encoder disaggregation is the serving stack admitting that a vision-language model is two systems with different cost curves stapled together. Whether you exploit that is a measurement question, and the first measurement is cheap. For the rest of the serving picture, start with LLM Inference Optimization.

Key Takeaways

1. Vision encode is a distinct serving phase. It is compute-bound, produces no KV cache, is priced per image and is cacheable across requests, which is why folding it into prefill inflates TTFT for every request, including text-only ones.
2. Every major stack ships it now. vLLM has had EPD since v0.11.1, put it on Model Runner V2 in v0.28.0, shipped networked NIXL and Mooncake transports in v0.30.0, and added a dynamic proxy and cross-encoder caching in v0.31.0. SGLang and NVIDIA Dynamo ship equivalents. Per-model support still varies, so verify your checkpoint.
3. The published gains are conditional, and the same sources show the losses. NVIDIA's up to 5x TTFT came from a quantized 122B MoE under heavy image load; in the same post colocated EPD fell to 0.65x of aggregated goodput on a dense 27B, and end-to-end gains shrink toward zero as outputs lengthen.
4. The deciding number is the encoder's share of request time. Measure it before anything else. On a large dense model the vision tower is under 2% of parameters and there is little to isolate.
5. Three tiers means three autoscaling signals. Scale encoders on images and visual tokens per second, prefill on input tokens, decode on concurrent sequences, and instrument encoder queue wait as its own latency component.
6. The encoded image is shared state. Key the cache on image hash, preprocessing config and encoder version, and flush on every model or resolution change; vLLM's shape guard catches token-count mismatches, not a same-shape stale encoder.
7. Try the cheap levers first. A capped resolution, a single-instance encoder cache, ViT data parallel and a colocated encoder recover most of the benefit without adding failure domains. On one card, stay fused.