LLM Capacity Planning Has a Half-Life: Why Your Serving Numbers Expire

The LLM capacity planning spreadsheet you wrote the day the model shipped is already wrong. Not because you fat-fingered a cell or misread a spec sheet. It is wrong because the serving stack it was pinned to has changed several times since, and each change moved a number you treated as fixed. Your serving numbers have a half-life, and nobody writes the expiry date on them.

This is the uncomfortable follow-on to every capacity decision. When you priced out self-hosting a frontier model at release, you produced two artifacts: a VRAM-and-cost bill that told you how many GPUs to buy, and a throughput baseline that told you how fast the thing would go. Both looked like measurements of the model. Neither was. Both were measurements of the model on a specific version of a specific serving stack, with the defaults that stack happened to ship that week. Change the stack and both numbers move, sometimes far enough to change the decision they were made to support.

The site's own self-hosting bill for frontier open MoEs had to write that bill as a constant. It computed a three-tier VRAM and cost invoice for models like Kimi K3 at release, and a static table is the only honest thing you can publish on release day, because the future defaults have not been written yet. Three months of release notes say the invoice is not a constant. It decays. This piece is about how it decays, why part of the decay is invisible in your own config file, and the re-measurement discipline that keeps a stale number from making a live decision.

A serving number is a snapshot with no expiry printed on it

Start with the thesis, because the specific releases are only evidence for it. A capacity number is a snapshot of a system that keeps moving. When you write "Kimi K3 needs roughly 1.68 TB of VRAM" or "we get N tokens per second at concurrency 32," you are recording an interaction between three things that drift at different rates: the model weights (mostly stable), the hardware (stable until you re-provision), and the serving stack (unstable by design). The number inherits the drift of its least stable input, and the serving stack is that input.

The cadence is not hypothetical. vLLM v0.28.0 landed on 2026-08-26 with 584 commits from 270 contributors. By 2026-10-18 the project had shipped v0.32.0, which alone carried 814 commits from 363 contributors, with v0.29, v0.30 and v0.31 in between. That is five minor releases in under eight weeks, each with its own "new defaults" section, and at the end of October v0.33.0 was already in release candidates. SGLang kept the same pace, from v0.5.18 on 2026-08-22 to v0.5.22 on 2026-10-19, the last of those alone carrying 950 PRs. The stack you benchmarked against in August is not the stack you are running in November, and the delta is measured in thousands of commits. Release notes are worth reading line by line for exactly this reason: the v0.25 note that "removed PagedAttention" deleted a kernel, not the memory model, and a summary read would have told you the opposite.

Two of your numbers decay in two different ways, and keeping them separate is the whole game.

The sizing number decays downward and in your favor: the stack finds memory it did not have before, so the model that needed a certain tier of hardware at release can quietly stop needing it. This is good news you will miss if you never re-run the question, because you already bought the hardware.

The throughput number decays in a way that lies to you: when you re-benchmark on an upgrade, the number moves, and you read the whole move as progress. Part of it is. Part of it is a default the new version changed on your behalf, a config choice you did not make and cannot see, because a default is precisely the value that is not written in your config file. That contamination is the eval-honesty problem at the center of this piece.

Half one: found memory and new kernels move the tier

Take the sizing bill first, because it is the cleaner case.

When the self-hosting piece sized Kimi K3, the binding constraint was memory. K3 is a 2.8T-total MoE with linear-attention (KDA) layers, shipped MXFP4-native, about 1.56 TB on disk, and the weights plus KV and runtime overhead put the practical requirement near 1.68 TB of VRAM. That number decides GPU count, GPU count decides node topology, and node topology decides whether you are buying one box or wiring two together across an interconnect. A memory figure is not a footnote in that chain; it is the first domino.

Now read vLLM v0.28.0's notes with a sizing hat on. Alongside Decode Context Parallel support and fused FlashKDA decode and prefill kernels for K3, the release adds "optional shared-expert sharding saving ~17 GiB of memory per GPU." State that the way the release states it, because it is load-bearing: about 17 GiB per GPU, freed, on the exact model class the sizing table was built for.

Seventeen GiB per GPU is not a rounding error. On an eight-way node it is about 136 GiB of aggregate headroom that did not exist at release. K3's release-day requirement overshot an eight-way B200 node by a margin of roughly that size, which is why the original bill pushed it to the B300 tier. Whether the saving actually closes that gap for you depends on your parallelism layout (the saving comes from no longer replicating shared experts on every GPU) and on how much KV headroom your traffic needs, so treat it as a question to re-run rather than a conclusion. But the magnitude is exactly the one that decides GPU count at the margin, and it arrived as a version bump and a config flag with zero change to the model or the hardware. The reader who filed the release-day invoice and moved on may now be over-provisioned and not know it.

The kernel work kept coming. v0.31.0 added routed-expert quantization for K3 and a fused KimiViT QK RoPE path reported up to 29x faster, vision-side work that moves the encode phase, which is its own serving bill; v0.32.0 added a FlashInfer fused speculative KDA path on Blackwell (SM100) and reports roughly 20% better time-to-first-token from prefill checkpoints on ROCm. None of those is a reason to re-buy hardware on its own. Together they are the reason a release-day throughput figure for K3 describes a system that no longer exists.

One caution keeps this from becoming its own overclaim: found memory is often optional. Shared-expert sharding is an opt-in path, not an automatic saving, and decode context parallelism is configured rather than inherited. So the sizing win is a lever you can see and choose to pull. That visibility matters, because the throughput half is the opposite.

Half two: changed defaults contaminate the throughput delta

Here is where re-benchmarking on upgrade quietly deceives you.

You upgrade the serving stack, re-run your throughput benchmark, and the number is higher. The natural read is "the new version is faster." Sometimes it is, because someone wrote a better kernel. But the same release that shipped the better kernel also changed defaults, and a default change moves your throughput without touching a line of your config. vLLM v0.28.0 raised the default max_num_batched_tokens from 8192 to 16384 and enabled prefix caching by default for Mamba models. Neither of those is a kernel. Both move throughput. And neither shows up in a diff of your own configuration, because you never set those values; the framework set them for you, and then the framework changed its mind.

The later releases make the point harder to ignore, because their default changes reach past throughput into capacity and quality. v0.32.0's "new defaults" include:

• GLM-5.x now defaults to an FP8 KV cache on SM100 when --kv-cache-dtype is left at auto. The release notes price it honestly: half the KV memory per rank (which moves your concurrency ceiling), 2.3% to 5.5% more throughput, and a 7% to 11% time-to-first-token cost in the PR's own benchmark. That is a trade, not a speedup, and it is also a precision change on the attention path, which can move accuracy on long-context workloads. Your config file still says auto, exactly as it did last month.
• MXFP4 W4A4 checkpoints fall back to W4A16 with a warning when no W4A4 kernel exists for the hardware. For an MXFP4-native model like K3 on a GPU without that kernel, the compute path your number was measured on is now recorded only as a log line.
• max_tokens is treated as an upper bound on output rather than reserved from the context window, so long prompts that were previously rejected are now admitted. That changes what your scheduler admits at a given concurrency, which changes the latency distribution your SLA was written against.

This is the eval-honesty catch aimed squarely at your own measurements. When you compare a number from v0.27 against a number from v0.32, you are not comparing two builds of the same configuration. You are comparing two configurations. A larger batched-token budget changes the throughput-latency operating point: it trades time-to-first-token and memory pressure for aggregate tokens per second, and whether that trade is good for your traffic is a decision, not a gift. An FP8 KV default trades a precision margin for capacity. If you read the resulting bump as pure engineering progress, you have credited the maintainers for a choice they made on your behalf, and you have lost the ability to answer the only question that matters operationally: which of my traffic shapes just got better, and which just got worse.

The honest-baseline move is to refuse the aggregate delta until it is decomposed. Here is a sample of changes across that window, sorted by what kind of change each one actually is:

The default rows are the contamination. They are the reason a raw "v0.32 is 1.3x faster" is not a build comparison. A default is the value your config does not mention, so a changed default is a change your config cannot record. If your re-benchmark protocol is "upgrade, re-run, report the delta," you are reporting the sum of every row as if it were only the build rows, and calling a config change a capability gain.

SGLang's v0.5.18 release makes the same point from the other direction. It reports Kimi K3 on AMD MI355X reaching 1.37x to 1.77x throughput and 1.45x to 2.42x better inter-token latency at concurrency 2 to 32, from a grouped-head MLA verify kernel. That is a genuine build gain, a kernel someone wrote. The same release reports overlapped checkpoint staging starting Qwen3-32B on H100 2.38x faster than the plain default (35.6s against 84.8s), gated behind an explicit --startup-weight-load-mode overlap flag. One improvement is a kernel you inherit; one is a flag you must choose. Folding both into "the new version is faster" erases the distinction between what you get without touching your config and what you get only if you opt in.

SGLang v0.5.22 (2026-10-19) adds a nastier case: a value you did write down can change meaning. The release flips the encoding of --disaggregation-decode-host-receive-threshold, so 1, the new default, now disables host receive and 0 now means always receive on host. A team that passed 0 to turn the feature off gets the feature forced on by an upgrade, with a config file that is byte-for-byte identical. Diffing your own config would show nothing; only the release notes and the resolved runtime state would.

That distinction is the whole discipline. A delta across a version boundary is a mixture of build gains you inherit, opt-in gains you have to choose, and default changes that were chosen for you. The first is progress. The second is available progress. The third is a different operating point the maintainers decided was a better default, which may or may not be better for your traffic.

What did they not measure, and does it survive production

Two reasoning moves fall straight out of this.

The first: what did they not measure? When a team re-benchmarks on upgrade and reports a clean improvement, the thing they did not measure is the default change hidden inside the version bump. They measured throughput before and after; they did not measure how much of the gap is a doubled batch budget or a halved KV precision. The benchmark that would separate them (same version, old defaults versus new defaults, model and hardware held fixed) is exactly the one nobody runs, because the upgrade path does not invite it. And when a default touches precision, the number that moved may be accuracy, which a throughput benchmark does not measure at all.

The second: does it survive production? A procurement decision does not survive contact with the release cadence. The VRAM bill you signed off on assumed a serving stack frozen at release, and the stack found 17 GiB per GPU a month later. The throughput SLA you promised assumed the operating point of the defaults you benchmarked, and every release since has moved some of them. Neither number was wrong when you wrote it. Both stopped describing your system the moment the stack moved, which it does on a schedule measured in weeks. A number that only holds until the next pip install is a photograph of a capacity plan, not the plan.

None of this is a knock on the maintainers. A serving stack that finds memory and picks more aggressive defaults is doing its job. The inference-optimization work is supposed to move these numbers, and K3 is a model whose linear-attention KDA layers were built to get cheaper to serve. The failure is not that the numbers move. The failure is treating a moving number as a fixed one, and reading a moved default as a delivered capability. That second error is the one the evaluation-crisis piece names for leaderboard scores: a number without its configuration and its version is half a story, and here the missing half is your own config file.

A re-measurement discipline

The fix is not to re-benchmark less. It is to re-benchmark on a schedule and to record enough alongside each number that a later you can tell progress from a default. Run this as a standing protocol, not a one-time exercise.

1. Record the stack version and its defaults with every capacity number. Not just "K3 needs 1.68 TB" but "K3 needs 1.68 TB on vLLM v0.27.x, max_num_batched_tokens at its shipped default, shared-expert sharding off, KV dtype resolved to BF16." A capacity number without its version-and-defaults stamp is uninterpretable six weeks later, because you will not remember which defaults were in force, and the config file will not tell you.
2. Snapshot the effective config, not the config you wrote. Dump the resolved runtime configuration (the values the engine actually used after applying its own defaults, including what auto resolved to) and store it with the benchmark, along with the startup log, since some fallbacks are recorded only as warnings. Your written config records what you overrode; the effective config records what actually ran.
3. Re-run the sizing question on a schedule, not at procurement. Tie the VRAM-and-cost estimate to the stack's release cadence. The bill decays downward, so the cost of not re-running it is silent over-provisioning: hardware you bought that a later stack no longer needs.
4. Treat any delta that arrives with a version bump as unattributed until the defaults are diffed. Before you credit an upgrade with a speedup, diff the effective config from step 2 against the old one. Attribute the part explained by changed defaults to configuration and only the remainder to the build. If you cannot diff the defaults, report the number as "v0.32 configuration," not "v0.32 build."
5. Re-run a quality check whenever a default touches precision. A changed KV dtype or a quantization fallback can move accuracy while throughput looks better. Keep a small fixed accuracy eval next to the throughput benchmark and run both on every upgrade.
6. Label the three kinds of improvement in writing. For each gain, mark it inherited build gain, opt-in gain, or changed default. Readers of your report (including future you) need to know which gains are free, which must be enabled, and which were chosen for them and might be worth reverting with an explicit override.
7. Re-decide, do not just re-measure. When the sizing number moves a tier, that is a procurement signal, not trivia. Freed memory can mean a smaller node on the next refresh or more concurrency headroom on the box you own. A moved default can mean an SLA you can tighten or a regression you need to pin back. The number changing is the input; the decision is the output.

The through-line: a self-hosting decision made at release is a snapshot whose expiry nobody wrote on it, and a comparison that spans a stack version boundary compares two configurations, not two builds. Record the version and the resolved defaults with every number, and the snapshot stops pretending to be permanent.

Key Takeaways

1. LLM capacity planning numbers have a half-life. The VRAM bill and throughput baseline you wrote at model release both decay because the serving stack keeps changing underneath them. vLLM went from v0.28.0 (2026-08-26) to v0.32.0 (2026-10-18) in under eight weeks, SGLang from v0.5.18 to v0.5.22 in the same window, and each release shipped its own "new defaults."
2. The sizing bill decays downward and in your favor, silently. vLLM v0.28.0 reported optional shared-expert sharding freeing about 17 GiB per GPU on Kimi K3, about 136 GiB across an eight-way node. That is the margin that decides GPU count, so a release-day sizing table can leave you over-provisioned, and you only find out by re-running it.
3. Throughput deltas across a version boundary are contaminated by default changes. v0.28.0 doubled max_num_batched_tokens to 16384 and turned on Mamba prefix caching; v0.32.0 made FP8 the auto KV dtype for GLM-5.x on SM100 and added a W4A16 fallback for MXFP4 checkpoints. None of these appear in your config file.
4. Separate build gains, opt-in gains, and changed defaults. SGLang v0.5.18's 1.37x to 1.77x K3 throughput on MI355X is a kernel gain you inherit; its 2.38x faster startup is an opt-in behind --startup-weight-load-mode overlap; a doubled batch default is a choice made for you. Only the first is unambiguous progress.
5. Snapshot the effective config, not the one you wrote. You cannot diff a change you never wrote, and SGLang v0.5.22 flipped the meaning of a flag value people did write. Store the resolved runtime config and the startup log with every number, or a future default change is undetectable from your records.
6. Some defaults move quality, not just speed. A KV precision change or a quantization fallback can make a throughput number look better while accuracy moves. Pair every throughput re-run with a fixed accuracy check.
7. Re-run sizing on the stack's release cadence, and re-decide when it moves. A moved tier is a procurement signal; a moved default is an SLA input. Log the number, then make the decision it implies.