Self-Hosting a Frontier Open MoE: The Real GPU Bill for a 1.6-2.8T Open-Weight Model

The open-weights wave of mid-2026 keeps handing you frontier models you can git clone. DeepSeek V4 shipped MIT-licensed and sitting on Hugging Face. Kimi K3, promised on Hugging Face this week, will be the largest open-weight model ever released at 2.8 trillion parameters. Downloading them is free. Serving them is not, and the distance between those two facts is where most plans to self-host a frontier open MoE quietly fall apart. Every one of these models leads with a small "active parameters" number that sounds like something you could run on hardware you already own. It is a trap, and it is the same trap in every announcement.

Self-hosting a frontier open MoE is not a download problem. It is a memory-capacity problem, and the spec sheet actively hides the number that decides it. DeepSeek V4-Pro is 1.6T total with 49B active per token; Kimi K3 is 2.8T total with roughly 50B active. People anchor on the 49B and the 50B because they look runnable. But the active count only tells you how fast the model decodes once it is loaded. What you have to buy is set by the total. Conflate the two and you under-provision a deployment by a factor of thirty to fifty, which is another way of saying the model never loads.

This piece is about the invoice, not the architecture. If you want the attention mechanisms that make these million-token contexts tractable, the DeepSeek V4 hybrid attention breakdown covers V4's compressed-and-selected design and the Kimi K3 KDA fact-check covers K3's linear-attention layout. Here the question is narrower and more expensive: what does it actually take to stand one of these models up on your own hardware, and when is that a worse idea than paying the API by the token.

What "open weights" actually buys you

Open weights buy you three things: ownership (the model runs where you say it runs), independence (no rate limits, no deprecation, no vendor reading your prompts), and, at sufficient volume, a lower marginal cost per token. Those are real and sometimes decisive. What open weights do not buy you is a small model. The MIT license on V4-Pro (1.6T total, 49B active) is the same license, and the same size file, as the one on V4-Flash (284B total, 13B active). The GPU bill is not the same at all.

This is why the frontier open labs now ship in tiers. DeepSeek splits V4 into Pro and Flash for exactly this reason: Flash is the variant a small team can realistically self-host, Pro is the frontier-scoring variant posting the headline coding numbers, and Pro is a data-center deployment whether you like it or not. Kimi K3 sits a full tier above even V4-Pro. Picking the variant is the first and most consequential decision, and it is not a quality decision, it is a capacity decision. Get the capacity math wrong and the quality question never comes up, because the model does not load.

The active-versus-total parameter trap

Here is the single most important fact about serving any mixture-of-experts model, and the one the "49B active" and "50B active" framing actively obscures.

A dense model reads every parameter for every token. A 50B dense model holds 50B parameters in memory and reads all 50B on each forward pass. If DeepSeek V4-Pro or Kimi K3 behaved like that, you could serve either on a couple of high-memory cards. Neither does.

An MoE holds all of its parameters resident but reads only a subset per token. It keeps its feed-forward layers split into many small experts and routes each token through only a few of them, plus a set of shared experts that fire for everything. K3 makes the sparsity vivid: 896 experts, 16 activated per token, roughly 1.8% of the pool touched at a time. That routing is what produces the 50B active figure. But routing is per-token and data-dependent. Across a batch, across a sequence, across your real traffic, every expert gets used. There is no subset you can leave on disk, because you cannot predict which experts the next token needs until you have already read the router output. So all 2.8T parameters (or all 1.6T for V4-Pro) have to be resident in fast memory, all the time.

This splits into two rules you should tape to the wall before sizing anything:

• Total parameters set your VRAM. You provision for 2.8T or 1.6T, not 50B or 49B. This is capacity, and it is non-negotiable.
• Active parameters set your speed. Decode is memory-bandwidth-bound, and per token you read only the active experts, so both models decode at a rate closer to a 50B dense model than to a multi-trillion one. This is throughput, and it is the good news.

The MoE bargain is exactly this: you pay multi-trillion memory prices to get roughly dense-50B decode speed. That is a genuinely good trade if you can afford the memory and a complete non-starter if you sized for the active count. It is the same principle behind the two numbers that govern local inference, VRAM to fit and bandwidth to run, taken to its most extreme case: MoE saves you bandwidth, never capacity.

The one-line version to carry into every sizing conversation: active parameters are a compute number, not a memory number.

The actual VRAM bill

Both models ship in FP4-class low precision, and that is the only reason these numbers are merely large rather than absurd. DeepSeek releases V4 as a mixed checkpoint: the MoE expert weights, the overwhelming bulk of the parameters, are stored in FP4, while attention, normalization, and router weights stay in FP8. Kimi K3 is promised in MXFP4. In both cases this is not a lossy step you apply and gamble on after the fact; it is how the model was trained and released. Roughly half a byte per parameter is the going rate. Whether 4-bit still holds quality on your task distribution rather than on perplexity is a separate question the quantization deep dive takes up, and one the 4-bit-floor section below sharpens for K3; here it is simply what sets the file size. Even at half a byte per parameter, the bill is steep.

V4-Flash is the entry point. Its weights are roughly 158 GB, and a full million-token KV cache adds only about 10 GB on top (more on why that number is so small below), for a total near 170 to 175 GB. That fits comfortably on two H200 SXM cards in a single pod, or on a pair of workstation-class Blackwell cards. This is a deployment a well-funded startup can own outright, and the one most self-hosting plans should actually be about.

V4-Pro lands near 900 GB of weights in its native FP4/FP8 mix. That does not fit on an 8-way H100 80GB node, which tops out at 640 GB of aggregate VRAM. It needs an 8-way H200 141GB node (1,128 GB aggregate) as the realistic single-box minimum, or a two-node deployment with the model sharded across the interconnect. This is a machine that rents for tens of dollars an hour and costs a quarter of a million dollars and up to buy.

Kimi K3 is the same problem one tier harder. At 2.8T parameters and roughly half a byte each, the weights alone come to about 1.4 to 1.5 TB. That overshoots even a full 8-way H200 node. Serving K3 realistically means an 8-way Blackwell (B200 at 192 GB, or B300 at 288 GB) node, or a two-node H200 deployment with the model sharded across the interconnect. One important caveat: as of this writing K3's weights have not landed (Moonshot promised them for July 27), so its active count, MXFP4 format, and expert layout are announced specifications rather than a checkpoint anyone has stood up. Treat the K3 figures here as the bill the spec implies, not a measured deployment, and re-check them against the real checkpoint the day it ships.

Notice what is not doing the damage. The KV cache, usually the villain of any serving story and the thing that breaks under concurrency and long context, is the cheap part for both of these models, and for opposite architectural reasons. V4's hybrid attention compresses the cache so hard that a full 1M-token context costs about a tenth of what the previous DeepSeek generation needed, on the order of 10 GB. K3 goes further still: it runs linear attention in three of every four layers, so only a quarter of the stack keeps a KV cache that grows with context at all. Either way, the weights are the wall. That inversion is worth internalizing, because it reverses the usual advice: for a frontier open MoE you provision for parameters first and context almost as an afterthought.

The 4-bit figure is a floor, not a dial

It is worth being precise about why these FP4 numbers are the real bill and not an aggressive setting you can dial back. For a normal open model you have a choice: serve the BF16 weights and eat the memory, or quantize to 4-bit after the fact and eat some quality loss on the tail. Kimi K3 removes that choice. Moonshot describes K3 as trained quantization-aware, with MXFP4 as its native format rather than a compression applied afterward, so the reference precision is the low one. K3 in BF16 would be about 5.6 TB and does not exist as a served artifact; the roughly 1.4 to 1.5 TB at 4-bit is the floor, full stop.

That cuts two ways. It means the FP4 footprint carries no post-hoc degradation to discover, which is the good news. It also means you cannot buy back tail accuracy by stepping up to a higher-precision checkpoint, because there is no higher-precision checkpoint to step up to. And native-low-precision is itself a claim: a quantization-aware-training result at 2.8T is a vendor hypothesis until the weights and technical report land, so the honest move is unchanged. Validate the served model on your own task distribution and your own context lengths, exactly as the quantization deep dive argues for any 4-bit deployment, because perplexity hides the damage that lands on long reasoning chains and long-context KV.

How you actually serve it: expert parallelism, not tensor parallelism

The instinct with a huge model is to reach for tensor parallelism: shard every weight matrix across eight GPUs and let the framework stitch the math back together. For a dense model that is correct. For a sparse MoE at this scale it is the wrong default, and the reason is communication.

Tensor parallelism requires an all-reduce across every participating GPU on essentially every layer. Inside a single node, over NVLink, that is tolerable. The moment you cross a node boundary, or the moment you are moving expert activations around on every token, the all-reduce traffic starts to dominate and your expensive GPUs spend their time waiting on the network instead of computing. This is exactly the regime K3 forces you into, because it does not fit in one node.

Expert parallelism is the shape that fits the model. Instead of splitting each expert across all the GPUs, you place whole experts on different GPUs and route each token to wherever its experts live, using an all-to-all dispatch and combine rather than a per-layer all-reduce. The dense parts (attention, the router, the shared experts) get data-parallel replicas. This maps the sparsity of the model onto the topology of the machine: only the tokens that need a given expert ever talk to the GPU holding it. Both vLLM and SGLang support this natively, along with the hybrid and linear attention layers these models use; SGLang v0.5.15 and vLLM v0.25.0 both ship the relevant MoE routing and attention kernels. You configure the parallelism, you do not implement it. But you do have to choose it, and "run it like a big dense model with tensor-parallel-size 8" is the choice that quietly halves your throughput.

The offload trap

When the weights do not fit, the tempting escape hatch is offload: keep the hot experts in VRAM and page the cold ones from CPU RAM or NVMe on demand. For a dense model, partial offload is a known and bounded tax, roughly an order of magnitude slower on the offloaded layers because you are reading them at PCIe bandwidth (tens of GB/s) instead of HBM bandwidth (thousands of GB/s).

For an MoE, offload is worse, and it is worse for a structural reason. Expert routing is unpredictable per token, so you cannot keep a useful "hot set" resident; the next token can demand an expert you just evicted. Every routing miss stalls the whole decode step on a PCIe fetch, and because the misses are scattered across your traffic rather than confined to a few layers, the stalls do not amortize. Offloading a dense model turns a fast decode into a slow one. Offloading a sparse MoE with unpredictable routing turns a bandwidth-bound decode into a PCIe-bound crawl. If your plan for V4-Pro or K3 involves NVMe offload to fit it on fewer GPUs, the plan is to serve single-digit tokens per second, which is another way of saying the plan is to run the smaller variant instead.

The bill versus the API

Now the reality-gap question that should have been first: should you self-host this at all?

Run the two columns honestly. An 8-way H200 node for V4-Pro rents around the low-thirties of dollars per hour at the current market median of roughly four dollars per GPU-hour, which is about 23,000 dollars a month if you keep it lit around the clock, before you have served a single production request at scale or paid an engineer to keep it alive. A K3-class deployment on Blackwell or two H200 nodes runs meaningfully higher. Buying the hardware is a quarter-million-dollar capital line and up, plus power and cooling. Against that, DeepSeek's own API lists V4-Pro at about 0.87 dollars per million output tokens and V4-Flash near 0.50, some of the lowest frontier-adjacent pricing available, and Moonshot's hosted Kimi API will price K3 in the same neighborhood once it ships.

The break-even is not close for most workloads. Put a number on it: a 23,000-dollar-a-month node divided by the V4-Pro API rate of 0.87 dollars per million output tokens is roughly 26 million output tokens a month, and that is only the crossover for the marginal cost of a node running flat out. It ignores the capital, the power, the failure headroom, and the engineer keeping a multi-node MoE stack alive, all of which push the real crossover higher. To win, you have to push that volume sustained, month after month, at high duty-cycle utilization. Bursty traffic, moderate volume, or anything that leaves the GPUs idle half the day loses to the API outright, because the API charges you nothing when you are not using it and your node charges you the same whether it is saturated or asleep.

So the honest decision rule is the open-weights-pragmatism one, applied without ideology: self-host a frontier open MoE when the model is the system, and rent it by the token when it is a component. The model is the system when you have a hard requirement the API cannot satisfy (data residency, no third party sees your prompts, an air-gapped environment, a custom fine-tune on top of the base weights) or when you are running genuinely high, steady volume where owning the marginal cost pays back the fixed cost. In every one of those cases, self-hosting is the right call and the six-figure bill is the price of a real requirement. Absent one of them, "the weights are free" is the most expensive sentence in the plan, because the weights were never the cost. The GPUs are.

For most teams that means V4-Flash on a two-GPU box for the workloads that genuinely need to stay in-house, and the API for everything else, including any exploratory use of V4-Pro or K3. That is not a compromise. It is what the numbers say, and the numbers do not care that the license was generous.

A sizing runbook before you commit hardware

Run this in order before you provision anything. It turns the spec sheet into an honest estimate and catches the trap at each step.

1. Compute the resident-weight floor from total parameters, not active. Total parameters times bytes-per-parameter (about 0.5 for FP4-class formats, 2 for BF16) is your floor. K3 is roughly 1.4 to 1.5 TB at 4-bit; V4-Pro roughly 900 GB. This number, not the active count, decides how many GPUs you buy.
2. Add the KV cache for your real context length. For these two models the cache is small (V4's compressed attention, K3's linear layers), but size it for your P95 context and concurrency, not the demo. For dense-attention models this line dominates instead.
3. Count GPUs, then check the interconnect. Divide floor plus KV plus overhead by per-GPU VRAM. If the answer crosses a node boundary (as K3 does), confirm you have fast inter-node fabric, because expert routing will cross it on nearly every token.
4. Choose expert parallelism, not tensor parallelism. Configure whole-expert placement with all-to-all routing. Running a big MoE with tensor-parallel-size 8 like a dense model is the silent throughput killer.
5. If you are tempted to offload, benchmark with offload on. Measure tokens per second with experts actually paged from CPU or NVMe, never the all-resident number. The PCIe fetch tax is the real latency, and for MoE it does not amortize.
6. Estimate break-even against the API. Price your monthly token volume at the hosted rate and compare to the fully loaded node cost (GPUs, power, ops). Below roughly the tens-of-millions-of-tokens-a-month mark, the API wins.
7. Validate quality on your own distribution before you scale. Run your eval at your real context lengths on the quantized served model. The vendor benchmark, and any quantization-aware-training claim, is a hypothesis until you measure it on your workload.

Key Takeaways

1. Total parameters set VRAM, active parameters set speed. Self-hosting a frontier open MoE means provisioning for all 1.6T (V4-Pro) or 2.8T (Kimi K3) parameters, not the ~50B active. Every expert has to be resident because per-token routing is unpredictable; the active count only tells you how fast it decodes once it is loaded.
2. The MoE bargain is dense-memory prices for sparse-compute speed. These models cost multi-trillion-scale VRAM but decode at a rate closer to a 50B dense model. That is a good trade if you can afford the memory and a non-starter if you sized for the active count.
3. The variant choice is a capacity decision, not a quality one. V4-Flash (~170 GB) fits two H200s and is a startup deployment. V4-Pro (~900 GB) needs an 8-way H200 node. Kimi K3 (~1.4-1.5 TB in MXFP4) overshoots even that, needing 8-way Blackwell or two H200 nodes. Pick the tier you can actually serve.
4. K3's 4-bit footprint is a floor, not a dial, and still a projection. MXFP4 is its native, quantization-aware-trained format, so ~1.4-1.5 TB is the real floor with no higher-precision checkpoint to fall back to (BF16 K3 would be ~5.6 TB and does not ship). But the weights were promised for July 27 and had not landed at time of writing, so the active count, format, and QAT claim are announced specs; re-verify against the real weights before committing hardware.
5. The KV cache is the cheap part, the weights are the wall. V4's hybrid attention shrinks a full 1M-token cache to about 10 GB; K3's linear attention keeps a growing cache on only one layer in four. For these models you provision for parameters first and context almost as an afterthought.
6. Use expert parallelism, not tensor parallelism. Placing whole experts on different GPUs with all-to-all routing fits the model's sparsity; sharding every matrix with per-layer all-reduce wastes the interconnect, especially once K3 forces you across a node boundary. vLLM and SGLang support the former natively, but you have to configure it.
7. Offload punishes MoE harder than dense models. Unpredictable routing means no stable hot set, so every miss stalls on a PCIe fetch. Offloading V4-Pro or K3 to fit on fewer GPUs turns a bandwidth-bound decode into a PCIe-bound crawl; the honest fix is the smaller variant.
8. The API usually wins on cost. An 8-way H200 node runs around 23,000 dollars a month lit around the clock (a K3-class box higher); at the V4-Pro API rate of 0.87 dollars per million output tokens that is a marginal-cost break-even near 26 million output tokens a month, before capital and ops push it higher. Self-host only when the model is the system (privacy, residency, air-gap, custom fine-tune, or high steady volume), not because the weights were free.