Linear Attention at Frontier Scale: Kimi K3's KDA Claim, Fact-Checked

Editor's note: Moonshot published the Kimi K3 weights and the technical report on July 27, 2026. The three claims below are re-graded against the released artifacts, and the specs have been corrected where the shipped model differs from the announcement.

The most informative number in Kimi K3's architecture is not 2.8 trillion. It is three to one. Moonshot AI's new flagship, announced July 16, 2026 as the largest open-weight model ever released, runs Kimi Delta Attention (KDA), a linear attention variant, in three of every four layers, and keeps exact full attention in the fourth. If linear attention had actually beaten full attention outright, that ratio would be four to zero. The layout is a confession and an achievement at the same time, and reading it correctly tells you more about where attention research stands in mid-2026 than the parameter count does.

The claim attached to K3 is a strong one: that this hybrid matches or beats full attention on quality while cutting KV cache by up to 75% and decoding up to 6x faster at a million tokens of context. Vendors assert quality-neutral efficiency wins all the time, and most of those assertions quietly die on contact with production. This one deserves a real fact-check, for two reasons. First, Moonshot has more published evidence behind it than the average architecture claim, including a paper with matched baselines and released checkpoints. Second, another frontier lab ran the same experiment at scale, published the opposite conclusion, and shipped accordingly. Both cannot be fully right at every scale, and the gap between them is exactly where the honest story lives.

The Claim, Stated Precisely

K3's headline specs first, because the architecture only makes sense against them. The model is a 2.8-trillion-parameter mixture-of-experts with 104B active parameters per token: 896 routed experts, 16 activated (about 1.8% of the pool), plus two shared experts that fire on everything. The stack is 93 layers, 69 of them KDA and 24 Gated MLA. Context window is one million tokens, and the model is natively multimodal through a 401M-parameter vision encoder Moonshot calls MoonViT-V2. The weights and the technical report went up together on Hugging Face and GitHub on July 27, 2026, under a bespoke revenue-tiered set of terms called the Kimi K3 License, as a single MXFP4 checkpoint with no separate base, instruct, or thinking variants. Note one spec correction the release forced: the roughly-50B active-parameter figure that circulated before launch was low by a factor of two. Moonshot still positions K3 as trailing the strongest proprietary models overall.

The attention claims break into three parts, and they have different evidential status:

1. Efficiency: up to 75% KV-cache reduction and up to 6.3x faster decoding at 1M-token context, versus an equivalent full-attention stack.
2. Quality: the hybrid matches or outperforms full attention, not just on short-context benchmarks but on long-context and reinforcement-learning training regimes.
3. Scale transfer: both of the above, demonstrated publicly at 48B parameters, carry to a 2.8T frontier model.

The first claim is close to arithmetic. If only one quarter of your layers maintain a KV cache that grows with context, and the other three quarters carry a fixed-size recurrent state, the memory math follows directly; the same logic drives every KV-cache lever covered in KV-cache engineering, where cache growth rather than weights is the binding constraint of long-context serving. The second claim is the interesting one. The third is the one the July 27 release changed the status of without settling, and it happens to be the exact step where the counter-evidence lives.

What Kimi Delta Attention Actually Is

Linear attention replaces the softmax lookup table with a fixed-size memory. Full attention keeps every previous key and value around and lets each new token compare against all of them, which is why it retrieves arbitrary earlier tokens exactly and why its cache grows without bound. A linear attention layer instead folds each token into a constant-size state matrix as it goes. Compute per token stays flat, memory stays flat, and retrieval becomes lossy by construction: the state can only give back what its update rule chose to keep.

The last decade of linear attention research is essentially a sequence of better update rules, and KDA sits at the current end of that line.

From delta rule to KDA

The delta rule is the family's key idea: instead of blindly accumulating key-value outer products (the failure mode of early linear attention, where the state fills with interference), each write first erases the old value associated with the incoming key, then writes the new one. Memory becomes something the layer edits rather than a pile it appends to. DeltaNet built a parallelizable layer around this rule, and Gated DeltaNet (Yang et al., ICLR 2025, arXiv 2412.06464) added a Mamba2-style forget gate on top, so the layer can also decay or clear state wholesale. That combination proved strong enough that Qwen3-Next shipped it at production scale in September 2025: 80B total, 3B active, Gated DeltaNet in three of every four layers, full attention in the fourth. Qwen3.5 inherited the same layout. The lineage is still moving; Gated DeltaNet-2 (arXiv 2605.22791) decouples the erase and write operations for finer memory control.

KDA, introduced in the Kimi Linear paper (arXiv 2510.26692, October 2025), is a refinement of Gated DeltaNet on one specific axis: where Qwen3-Next's gate is a scalar per attention head, KDA gates each feature channel independently. A head-level gate forgets everything or nothing at a given rate; a channel-level gate can hold one feature of a memory while decaying another. The paper pairs this with a specialized chunkwise kernel (a constrained variant of the Diagonal-Plus-Low-Rank transition formulation) that keeps the finer gating hardware-efficient. None of this is marketing framing; the mechanism, the kernels, and the checkpoints are public.

The 3:1 hybrid layout

Kimi Linear and K3 both alternate three KDA layers with one full-attention layer using Multi-head Latent Attention, DeepSeek's compressed-KV formulation. The released K3 config confirms the ratio the announcement described: 69 KDA layers and 24 Gated MLA layers across 93 total. The full-attention layers are the global lookup path: any token can still reach any other token exactly, once every four layers. The KDA layers carry the bulk of sequence mixing at linear cost.

This is worth naming plainly: the 3:1 hybrid is the same conclusion the state-space-model world reached when Jamba, Nemotron-H, Granite 4.0, and Falcon-H1 all independently kept a minority of attention layers because pure recurrent stacks kept failing associative recall. That piece makes the general argument (linear-time sequence models genuinely win throughput and memory, structurally lose associative recall, and therefore ship as hybrids); K3 is the same thesis run at the largest scale anyone has attempted, which is why it is worth grading claim by claim rather than accepting or dismissing wholesale. A fixed-size state, however cleverly gated, cannot guarantee verbatim retrieval of an arbitrary token seen 400K positions ago; the periodic full-attention layer is what restores that guarantee. So the precise claim K3 embodies is not "linear attention won." It is "a hybrid with a sufficiently expressive linear layer loses nothing, and the linear layer is now good enough to carry 75% of the stack." That is a smaller claim, and a much more defensible one.

The Evidence Audit

What the Kimi Linear paper actually shows

Credit first, because by vendor-architecture standards the Kimi Linear evidence is unusually good. The paper's comparisons pit the KDA hybrid against a full-attention MLA baseline and a Gated DeltaNet hybrid under the paper's fair-comparison protocol (matched training conditions rather than cherry-picked checkpoints), and Moonshot released the artifacts: 48B-total, 3B-active base and instruct checkpoints trained on 5.7T tokens with 1M context, plus open kernels. The reported numbers: on MMLU-Pro at 4K context, 51.0 for the hybrid at full-attention speed; on RULER at 128K context, 84.3 with a 3.98x speedup; at 1M tokens, up to 6.3x faster time-per-output-token than the MLA baseline. The paper also reports the hybrid holding up through RL post-training, which matters because reasoning-model training is where long generated sequences stress attention hardest, and it is the regime every frontier lab actually cares about in 2026.

Released weights, matched baselines, open kernels, and a claim tested across pretraining, long context, and RL: this clears the bar we set when DeepSeek's DSpark numbers shipped vendor-only, and it is more than Qwen published for the same layout. If every architecture claim arrived with this much checkable material, fact-check articles would be shorter.

What it does not show

Three gaps, in ascending order of importance.

First, the benchmark axes. RULER-style long-context retrieval is the friendliest possible test for a memory-constrained architecture, because needle tasks share surface vocabulary between query and needle. As we covered in the effective-context-length analysis, models that ace needle tests collapse on NoLiMa-style probes that remove the lexical shortcut, and that collapse is worst exactly where a bounded state has least room per token. A published NoLiMa-style, shortcut-free retrieval curve for KDA at 512K to 1M tokens does not exist yet. Until it does, "84.3 on RULER at 128K" is evidence about the easy version of the recall question.

Second, the quality deltas are parity-shaped, not dominance-shaped. The honest summary of the paper's tables is "matches full attention within noise on most axes, wins some, at much better efficiency." That is a genuinely strong result. But "beats full attention" in the release framing rounds up from parity-plus-efficiency, and parity measured on a benchmark suite is exactly the statement the next section's counterexample is about.

Third, and decisive: every matched-baseline number is from the 48B-A3B research model, and the July 27 technical report did not change that. K3 is 58 times larger in total parameters and 35 times larger in active parameters, trained on a different mixture, with additional architectural changes (Attention Residuals, reported at +7.5 on GPQA-Diamond and +3.6 on Minerva Math in its own technical report, arXiv 2603.15031, plus a new MoE stack running 896 experts with Quantile Balancing for stability under extreme sparsity). The report does carry the hybrid-ratio ablation, and it is a good one. On validation perplexity the 3:1 layout lands at 5.65, against 5.66 for 1:1, 5.70 for 7:1, 5.82 for 15:1, and 5.77 for pure full attention. That last number is the whole argument in one figure: the hybrid does not merely survive the swap, it edges the full-attention arm while dropping three quarters of the KV cache. The report also fits Chinchilla-style scaling curves across five model sizes and reports KDA reaching equal loss at roughly 1.16 times less compute than MLA, which is the closest thing anyone has published to evidence that the advantage does not erode as models grow.

Read the fine print on both, though. The ablation runs on the Kimi Linear testbed, not at 2.8T, so the matched full-attention comparison is still a research-scale result carried forward by assumption. And the scaling-law fit extrapolates a loss-versus-compute curve, which is precisely the metric that does not break. MiniMax's reported failure was never a perplexity failure; it was multi-hop reasoning degrading while the loss curves looked fine. Extrapolating the healthy metric does not bound the sick one. The load-bearing table, hybrid versus matched full-attention baseline at frontier scale, is not in the report, and in fairness it is the most expensive table any lab could be asked to produce: it requires training a 2.8T full-attention twin that nobody has a commercial reason to build.

The K3 benchmark table Moonshot published (93.5 on GPQA-Diamond, 91.2 on BrowseComp, 88.3 on Terminal-Bench 2.1, 67.5 on DeepSWE) is vendor-measured. Third-party reproduction became possible on July 27 and has barely started. As of this update the only independent number is Artificial Analysis's Intelligence Index, which puts K3 at 57 and fourth of 189 models on a nine-eval composite bought through the API. That is a real independent result and a strong one, but it measures the model, not the architecture: no aggregate index can separate what KDA contributed from what data, scale, and post-training contributed. Per the evaluation-crisis discipline, the vendor table is still a hypothesis. The architecture claim and the model claim were welded together in the announcement and stay welded together in the release; they should be evaluated separately.

The Counterexample Moonshot Has to Answer

The strongest reason for caution is not theoretical. It is that MiniMax already ran this play at scale and reversed it in public.

MiniMax-Text-01 (January 2025) was the first frontier-scale linear hybrid: Lightning Attention interleaved with full attention at 456B parameters. Ten months later, MiniMax shipped M2 with full attention in every layer and published an unusually candid postmortem titled "Why did M2 end up as a full attention model?" Their findings, compressed: the hybrid matched full attention on the standard benchmark suite at development scale, and then showed "clear deficits in complex, multi-hop reasoning tasks" that only became visible at larger scale. Their phrase for the general lesson is the one worth keeping: benchmarks are a leaky abstraction; reducing attention complexity extracts a hidden cost somewhere, and the suite you watch may not contain the task that pays it. They also flagged operational problems that never show up in papers: linear attention is far more sensitive to numerical precision than full attention, and the serving infrastructure (prefix caching, speculative decoding) assumed KV semantics that linear layers break. When MiniMax came back to efficient attention with M3, they chose sparse attention, not linear.

Two things keep this from being a refutation of K3. Lightning Attention is a weaker mechanism than KDA: no delta rule, no fine-grained gating, exactly the expressiveness gaps the DeltaNet line was built to close, so Moonshot can argue the deficit MiniMax hit was mechanism-specific rather than category-wide. And the infrastructure objections have aged badly, which the release made concrete: both vLLM and SGLang shipped day-0 K3 support on July 27, Moonshot upstreamed the KDA prefix-caching implementation into vLLM itself, and vLLM separated the physical KDA state-block size from prefix-match granularity so that partial prefix hits work on a hybrid stack at all. The tooling gap MiniMax described in late 2025 has largely closed. What has not closed is the precision objection, and the release sharpened it: K3 ships MXFP4 weights with MXFP8 activations, trained quantization-aware from SFT through RL, and the day-0 serving writeups put a floor of roughly 2e-3 on RL training KL divergence bounded by exactly that quantization precision.

But the core of the M2 result is precisely the step Kimi's public evidence has not yet covered: quality parity at research scale failing to survive the jump to frontier scale, on tasks (multi-hop reasoning over long context) that the standard suites under-sample. MiniMax measured the failure at hundreds of billions of parameters. Kimi Linear's parity evidence stops at 48B, and the K3 report's own ratio ablation stops there too. K3 is the scale where the question gets decided, and as of July 28 the deciding evidence is downloadable rather than published: the weights are public, the probes are runnable on eight Blackwell-class GPUs, and nobody has published the results.

Two Frontier Answers to the Same Problem

It clarifies the bet to notice that the two most architecture-aggressive open-model labs looked at the same enemy, the KV cache at million-token context, and chose opposite weapons. DeepSeek kept exact attention everywhere and made it cheaper: MLA compresses the cache, and V4's interleaved sparse design attends over compressed and selected subsets, landing at 27% of single-token FLOPs and 10% of the KV cache versus its predecessor at 1M tokens. Every token remains addressable; the risk is that the selector skips the token you needed. Moonshot instead replaced three quarters of the attention layers with a recurrent state; nothing is skipped at selection time, but history is compressed lossily on write, and the risk is that the gate decayed the fact you needed before you asked for it.

Neither lab shipped a pure design. DeepSeek's sparse layers lean on compressed-but-complete representations; Moonshot's linear layers lean on periodic exact-attention layers. Frontier long-context architecture in 2026 is a question of which failure mode you would rather manage, and both answers hedge. For the inference-stack consequences of that convergence, the inference optimization guide covers where each design pays off in serving.

The Adoption Protocol

K3's weights landed on July 27, which makes this checklist runnable rather than hypothetical. Here it is, in order. It applies equally to any linear-hybrid model, including Qwen3.5 deployments today.

1. Start from what the tech report's ablation section actually covers. The report shipped, and the answer is that its matched hybrid-versus-full-attention comparison sits at Kimi Linear testbed scale, supplemented by a five-size scaling-law fit, not at 2.8T. The architecture claim therefore still rests on research-scale evidence extrapolated upward. Budget your own evaluation accordingly rather than treating the report as the ablation.
2. Run a shortcut-free recall probe. NoLiMa-style retrieval (no lexical overlap between query and target) at 128K, 512K, and 1M, with the needle placed early in the context. This is the axis a bounded state fails first and needle tests hide.
3. Test multi-hop reasoning at depth, not just retrieval. Chains where hop N depends on a fact from hop 1, tens of thousands of tokens back. This is the specific deficit MiniMax reported emerging at scale.
4. Check verbatim copying of long spans. Ask for an exact reproduction of a 2K-token block seen 500K tokens ago. Exact copying is the purest bounded-state stressor and correlates with citation and code-editing fidelity.
5. Accept that you cannot diff against a reference precision, and plan around it. Linear-attention state is more precision-sensitive than KV attention, and K3 ships MXFP4 weights with MXFP8 activations as the only published checkpoint. There is no BF16 K3 to diff against, so the standard quantization control is simply unavailable: any long-context quality problem you find is jointly attributable to KDA and to 4-bit numerics, and released artifacts cannot separate them. Benchmark against a different model you already trust at your context lengths, and treat precision sensitivity as untested rather than cleared.
6. Verify your engine's feature parity for hybrid layers. Prefix caching, speculative decoding, and paged memory were built around KV semantics. Confirm your vLLM or SGLang version handles KDA layers in each path you rely on, at your batch sizes.
7. Benchmark at your context distribution. The 6.3x decode number is a 1M-token figure. At 8K average context, the KDA layers are saving memory you were not short of, and the win may be a rounding error.

Verdict

Fact-checked as three separate claims, and re-graded on July 28 against the released weights and technical report, the scorecard reads:

Efficiency: was established, stays established, now with production numbers behind it. The layout arithmetic held and the shipped config confirms it at 69 KDA layers to 24 Gated MLA. Both major serving stacks shipped day-0 support with real figures attached: a constant KDA state of roughly 54 MB per sequence against MLA's roughly 27 KB per token, and SGLang reporting logical KV capacity rising from 1.5M to 12.2M tokens on identical hardware under its decode-context-parallel path. One caveat survives the release: the headline 6.3x decode figure is still the 48B Kimi Linear number, and no matched 2.8T speedup ratio has been published, because the full-attention twin you would measure against does not exist.

Quality parity at 48B-A3B: was established, stays established, and the release did not extend it. The matched-baseline evidence, including the pure-full-attention arm at 5.77 validation perplexity against the 3:1 hybrid's 5.65, is real and remains the strongest case any lab has published for a linear-attention layout. The technical report restated this evidence at the same scale rather than adding to it higher up.

Scale transfer to 2.8T: moves from unverifiable to checkable, and stops there. This is the grade the release changed, and it changed less than the occasion suggests. What July 27 settled is shippability: a 3:1 linear hybrid trains, converges, and serves at 2.8 trillion parameters. That was not a given, and it is a genuine result that the MiniMax reversal made look unlikely. What it did not settle is whether quality transferred. The only independent measurement so far is an aggregate index, which grades the model and cannot separate the architecture's contribution from data, scale, and post-training. No shortcut-free retrieval curve at 512K to 1M has been published by anyone. No multi-hop-at-depth result has either. And the release quietly added an obstacle to ever settling the question cleanly: with MXFP4 as the only published checkpoint, the architecture claim and the precision claim are welded together, so a long-context failure cannot be pinned on KDA rather than on 4-bit numerics using released artifacts alone. Weights make a claim checkable. Only measurement makes it verified, and that measurement has not been done.

The MiniMax counterexample stands unamended. Nothing published since the weights dropped speaks to the specific failure mode M2 reported, which means the case for caution rests exactly where it did on July 19, on evidence nobody has gone looking for yet.

The 3:1 ratio remains the most honest sentence Moonshot has published about linear attention: three layers of trust, one layer of insurance. The insurance layer is load-bearing, and everyone shipping this architecture knows it.

Key Takeaways

1. Kimi K3 (2.8T total, 104B active, 1M context) runs Kimi Delta Attention in 3 of every 4 layers, 69 KDA against 24 Gated MLA across 93 layers in the shipped config; the claimed payoff is up to 75% less KV cache and up to 6.3x faster decoding at 1M tokens.
2. KDA is a gated-DeltaNet refinement: it upgrades Qwen3-Next's per-head forget gate to per-channel gating, so the fixed-size state can decay individual features instead of whole memories.
3. The public evidence is real but still stops at 48B. Kimi Linear's matched-baseline comparisons, released checkpoints, and RL-regime results are unusually strong for a vendor claim, and the July 27 technical report restated them at the same scale rather than above it; its ratio ablation and five-size scaling-law fit are research-scale evidence extrapolated upward, not a 2.8T measurement.
4. The claim is parity-plus-efficiency, not dominance. The 3:1 layout itself concedes that periodic exact attention is still required; a pure linear stack at frontier scale remains unshipped by everyone.
5. MiniMax's M2 reversal is the standing counterexample: hybrid linear attention matched benchmarks at development scale, then showed multi-hop reasoning deficits at larger scale, and M2 shipped full attention. Benchmark parity is a leaky abstraction.
6. The under-measured axes are shortcut-free recall and multi-hop at depth. RULER-style needle scores flatter bounded-state architectures; NoLiMa-style probes and long-range dependency chains are where the design either holds or breaks.
7. Serving infrastructure has caught up: vLLM and SGLang both shipped day-0 K3 support on July 27, with Moonshot upstreaming KDA prefix caching into vLLM itself, which closes most of the 2025-era operational case against linear hybrids.
8. The release makes the 2.8T claim checkable, not verified, and complicates checking it. Public weights mean anyone with eight Blackwell-class GPUs can run the probes, but the only independent result so far is an aggregate index that cannot isolate the architecture, and because MXFP4 is the sole published checkpoint there is no reference precision to diff against, so KDA quality and 4-bit numerics cannot be separated with released artifacts.
9. Before adopting any linear-hybrid model, run the seven-step protocol above on your own context distribution; the 6.3x figure is a 1M-token number that may be worth nothing at 8K.