Sparse Attention in 2026: Why It Finally Had to Be Native

Sparse attention spent a decade as a benchmark trick that fell apart in production. Somewhere between February 2025 and now it stopped doing that, and the design change responsible is not a new sparsity pattern. It is where the sparsity gets introduced: trained in from pretraining, not bolted on at inference. Every long-context attention mechanism that shipped inside a frontier model in the last eighteen months (DeepSeek's NSA and the DeepSeek Sparse Attention line that now runs in V4-Pro and V4-Flash, Moonshot's MoBA, MiniMax's MSA in M3) is natively trainable sparse attention. The model learns with the sparse pattern in place, and the gradient flows through the selection.

That distinction is why a decade of "approximate attention" papers never replaced full attention while this generation sits inside deployed models. It is also more subtle than the slogan, because post-hoc sparsity is not useless: it ships too, in narrower conditions. The interesting question is what native training buys that no amount of inference-time cleverness recovers.

This is the reality-gap pattern at the architecture level. Research showed sparse attention works, real systems behaved differently, and the gap turned out to be a question of when the sparsity was introduced. The piece sits next to our DeepSeek V4 attention deep dive, which takes one specific hybrid design apart. Here the goal is the general landscape and the principle that separates the methods that ship from the ones that benchmark.

Why sparsity had to become native

Why full attention is the thing to beat

The standard attention layer compares every token to every other token, so compute grows with the square of sequence length and the KV cache (the stored keys and values you attend back over) grows linearly with context. At long context this is the dominant cost in both directions: FLOPs during prefill, and the memory that inference optimization practice treats as the real serving bottleneck. The promise of sparse attention is obvious. Most token-to-token comparisons carry little information, so if each query attends to a well-chosen subset of keys instead of all of them, you recover most of the quality at a fraction of the cost.

The promise is old: sliding windows, fixed strided patterns, low-rank approximations, locality-sensitive hashing. The recurring disappointment is equally old. A model trained with full attention and then run with a sparse pattern degrades, often badly, on exactly the long-context retrieval tasks sparse attention was supposed to enable. The pattern looks fine on perplexity and falls over on the task. Understanding why is the key to the current designs.

What post-hoc sparsity actually costs

When you train a model with full attention, its weights learn to rely on full attention. The attention distributions, the way information routes through layers, the redundancy the model builds in: all of it assumes every token can see every other token. Impose a sparse mask afterward and you are running the model out of distribution. The tokens it would have attended to are masked, and it never learned to compensate. The gradient never saw the sparse pattern, so there was no pressure to make the kept tokens sufficient.

The honest version of this is narrower than the slogan, and the best evidence is the largest controlled study of training-free sparse attention to date. Nawrot et al., "The Sparse Frontier" (arXiv 2504.17768), swept Qwen2.5 models from 4B to 72B, sequences from 16K to 128K, and sparsity up to 0.95. Four findings are worth carrying: at long enough sequences an isoFLOPS comparison favors larger-and-sparser over smaller-and-dense; decoding tolerates considerably more sparsity than prefilling, and the tolerance grows with model size; no single sparsification strategy wins across tasks and phases; and even moderate sparsity produces significant degradation on at least one task in the suite. That last one matters most. Training-free sparsity does not fail on average, it fails somewhere specific, and which somewhere depends on your task.

That is why post-hoc methods are genuinely deployed rather than merely published. Qwen2.5-1M ships a MInference-derived Vertical-Slash prefill sparsity, reporting 3x to 7x faster processing of 1M-token inputs (vendor-reported; the team notes the unmodified MInference configuration degraded badly past roughly 400K tokens before they refined the sparsification and position handling). Vertical-Slash is now in vLLM. If your problem is prefill latency on very long inputs against a model you did not train, this is a real lever.

What post-hoc sparsity cannot do is change what the model needed in the first place. It is a decision made after the weights are frozen, so it can only search for redundancy the model happens to contain, tuned per task and validated per deployment. Native sparsity is a different kind of object: the pattern is a structural property the weights were built around.

What native training actually buys

Three things, and they are worth separating because vendors blur them.

The kept set is optimized, not assumed. With the selector in the training loop, the gradient can push the model to route information through tokens the sparsity keeps. Post-hoc, you inherit whatever redundancy is already there.

The speedup covers training, not just serving. A method that is only sparse at inference still pays full quadratic cost during pretraining. A natively trainable one is faster in prefill, decode, forward, and backward. NSA's headline result is exactly this: substantial speedups on 64K-length sequences across decoding, forward propagation, and backward propagation, with the pretrained sparse model matching or exceeding full-attention baselines on general, long-context, and instruction-reasoning benchmarks (DeepSeek's own measurements).

The sparsity can be aggressive. Post-hoc methods are bounded by how much redundancy the dense model happens to have. Native methods set the budget and train into it, which is how MSA reaches a claimed 28.4x reduction in per-token attention compute at 1M context without the quality collapse a training-free method at that ratio would produce.

Only the first of those three requires a full pretraining run, which is why the "native" line is really about the gradient, not the budget.

The shape they all landed on

The designs that shipped

Five, and they share a structure worth naming: a cheap mechanism decides which tokens matter, then expensive attention runs only over those. Call it the selector-plus-dense-inner-loop pattern.

Sliding-window attention is the boring baseline that everyone skips, and it is trained-in native sparsity of the simplest kind. Gemma 3 interleaves five local layers (1024-token window) per global layer, a 5:1 ratio tightened from Gemma 2's 1:1 with a 4096 window, and Google's ablations report only minor quality impact for substantial memory and compute savings. GPT-OSS 120B uses alternating local-global layers too. There is no learned selector here, just a fixed pattern chosen before training. It is the cheapest thing that works, it is in production at scale, and any new sparse design that cannot beat it on your workload is not worth the kernel maintenance.

NSA (Native Sparse Attention, DeepSeek, arXiv 2502.11089) is the reference design for learned selection. Three parallel branches feed a learned gate: coarse compression (token blocks mean-pooled into single entries, preserving global context), fine-grained selection (top-k blocks chosen using the compression branch's own attention scores), and a sliding window for local precision. Its second claim is hardware alignment, meaning the sparse pattern is designed to map onto how GPUs actually move memory rather than only to reduce theoretical FLOPs, implemented in Triton kernels. Worth stating plainly: NSA is a published architecture, not a released model. Its ideas shipped through its successor.

DeepSeek Sparse Attention (DSA) is the version that left the lab, and it is now two generations deep. It is built around a "lightning indexer," a small FP8, few-head network that scores preceding tokens cheaply and selects the top-k key-value entries before the expensive attention runs, turning the layer into a retrieve-then-attend operation. Because the indexer runs in low precision with few heads, its own cost stays minor against the dense attention it replaces. The public evidence is strongest from its first deployment in V3.2 (September 2025, top-k of 2048): roughly 3x to 6x long-context cost reduction at 128K with benchmark parity, and sparse and dense training curves tracking each other on BrowseComp and SWE-bench Verified across 1400+ steps, which is a more informative artifact than a final-score table. Both figures are DeepSeek's own. The current deployment is V4-Pro and V4-Flash (MIT, April 2026), where the same retrieve-then-attend mechanism runs over a compressed cache as Compressed Sparse Attention; that generation is the reason to treat native sparsity as production-grade rather than experimental, and the V4 deep dive takes its specifics apart.

MoBA (Mixture of Block Attention, Moonshot, arXiv 2502.13189) carries the mixture-of-experts intuition into attention: partition the context into blocks, use a gating mechanism to route each query to the most relevant blocks, attend fully within those, skip the rest. It follows what the authors call a "less structure" principle, letting the model decide where to attend instead of imposing a predefined bias, and it can transition between full and sparse attention without retraining, which is a real deployment convenience. It was deployed to serve Kimi's long-context requests, and the code is open.

MSA (MiniMax Sparse Attention, arXiv 2606.13392, June 2026) is the newest and the most interesting one, because of who shipped it. MSA is blockwise sparse attention built on a grouped-query-attention backbone: a lightweight index branch scores and selects top-k KV blocks per GQA group, and a main branch runs exact block-sparse attention over the selection. The co-design is where the work is, with an exp-free top-k selection and a "KV-outer-gather-Q" access pattern chosen for how the kernel actually touches memory. MiniMax reports 28.4x lower per-token attention compute at 1M context, 14.2x prefill and 7.6x decode wall-clock speedup on H800, and quality on par with GQA. Those are vendor numbers on vendor hardware, but the kernels and the model (MiniMax-M3, 428B total, roughly 23B active, 1M context, June 1, 2026) are public, so they are checkable in a way most architecture claims are not.

The convergence is the signal. Four independent groups arrived at "learn a cheap selector, attend densely over the selection, train the whole thing together." When designs converge under competitive pressure, the shared structure is usually the load-bearing idea.

The selector is the whole ballgame

Once you accept the pattern, every remaining question is about the selector, and this is where current work concentrates. A recent example makes the point sharply: "Hierarchical Sparse Attention Done Right" (Hu et al., arXiv 2607.02980, July 2026) argues that existing chunk-wise methods fall short of full attention specifically because their chunk selection is inaccurate, and proposes learning the selection end to end through the language-modeling loss. It reports extrapolation past 64x the training context length at 90% retrieval accuracy, plus conversion of existing full-attention models through lightweight continued pretraining. Treat the numbers as a preprint's own until reproduced, but the framing is right: the compute story is settled, and the open problem is selection quality.

That also settles what "native" means, and it is not a binary. The strict version is pretraining from scratch with the pattern in place, which is what NSA, DSA, MoBA and MSA describe. The cheaper version is continued pretraining on a dense checkpoint until the model adapts to the selector, which is what the conversion results above claim and what most teams can actually afford. What both share, and what pure inference-time masking lacks, is that the gradient sees the sparse pattern at some point. That is the line.

Not every "sparse" claim is the same bet

A different axis: differential attention

Not every advance with "sparse" attached to it is about efficiency, and conflating the two is the most common error in this literature. The Differential Transformer (Microsoft, arXiv 2410.05258, ICLR 2025) computes attention as the difference between two separate softmax attention maps. The subtraction cancels common-mode noise the way a differential amplifier does, suppressing attention to irrelevant context. The resulting patterns are sparser and cleaner, but the objective is quality and robustness (less attention to distractors, better key-information retrieval, less hallucination attributable to attention noise, fewer activation outliers), not a smaller compute bill.

The follow-up work sharpens the distinction rather than blurring it. DIFF Transformer V2 (Microsoft, January 2026) reworks the mechanism for stability and kernel compatibility, not the cost model: it doubles query heads while holding KV heads constant so standard FlashAttention works without custom kernels and decoding speed matches a baseline Transformer, removes the per-head RMSNorm that caused gradient instability at scale, and replaces the globally shared lambda with a token-specific projection. The reported gap is a 0.02 to 0.03 lower language-modeling loss at 1T training tokens, which the authors present as in-progress rather than final, with no shipped model behind it. The honest read in mid-2026: differential attention is a well-motivated research line with no frontier deployment attached, a different evidential status from DSA or MSA.

Here is the part worth stating as a position, because it inverts the article's own thesis: differential attention is the one attention rework that retrofits onto a pretrained model, precisely because it is not an efficiency mechanism. DEX (Understanding Differential Transformer Unchains Pretrained Self-Attentions, NeurIPS 2025, arXiv 2505.16333) adds a lightweight differential operation on the output-value matrix while reusing the existing softmax scores, and reports substantial gains across benchmarks with adaptation data under 0.01% of pretraining. Retrofitting works here because you are adding a filter on top of information the model already computed. It fails for sparse attention because there you are removing information the model learned to depend on. Adding signal is cheap after the fact; removing tokens is not.

So the first question to ask of any sparse-attention claim is which axis it sits on. Are they buying speed, or buying signal? A paper that improves one tells you nearly nothing about the other.

Sparse or linear: the fork that defined 2026

The clearest way to see where sparse attention landed is to notice who bet against it. Long-context architecture in 2026 split into two answers to the same enemy (the KV cache at million-token context): compress and sparsify exact attention, or replace most layers with a fixed-size recurrent state. Moonshot took the second road for its flagship, running Kimi Delta Attention in three of every four layers in K3, whose weights and 47-page technical report landed on July 26-27, 2026. We fact-checked that claim separately in the Kimi K3 analysis.

MiniMax is the interesting trajectory, because it went both ways and published the reasoning. MiniMax-Text-01 was a linear-attention hybrid; M2 reverted to full attention in every layer after the hybrid showed multi-hop reasoning deficits that appeared only at larger scale; M3 came back to efficient attention through MSA, which is sparse, not linear. One lab, three architectures, three public positions. That is a stronger signal than any single vendor benchmark, and it points at sparse attention as the lower-risk of the two efficiency bets, because nothing is compressed away on write. Every token stays addressable; the risk is only that the selector does not select it.

Notably, Moonshot itself has now shipped both: MoBA for long-context serving, KDA in K3. Neither lab landed on a pure design, and neither treats the choice as settled.

Putting a sparse-attention claim to the test

What this does not solve

The tradeoffs are specific.

Native training is a commitment. You cannot take a frontier dense model and get native-sparse benefits by flipping a flag. Continued pretraining is the cheap path, and it is still a training run with an eval budget attached, which is why these designs live in labs that own their base models.

Sparsity is a bet about where the information is, and the bet can be wrong for your distribution. A selector tuned on general long-context data may drop exactly the tokens your domain depends on. This is the effective-context-length gap that already separates advertised windows from usable ones, now mediated by what the selector chose to keep. And per the Sparse Frontier result, the damage shows up on one task, not on the average, so an aggregate score is exactly the wrong instrument for finding it.

The efficiency win is a long-context number. It is largest where you need it and smallest where you do not. At short context, full attention is cheap and the selector's own overhead can make sparse attention a wash or worse. Read the speedup at the sequence length you actually run, not the one in the abstract.

Retrieval benchmarks flatter selectors. Needle-in-a-haystack tasks share surface vocabulary between query and target, which is precisely the signal a cheap indexer scores well on. Strip the lexical shortcut and the selector has a harder problem than the benchmark implies. This is the evaluation-crisis discipline applied to an architecture claim: the benchmark that looks hardest for the mechanism is usually the one nobody ran.

An evaluation protocol for a sparse-attention claim

Whether you are choosing a base model or reading a paper, the same seven steps separate the claim from the result. Run them in order.

1. Classify the axis first. Efficiency (NSA, DSA, MoBA, MSA, sliding window) or quality (differential attention)? A speed number and a robustness number are not comparable evidence, and vendors mix them freely.
2. Establish whether the sparsity was in the gradient. Pretrained with the pattern, continued-pretrained into it, or masked at inference? These are three different products. Only the first two let you trust an aggressive sparsity ratio.
3. Set sliding window as your baseline, not full attention. Gemma 3's 5:1 local-global interleave is nearly free to implement and already in production. A learned selector has to beat it on your workload, not just beat dense attention on cost.
4. Read the speedup at your sequence length and your phase. Prefill and decode have different sparsity tolerances (decode tolerates more, and the gap widens with model size). A 1M-token prefill figure tells you nothing about decode at 32K, which may be the regime you actually serve.
5. Probe retrieval without the lexical shortcut. Needle tests overstate selector quality. Use targets that share no surface vocabulary with the query, placed early in the context, at the lengths you run.
6. Test multi-hop dependency at depth, not just single-fact recall. One selector miss inside a chain kills the whole chain, so failures compound in a way single-fact retrieval never surfaces. This is the axis where MiniMax's linear hybrid broke, and it under-samples in every standard suite.
7. Report per-task deltas, never the suite average. Moderate sparsity degrades at least one task while the mean holds. If you only publish the mean, you have not measured the thing that will page you.

Where this leaves the argument

If you train models at long context, native sparse attention is a credible default rather than a research gamble, and the deployed DeepSeek V4 and MiniMax M3 generations are the existence proofs. If you consume models, prefer base models that built sparsity in, treat training-free sparsity as a prefill-latency tool with a task-dependent envelope, and benchmark long-context retrieval on your own data.

The decade-long failure of sparse attention was not that the idea was wrong. The sparsity arrived too late, after the model had already learned to need everything. Training it in is what fixed it, and that is the line separating the methods that ship from the ones that benchmark. It is the same pattern that runs through the Transformer architecture story and the KV-cache engineering constraints that make long context expensive to begin with: the deployment-grade design is rarely one sequence-mixing strategy applied after the fact.

Key Takeaways

1. The fix was native sparsity, not a new pattern. Sparse attention failed for years because it was bolted onto dense-trained models at inference. The designs that ship (NSA, DSA, MoBA, MSA) train the sparse pattern in, so the gradient optimizes the kept set instead of assuming it.
2. Post-hoc sparsity is narrower, not useless. The Sparse Frontier (arXiv 2504.17768) shows training-free sparsity is viable at long context, that decoding tolerates more of it than prefilling, and that isoFLOPS favors larger-and-sparser models. But even moderate sparsity degrades at least one task, so it is a per-deployment tuning problem, not a free upgrade. Qwen2.5-1M ships exactly this for prefill.
3. Native means the gradient saw the pattern, not that you pretrained from scratch. Continued pretraining a dense checkpoint into a selector is the affordable path and counts. Inference-time masking alone does not.
4. The shipped methods share one structure. NSA (compress, select and window under a learned gate), DSA (FP8 lightning indexer, top-k retrieve-then-attend, first in V3.2 and now in DeepSeek V4), MoBA (MoE-style block routing, serving Kimi long context), MSA (blockwise top-k per GQA group, in MiniMax M3). All learn a cheap selector, then attend densely over the selection.
5. Sliding window is the baseline any new design must beat. Gemma 3's 5:1 local-global interleave with a 1024-token window is trained-in sparsity with no learned selector, in production at scale, with minor reported quality impact. Benchmark against it, not against dense attention.
6. The selector is the open problem. Compute reduction is solved; selection accuracy is not. Recent work (arXiv 2607.02980) argues chunk-selection error is why sparse methods still trail full attention, and needle-style benchmarks flatter selectors by sharing vocabulary between query and target.
7. Differential attention is a different axis, and the one thing that retrofits. Subtracting two softmax maps buys signal, not speed, and DIFF V2 (January 2026) has no shipped model behind it. DEX (NeurIPS 2025) grafts it onto pretrained weights with under 0.01% adaptation data, which works because adding a filter is cheap after the fact while removing tokens is not.
8. MiniMax's trajectory is the strongest available evidence, and it points at the selector risk. Linear hybrid, then full attention after multi-hop deficits appeared at scale, then sparse via MSA. A sparse design compresses nothing away on write, so the only failure is a selector miss. Read the speedup at your sequence length and report per-task deltas; run the seven-step protocol before believing either number.

The Acing AI newsletter covers attention research for the part the abstract skips: where the speedup is measured and what it cost. Subscribe for the grounded version.