State Space Models in 2026: The Recall Gap, and What Finally Closed It

State space models were supposed to retire the Transformer. Two and a half years after Mamba, the honest scorecard is narrower and more interesting: linear-time sequence models won the throughput and memory argument, lost the in-context recall argument on their own, and the production answer that emerged across 2025 and 2026 is not "SSM instead of attention" but a hybrid stack that keeps a minority of attention layers and replaces the rest. State space models are now a component, not a replacement. The reason is a measurable capability gap, and the reason the gap stopped being fatal is a specific, published ratio.

This is the reality-gap version of a debate that the "attention is quadratic, therefore doomed" framing keeps getting wrong. The quadratic cost is real and the linear-time alternative is real, but the part that decides architecture choices is what a bounded recurrent state can and cannot do, which is exactly the part the throughput charts leave out. The arc mirrors the one in our Transformer architecture explainer: the design that wins is the one that survived its own failure modes, not the one with the prettiest asymptotics.

The trade: throughput bought, recall lost

What a recurrent state actually buys you

A Transformer's attention compares every token to every other token, so compute grows with the square of sequence length and the KV cache grows linearly with context until it dominates serving memory. That cache, not the weights, is the binding constraint of long-context serving, which is the argument KV-cache engineering makes at length. A state space model instead carries a fixed-size recurrent state forward through the sequence, folding each new token into that state through a structured linear recurrence. Mamba's contribution was making the recurrence input-dependent (selective) while keeping it parallelizable at training time, so it trains like a convolution and runs like an RNN.

The consequences are concrete. Compute scales linearly with sequence length. The "cache" is a constant-size state rather than one that grows with every token, so memory at long context is flat instead of climbing. That is why the throughput numbers look the way they do. NVIDIA's Nemotron-H (arXiv 2504.03624), which swaps most attention layers for Mamba-2 blocks at 8B, 47B, and 56B, reports up to roughly 3x faster inference than a similarly sized Transformer at matched or better accuracy. IBM reports over 70% lower memory and roughly 2x faster inference for Granite 4.0 in long-context and multi-session serving. Both figures are the vendor's own, measured on the vendor's stack, and both are structurally plausible rather than surprising: you removed the thing that grows.

One clarification that saves a lot of confused reading. The "SSM versus linear attention" split has mostly dissolved. Mamba-2 established the duality between selective state space models and a form of masked linear attention, and the DeltaNet line arrived at the same object from the other direction. In 2026 both families are gated linear recurrences with a matrix-valued state, differing in how the state is written and decayed rather than in kind. Mamba-3 (Lahoti et al., ICLR 2026, arXiv 2603.15569) pushes the SSM side further with trapezoidal discretization, a complex-valued update that improves state tracking, and a MIMO formulation for arithmetic intensity, reporting comparable perplexity to Mamba-2 at half the state size. RWKV-7 "Goose" (arXiv 2503.14456) got there from the RNN side with a generalized delta rule, vector-valued gating, and in-context learning rates, and its 2.9B model matched or beat 3B-class baselines on far fewer training tokens. RWKV-8 is in development and should be treated as unreleased. When you read "linear attention" in one paper and "state space model" in another, assume they are arguing about the same machine.

The catch is what a fixed-size state can hold.

The recall gap is the load-bearing weakness

Compress an unbounded history into a fixed-size state and you have, by construction, a lossy memory. The question is what gets lost, and the literature converged on a sharp answer: associative recall, the ability to retrieve a specific earlier token verbatim. "The API key was X; what was the API key?" is the canonical form. Attention does this for free because it can address any prior position directly. A recurrent state has to have decided, at the moment it read that token, to keep it, and to not have overwritten it since.

The benchmarks make this precise. On associative-recall and copying tasks, which correlate strongly with language-modeling quality, controlled studies find Transformers and a few attention-like variants succeeding while pure recurrent models lag, with Mamba and DeltaNet close behind and earlier designs (H3, Hyena) failing outright. Work on the gather-and-aggregate mechanism (arXiv 2504.18574) traces the gap to a specific circuit: both families rely on gather heads that pull relevant pieces out of context and aggregate heads that fuse them, and recurrent models struggle to form sharp versions of both. This is not a sampling temperature you tune your way out of. It is what a bounded state cannot represent.

It also interacts badly with the thing SSMs were sold on. The pitch is long context, but long context is exactly where you most need to retrieve a specific earlier fact, and a fixed state has less room per token the longer the sequence gets. The effective-context-length problem that already haunts Transformers, where the usable window is a fraction of the advertised one, is sharper for a pure recurrent model. Throughput holds at 100K-plus tokens; precise retrieval over that span is the weak axis, not the strong one.

What actually closed the gap

Here is the part that has changed since the pure-versus-hybrid argument started, and the part most 2024-era explainers still get wrong. The gap is not closed by making the recurrence better. It is closed by how many attention layers you leave in, and the field now has a number.

A systematic study of hybrid linear attention (arXiv 2507.06457) trained matched hybrids across mechanisms and ratios and found the useful band sits between 3:1 and 6:1 linear-to-full. Language-modeling quality is nearly flat across ratios, so perplexity tells you almost nothing here. Recall is what moves: at 3:1, most of the tested architectures approach or exceed the full-attention baseline on RULER, at roughly 4x to 7x less KV cache. The study also isolates which memory mechanisms matter, and the answer is not "the biggest state." It is selective gating (learned token-wise gates, as in Gated DeltaNet and HGRN-2), hierarchical recurrence (a coarse summary path alongside a token-detail path, which roughly doubled recall in their ablation), and controlled forgetting (delta-rule erase-before-write or gated decay, which prevents the state crowding that sinks naive linear attention). A memory you can edit and clear beats a memory you can only append to.

Two caveats before anyone quotes that as "solved." Those hybrids were trained at research scale, and RULER is a needle-style benchmark whose queries share surface vocabulary with the needle, which is the friendliest possible test for a bounded state. Matching a Transformer on RULER at 3:1 is a real result and a partial one.

The verdict from production

Everyone shipped the same shape

The fix that shipped is boring and effective: keep attention where it earns its cost, use recurrent layers everywhere else. You pay quadratic cost on a minority of layers instead of all of them, and you recover most of the throughput and memory win without surrendering retrieval. What makes this more than an aesthetic preference is that labs with no shared codebase, no shared training data, and competing commercial interests all landed inside the same band.

The pattern across all of them is the same admission: attention is doing something the recurrence cannot, and the cheapest place to buy it is a handful of layers. AI21, NVIDIA, IBM, TII, Zyphra, Alibaba, and Moonshot did not converge on hybrids out of shared taste. They converged because the pure-recurrent ablations underperformed on recall-heavy evaluation and the hybrid ablations did not. Convergence under competition is the strongest evidence available about where the line sits.

Note also what nobody shipped. Falcon-Mamba (TII, 2024) remains the notable pure-SSM production attempt at 7B, and it has no frontier-scale successor from anyone. Two and a half years of well-funded effort has produced exactly zero pure recurrent models above the mid-size range. That is not an accident of research fashion.

The frontier-scale test, and what it settled

The general thesis above is now stated at frontier scale, which is why this piece has a companion. Moonshot's Kimi K3 is the 2.8-trillion-parameter version of the same bet, and we fact-checked its architecture claim in detail in Linear Attention at Frontier Scale: Kimi K3's KDA Claim. Read that piece for the claim-by-claim audit; here is only what the released weights change about the general argument.

The weights and the technical report landed on schedule in late July 2026, so K3 is no longer an announcement. Three things it does settle. First, the layout ships: a 3:1 linear-to-full stack trains stably and serves at 2.8T total parameters with a 1M context, with day-zero support in vLLM and SGLang. Second, the report publishes a ratio ablation, and it is the most useful artifact in the release: the optimum sits at 3:1 (validation perplexity 5.65, against 5.66 at 1:1, 5.70 at 7:1, 5.82 at 15:1, and 5.77 for a pure full-attention arm), which lands squarely inside the 3:1 to 6:1 band the smaller matched studies found. Read the fine print, though, because it is easy to overclaim here. That ablation runs on the Kimi Linear research testbed, not at 2.8T, and the report's other supporting evidence is a Chinchilla-style fit across five model sizes reporting KDA reaching equal loss at roughly 1.16 times less compute. Both are research-scale results extrapolated upward, not frontier-scale measurements. The load-bearing table, hybrid against a matched full-attention baseline at 2.8T, is not in the report, and in fairness nobody has a commercial reason to train the full-attention twin it would require. Third, the release adds a mechanism the SSM literature had not been emphasizing: Attention Residuals (Kimi Team, arXiv 2603.15031), which lets a layer read selectively from earlier layers instead of only the one below it, reported to match a baseline trained with 1.25x more compute. The recall problem, it turns out, has a depth axis as well as a sequence axis.

Now the honest limits, because the weights settle less than the launch coverage implies. K3 is a hybrid, so it confirms the hybrids-won thesis rather than reopening the pure-SSM case; the quarter of layers that stayed exact are the insurance, and Moonshot kept them. The specific failure mode that has historically killed linear hybrids at scale, multi-hop reasoning deficits that benchmark suites under-sample, is not resolved by a benchmark table, and no shortcut-free retrieval curve for KDA at 512K to 1M tokens has been published by anyone. And K3 ships MXFP4-quantized with quantization-aware training from the SFT stage onward, with no higher-precision checkpoint to fall back on, which means the architecture claim and the 4-bit precision claim arrive welded together. You cannot validate one without the other on the artifact you can actually download. Treat the release as the start of the evaluation rather than the end of it, per the standing evaluation-crisis discipline.

Two ways the gap comes back

Convergence is not consensus, and two results keep the question live.

The first is MiniMax, which ran the frontier-scale linear-hybrid play at 456B with MiniMax-Text-01 and then reversed it in public, shipping M2 as a full-attention model with a postmortem: benchmark parity at development scale, multi-hop reasoning deficits at larger scale, and serving infrastructure that assumed KV semantics linear layers break. Their general lesson is the one worth keeping, that benchmarks are a leaky abstraction and reducing attention complexity extracts a cost somewhere the suite you watch may not measure. When MiniMax came back to efficient attention with M3, it chose block-sparse attention over a GQA backbone rather than linear recurrence. The K3 fact-check works through how much that constrains the linear-hybrid case; the short version is that Lightning Attention lacks both the delta rule and fine-grained gating, exactly the properties the ratio study found decisive, so the reversal narrows the claim rather than killing it.

The second is newer and less discussed, and it is the result I would put in front of anyone about to adopt a hybrid. "Attention Amnesia in Hybrid LLMs" (Zhou et al., June 2026, arXiv 2606.11052) reports that chain-of-thought fine-tuning degrades long-range recall in hybrid models, with the damage scaling by how far back the target sits. The post-training stage that everyone now runs to make a model reason can quietly suppress the pathway that made the hybrid safe to use at long context, and the authors propose a targeted repair (QK-Restore) rather than treating it as a data problem. The operational consequence is blunt: a recall number measured on the base checkpoint does not transfer to the reasoning checkpoint you actually deploy. The recall gap is not a property an architecture has once. It is a property a specific checkpoint has, after everything you did to it.

Deciding for your own workload

Where a recurrent state is still the right machine

Naming the weakness is not dismissing the architecture. The tradeoff cuts both ways, and there are regimes where the recurrent state is straightforwardly the better machine.

The clearest is throughput-bound generation over long inputs where verbatim recall is not the task: streaming, summarization where gist matters more than exact quotation, audio and other continuous signals, and on-device settings where the flat memory profile decides whether the model runs at all. Zamba2-VL is a good illustration of the shape, reporting roughly an order of magnitude lower time-to-first-token on a 32K prefill against dense-Transformer VLM backbones, which is the vendor's number but the right kind of number: the win is concentrated in prefill and memory, exactly where the structural advantage lives.

The second is as a component in a larger system. A hybrid paired with retrieval sidesteps the recall gap rather than solving it: you do not ask the recurrent state to memorize a fact you can fetch from an index. The architecture's weak axis, exact lookup, is precisely what an external store is good at.

What you should not do is treat a throughput benchmark as an architecture verdict. A model that is 3x faster and quietly worse at retrieving a value you placed 40K tokens ago has not dominated the Transformer; it has made a trade, and the only way to price that trade is to measure recall on your own data.

The recall audit: seven steps you can run this week

This is the protocol for deciding whether a hybrid is safe for your workload. It applies to Kimi K3, Qwen3.5, Nemotron 3, Granite 4, and anything else with a linear-to-full ratio in its config. Run it against a full-attention model of comparable capability as the baseline, because a number without a baseline is not a result.

1. Read the config, not the marketing. Find the linear-to-full ratio and the gating mechanism in the model config. Below roughly 6:1 with a delta-rule or gated-decay memory, the published evidence says recall is recoverable. Above that, or with an ungated accumulate-only linear layer, assume recall is the risk and weight the rest of this protocol accordingly.
2. Build a shortcut-free retrieval probe. Needle tests where the query shares vocabulary with the needle flatter bounded-state architectures. Follow the NoLiMa construction: paraphrase the needle so there is no lexical overlap with the query, and place it early in the context. Measure at your real context lengths, not at 4K.
3. Test multi-hop at depth, not just retrieval. Chains where hop N depends on a fact introduced tens of thousands of tokens earlier. This is the specific deficit MiniMax reported emerging at scale, and it is the axis standard suites under-sample.
4. Check verbatim copying of long spans. Ask for exact reproduction of a 2K-token block seen far back in the context. Exact copying is the purest bounded-state stressor and it predicts citation and code-editing fidelity better than any aggregate score.
5. Re-run steps 2 to 4 on the checkpoint you will actually deploy. If you fine-tune, especially on reasoning traces, measure recall again after. Attention amnesia means base-model recall numbers do not transfer across post-training, and this is the step teams skip.
6. Diff the served precision against a reference. Linear recurrent state is more precision-sensitive than KV attention. Where a higher-precision checkpoint exists, diff long-context outputs against it. Where it does not (K3's MXFP4 is the current example), say so explicitly in your writeup, because you are validating architecture and quantization as one bundle whether you like it or not.
7. Benchmark at your context distribution, then decide. The headline speedups are 1M-token figures. At an 8K average context you are saving memory you were not short of, and the throughput win may round to nothing while the recall risk stays real. If your workload never exercises the strength, the trade is all cost.

The architectures that survived, the interleaved hybrids from AI21, NVIDIA, IBM, Alibaba, and Moonshot, survived because they were measured on the axis that breaks pure recurrent stacks. Borrow that discipline. The speedup is real and worth having, on the condition that you have checked it did not cost you the lookup your product depends on. That is the same hybrid-by-necessity logic that produced DeepSeek's interleaved attention design and shows up again in the serving tradeoffs covered in our inference optimization guide: no single sequence-mixing strategy wins outright, so the deployment-grade answer combines them.

Key Takeaways

1. SSMs won throughput and memory, not the whole argument. Linear-time sequence mixing and a constant-size state give flat long-context memory and, per NVIDIA's own measurements on Nemotron-H, up to roughly 3x faster inference at matched accuracy. That half of the original pitch held.
2. The recall gap was the load-bearing weakness. A fixed-size state is a lossy memory, and what it loses first is associative recall: retrieving a specific earlier token verbatim. Attention does this for free; a recurrence has to have chosen to keep it and to not have overwritten it.
3. The gap is structural, not a tuning artifact. Controlled associative-recall and copying studies show Transformers succeeding where pure recurrent models lag, traced to gather-and-aggregate circuits that recurrent stacks struggle to form sharply.
4. What closed it is a ratio, not a better recurrence. A matched study across mechanisms puts the useful band at 3:1 to 6:1 linear-to-full, where hybrids approach or exceed Transformer recall on RULER at 4x to 7x less KV cache. The mechanisms that matter are selective gating, hierarchical recurrence, and controlled forgetting, so an editable state beats a bigger one.
5. Every lab converged on the same shape. Jamba, Nemotron 3, Granite 4, Falcon-H1, Zamba2, Qwen3.5, and Kimi K3 all keep a minority of attention layers. Nobody has shipped a pure recurrent model above the mid-size range, and that absence is evidence.
6. Kimi K3's released weights settle shippability at 2.8T, not shortcut-free recall. The report's ratio ablation lands at 3:1 and corroborates the smaller studies, but it runs at research-testbed scale rather than at 2.8T, so it is extrapolated upward rather than measured there; multi-hop at depth and shortcut-free retrieval at 1M remain unpublished, and MXFP4-only serving welds the architecture claim to the precision claim.
7. Recall is a property of a checkpoint, not an architecture. Chain-of-thought fine-tuning has been shown to degrade long-range recall in hybrids, so base-model recall numbers do not transfer to the reasoning checkpoint you deploy. Measure after post-training.
8. SSM-heavy stacks remain the right call for throughput-bound, low-recall, long-sequence work, and they pair cleanly with retrieval, whose strength is exactly their weak axis. Run the seven-step recall audit against a full-attention baseline before you believe any speedup.

The Acing AI newsletter tracks architecture shifts the way this piece does: which trade each new design is actually making. Subscribe for the version with the failure mode attached.