Attention Beyond Quadratic
Every long-context system is a bet about which parts of attention you can afford to skip. This path starts with what a context window actually delivers under measurement, then walks the ladder of answers in order: sparse attention that keeps the mechanism and prunes the work, the hybrid stacks shipping that idea at frontier scale, linear attention that removes the quadratic term from most layers, and the state space models that go fully recurrent and run straight into the recall gap. Every step names what it gives up.
Effective Context Length: Why 1M-Token Windows Fall Short, and When RAG Still Wins
Start with the measurement, because every technique in this path is a bet about what you can afford to skip and none of those bets is worth making until you know what the window was delivering in the first place. This piece separates the advertised context window from the effective one using RULER and the lost-in-the-middle results. Focus on the gap between the two numbers, since that gap is the budget the rest of the path spends.
Sparse Attention in 2026: Why It Finally Had to Be Native
The first answer: keep attention and attend to less. This step covers the native versus post-hoc split, the dividing line that decides whether sparsity is a property of the trained model or a patch applied at serving time. Focus on why training and serving have to agree on the sparsity pattern, and on sliding window as the honest baseline every clever scheme has to beat.
DeepSeek V4 and the Hybrid Attention Bet
Sparsity as actually shipped at frontier scale. Placed here because it turns the previous step's taxonomy into a released architecture that mixes sparse and dense layers rather than committing to either. Focus on what the hybrid buys at long context, and on separating the parts of the design that are load bearing from the parts that are artifacts of a training budget.
Linear Attention at Frontier Scale: Kimi K3's KDA Claim, Fact-Checked
Now drop the quadratic term outright in most layers. Kimi K3 runs a linear attention variant in three of every four layers at 2.8T parameters and claims it beats full attention on quality, which is the strongest form of the claim anyone has made. Focus on the three claims the piece grades separately, because efficiency, quality parity and transfer to frontier scale carry very different amounts of evidence.
State Space Models in 2026: The Recall Gap, and What Finally Closed It
Finish at the fully recurrent end of the spectrum, where there is no attention left to prune. State space models won throughput and lost in-context recall, and the reason every production stack that adopted them kept some attention layers is the same reason the previous step is a ratio and not a pure design. Focus on the recall gap and on why it closed at a layer ratio rather than at a better recurrence.