Latent Reasoning: The Open Problem of Thinking Without Words

Abstract Chain-of-Thought, a method published in April 2026, compressed a model's reasoning into up to 11.6 times fewer tokens while holding accuracy roughly flat on math and multi-hop benchmarks. That is the kind of number that makes latent reasoning look inevitable: if most of the tokens in a chain-of-thought are compressible without losing the answer, then most of them were never doing the reasoning in the first place. They were narration.

That observation is the uncomfortable premise underneath the entire latent reasoning research program. The visible reasoning trace you read off a modern model, the neat "let me think step by step" scaffold, may be a partial interface to a computation that is actually happening somewhere else, in a form that has no clean translation into words: in the model's hidden states, or in tokens that carry reasoning but no language. Latent reasoning is the attempt to reason there directly and skip the narration.

This is one of the most active open problems in language model research right now, and it is worth being precise about its status: it is a research direction with real results at small scale and real, unsolved obstacles at every scale that matters. It is not a production technique in 2026. This piece walks through what the field has actually established, using three papers as the spine, and is equally clear about what it has not.

Why anyone wants to reason without words

Chain-of-thought works. Making a model emit intermediate steps before its answer is the single change that turned base models into reasoning models, and spending more inference compute on longer chains is a real, if bounded, lever. None of the latent reasoning work disputes this. It disputes the medium.

Explicit chain-of-thought forces every intermediate thought through the discrete language channel. Each reasoning step has to be a token, sampled from a vocabulary, committed to before the next step begins. That constraint carries two costs.

The first is bandwidth. A token is a low-dimensional, discrete commitment. The model's hidden state at that position is a high-dimensional continuous vector carrying far more information than the single token it gets collapsed into. When you force reasoning to route through tokens, you throw away most of that width at every step. The survey literature frames this directly: natural-language chain-of-thought "limits the model's expressive bandwidth."

The second is commitment. Sampling a token is a hard decision. Once the model writes "the answer is likely 7," it has committed to a path, and continuing the chain tends to rationalize that commitment rather than reconsider it. Reasoning in a continuous space, by contrast, lets the model hold a superposition of partial hypotheses and defer the commitment, something closer to a breadth-first search over possibilities than a single depth-first plunge.

Both costs point the same way: keep the reasoning in the latent space, and only pay the token tax at the very end when you actually need to emit an answer. The question the field is working through is whether you can do that without losing the two things the discrete trace quietly gave you: the ability to train the reasoning with reinforcement learning, and the ability to read it.

A map of the field

"Latent reasoning" is not one technique, and the term hides two genuinely different mechanisms that get lumped together. The 2025 survey by Zhu and colleagues, currently the best single map of the area, organizes the space along a distinction worth internalizing before looking at any specific paper: reasoning can be extended horizontally, across the sequence, or vertically, through the network's depth.

Horizontal (token-wise) latent reasoning is the family the three papers in this piece belong to. It keeps the standard transformer but changes what occupies the reasoning positions in the sequence. Instead of language tokens, those positions hold something else, and this is where the discrete-versus-continuous split lives:

• Continuous tokens replace the reasoning steps with high-dimensional vectors, typically the model's own hidden states fed back as inputs. Coconut is the canonical example, and CODI is its self-distillation cousin. This is the most expressive form and the hardest to train and read.
• Discrete tokens replace the reasoning steps with symbols from a reserved vocabulary. This ranges from the earliest and crudest version, filler or "pause" tokens that simply buy extra compute without carrying content, through learnable special tokens that mark internal rationales, up to Abstract Chain-of-Thought's reserved reasoning alphabet and SWITCH's boundary-marked blocks. Every step toward discreteness trades expressive width for tractability.

Vertical (architectural) latent reasoning does something structurally different: it reuses the network's own layers as the reasoning loop. Rather than adding reasoning positions along the sequence, it passes a hidden state through a block of layers repeatedly, so the model can spend more depth, and therefore more computation, on a hard token without emitting anything. The clearest instance is the recurrent-depth approach of Geiping and colleagues, whose Huginn model loops a recurrent block a variable number of times at inference. This is latent reasoning as a test-time compute knob: the same architecture can think for one iteration or twenty, with no change to the output length.

The vertical family is where latent reasoning and test-time compute scaling touch, and it is worth being precise about the relationship because they are easy to conflate. Test-time compute, as a lever, is about how much computation you spend at inference, whether through longer sampled chains, more parallel samples, or verifier-guided search. Latent reasoning is about where the reasoning is represented. Recurrent-depth sits in the intersection: it spends more compute (the test-time-compute story) by iterating in latent space (the latent-reasoning story) rather than by generating more tokens. The bounded-lever conclusion from that earlier analysis carries over intact. Extra latent iterations, like extra sampled tokens, buy accuracy on hard, checkable problems and then saturate; they do not manufacture capability the base model lacks.

Two more axes complete the map. The first is the training strategy, which is largely orthogonal to the medium: a curriculum that progressively swaps explicit steps for latent ones (Coconut, SWITCH), self-distillation from an explicit-CoT teacher (CODI, and Abstract CoT's warm-up phase), and reinforcement learning on the outcome (Abstract CoT's RL stage, SWITCH's Switch-GRPO). Most working systems combine two of these. The second is the frontier direction the survey flags as the most speculative: infinite-depth latent reasoning via masked diffusion, where reasoning is a globally consistent, reversible refinement process rather than a left-to-right pass. That line connects latent reasoning to the diffusion-model resurgence in language, and it is early enough that no result there belongs in a practitioner's decision.

With that map in hand, the three anchor papers stop looking like competitors and start looking like coordinates: all three are horizontal methods, tracing a path from continuous (Coconut) to discrete (Abstract CoT) to discrete-bounded-continuous (SWITCH), each move chosen to pay down a specific debt the previous one left open.

The three anchor papers

Read in sequence, the three are the field talking itself out of pure continuity. Coconut opens the paradigm and names the two debts; the two 2026 papers each pay one down.

Coconut: reasoning as a continuous thought

The reference point for the modern wave is Coconut (Chain of Continuous Thought), from Hao and colleagues at Meta and UC San Diego, published in late 2024 and accepted to COLM 2025. Its mechanism is the cleanest statement of the idea.

In a normal decoder, each step samples a token, embeds it, and feeds that embedding back as the next input. Coconut intercepts that loop. Instead of decoding a token, it takes the model's last-layer hidden state, the "continuous thought," and feeds it straight back in as the next input embedding. An entire reasoning step now happens in latent space, between two visible text tokens, with nothing emitted. The model ponders in vectors, then switches back to language mode only to write the final answer.

Training this is not a matter of flipping a switch, because a fresh model has no idea what to do with a hidden state presented as an input. Coconut uses a multi-stage curriculum that starts from ordinary explicit chains and progressively replaces the written reasoning steps with continuous thoughts, a few latent positions standing in for each removed language step, until the model learns to carry the reasoning internally. It is, in effect, weaning the model off words.

The payoff Coconut reported was not raw accuracy so much as the shape of the tradeoff. On logical reasoning tasks that require planning and search (GSM8K for math, ProntoQA and ProsQA for logical deduction), continuous reasoning matched or beat explicit chain-of-thought while emitting far fewer tokens, and it did so specifically because the latent representation could keep multiple candidate paths alive instead of committing early. That is the breadth-first behavior, and it is the clearest existence proof that reasoning in latent space is a real capability, not just compression.

The breadth-first claim deserves a beat, because it is the part that makes latent reasoning more than a compression trick. When a model writes an explicit chain, each step is a hard commitment sampled from the vocabulary, and a wrong early commitment tends to propagate: the rest of the chain rationalizes it. A continuous thought is not a single choice but a distribution-like vector that can encode several partial hypotheses at once, so the model can defer resolving which branch is correct until later evidence accumulates. On ProsQA, which is built to require exactly this kind of search over a graph of implications, that is where explicit chains stumble and continuous thoughts hold up. The lesson is that the medium is not neutral: reasoning in latent space changes how the model searches, not just how many tokens it prints.

But Coconut also exposed the two debts that define the rest of the field. It was demonstrated at GPT-2 scale. And feeding hidden states back as inputs breaks two things practitioners rely on. You cannot easily optimize it with on-policy reinforcement learning, because there is no discrete token to assign a probability to, so the policy ratio that methods like GRPO depend on is not well defined at the latent positions. And you cannot read it, because a continuous thought is a vector with no guaranteed mapping back to language. Coconut opened the paradigm and, in the same move, handed the field its two hard problems: it is hard to train and hard to audit.

The efficiency axis: discrete abstract tokens

The first debt the field went after was efficiency, and the cleanest 2026 result on that axis is Abstract Chain-of-Thought, from Ramji, Naseem, and Fernandez Astudillo. The core design choice is a retreat from Coconut's pure continuity: instead of free continuous vectors, the model reasons in a short sequence of discrete tokens drawn from a small reserved vocabulary. These "abstract" tokens are not words. They are learned placeholders for reasoning operations, but crucially they are still discrete symbols the model chooses, which is what makes them tractable.

The training pipeline runs in three phases. First, a warm-up loop alternates between extracting latent reasoning patterns from verbal chains (masking parts of explicit traces and fine-tuning the model to fill them with abstract tokens) and self-distillation using constrained decoding that restricts the output to the reserved tokens, forcing the model to compress its reasoning into that narrow alphabet. Second, reinforcement learning optimizes which abstract token sequences actually lead to correct answers. The result is a model that reasons in a compact code before switching to language for the answer.

The headline is efficiency: up to 11.6 times fewer reasoning tokens than explicit chain-of-thought, at comparable accuracy, evaluated across mathematical reasoning (AIME, MATH), graduate-level QA (GPQA), and multi-hop retrieval (HotpotQA), on 7B-scale base models including IBM's Granite 4.0 Micro and the OLMo 3 series. The self-distillation idea has a close cousin in CODI (Shen and colleagues), which distills reasoning into continuous space by aligning the hidden states of a designated token and reported a 3.1 times compression on GSM8K at GPT-2 scale, the first implicit-CoT method to match explicit CoT at that scale.

Here the eval-honesty caveat matters, and it is the kind of thing a leaderboard number hides. "Up to 11.6 times fewer reasoning tokens" is a statement about the reasoning segment, not about end-to-end cost. It does not include the answer tokens, it is the best case ("up to"), and token count is not latency: an abstract-token step still runs a full forward pass. The efficiency is real and it is meaningful, but the honest framing is "far fewer intermediate tokens," not "an order of magnitude cheaper to serve." What you are mostly buying is shorter decode sequences, which matters most exactly where chains are longest.

The optimizability and interpretability axis: SWITCH

The more ambitious 2026 result goes after both remaining debts at once. SWITCH, from Yang and colleagues, starts from an explicit diagnosis: existing latent methods "are difficult to optimize with standard on-policy reinforcement learning and hard to interpret causally." Those are Coconut's two debts, named. SWITCH's proposal is a single structural change that addresses both.

It wraps the latent reasoning block in explicit boundary tokens, written <swi> and </swi>. Between them, the model runs its hidden-state recurrence; outside them, it is in normal token mode. Those two discrete tokens do a surprising amount of work.

On training, the boundary tokens make the latent block compatible with policy-gradient RL. Because entering and leaving the latent computation is now marked by discrete decisions, the policy ratio stays well defined at every decision point, and the authors' Switch-GRPO objective can propagate gradients through the recurrent latent computation. They pair it with a visible-to-latent curriculum, starting with explicit reasoning and progressively pushing it into the latent block as training proceeds, the same weaning strategy Coconut used, now made RL-friendly. The models are Qwen-scale (a 1B base for the main evaluation, plus a Qwen3-8B variant released as a LoRA), and the method reports gains on math reasoning (GSM8K, MATH) against latent baselines including Coconut.

On interpretability, the boundary tokens give what the paper calls "a natural foothold for mechanistic analysis." Because you know exactly where the latent computation starts and stops, you can probe and causally intervene at those points. The findings are the interesting part: the switching behavior is learned, not stylistic, it does causally important problem-specific work, and, strikingly, the useful computation concentrates at a single hidden-state transition right as the model enters the latent block. That is a real, testable claim about where the reasoning lives, which is more than any purely continuous method could offer.

Three papers, three axes

The three anchor papers are not competing implementations of one idea. They are attacks on different debts, and the through-line is a steady retreat from pure continuity back toward discreteness, because discreteness is what buys back trainability and a foothold for analysis.

Read down the "reasoning medium" row and the story is clear. Coconut's continuous vectors are the most expressive and the least tractable. Abstract CoT's reserved tokens give up some expressive width to become optimizable and compressible. SWITCH keeps continuous recurrence inside the block but wraps it in discrete markers so the outside world can train and probe it. Each step toward discreteness is a step toward something you can actually work with, which is a quiet admission that pure latent reasoning, the Coconut ideal, is not yet something the current toolchain can handle.

What latent reasoning has not solved

The three papers establish that latent reasoning works and can be made trainable and probeable. Two things they do not establish are whether the results mean what they appear to, and whether any of it survives outside the small-model lab. This is where the direction earns its "not yet."

Reading the benchmarks honestly

Before treating any of these numbers as a verdict, it is worth asking what they actually measure, because the benchmark selection in this field is not neutral. Look at the tasks: GSM8K, MATH, AIME, ProntoQA, ProsQA, GPQA, HotpotQA. Every one is a problem with a short, checkable final answer and a reasoning process that benefits from search or multi-step deduction. That is precisely the regime where latent reasoning's breadth-first advantage should show up, and precisely the regime where "fewer intermediate tokens at equal accuracy" is easy to define because accuracy is a clean binary.

This is not a criticism of the papers; it is the correct place to start proving a new idea. But it does bound the claims. None of these results tell you how latent reasoning behaves on open-ended generation, on tasks where the "answer" is a paragraph rather than a number, or on problems where the reasoning is constitutive of the output rather than scaffolding for it. Wang's 2026 analysis makes exactly this distinction: latent methods dominate ordinary reasoning tasks, but surface chain-of-thought reasserts itself when the trace is the product (tool use, code that must be read, reasoning a human has to check), and pure extra compute wins in heavily search-bound settings. The honest reading is that latent reasoning has a home turf, and the benchmarks are played on it.

The comparison that would settle the matter is the one that is hardest to find in these papers: the honest baseline. What does a strong explicit chain-of-thought model, at the same parameter count and the same inference compute budget, score on the same tasks? Coconut's contribution was framed as a better accuracy-versus-efficiency tradeoff, which is a two-axis claim and the right way to state it. But "up to 11.6 times fewer reasoning tokens at comparable accuracy" invites a single-axis reading (efficiency for free) that the two-axis reality does not support: you are trading a readable, RL-mature, well-understood explicit trace for a compressed one, and "comparable accuracy" is doing quiet work in that sentence. Whenever a latent method reports a headline efficiency multiple, the first question is what it gave up to get there and against which baseline, and the second is whether the comparison held compute, not just token count, fixed. The broader evaluation problem applies with full force here: a compression number without a matched baseline and a fixed compute budget is half a result.

The reality gap: what discreteness does not buy back

Here is where the site's usual question applies. Research shows latent reasoning works; what happens when you ask it to survive contact with the things production and science actually require? Two gaps stand out, and both are the subject of active, skeptical 2026 work.

The first gap is interpretability, and it is more subtle than "latent reasoning is a black box." The implicit promise of SWITCH's boundary tokens is that discreteness restores auditability. But a boundary marker tells you where the computation happens, not what it computed. You can probe the entry transition; you still cannot read a continuous thought the way you can read "3 times 4 is 12." Dilgren and Wiegreffe's 2026 study asks the question directly in its title, "Are Latent Reasoning Models Easily Interpretable?", and the answer is a firm no: standard interpretability tools do not cleanly decode what these models do internally. A foothold is not a translation.

It gets worse when you look at faithfulness over time rather than at a single checkpoint. Jin, Yang, and Wang track latent reasoning faithfulness along training trajectories and find that internal representations stay unfaithful through most of training, and that a model can produce the right answer while its internal reasoning does not actually drive that answer. Correct output, flawed or irrelevant internal process. If you were hoping latent reasoning would give you a trustworthy internal trace to monitor for safety or debugging, this is the opposite of reassuring.

But the honest version of this critique cuts both ways, and this is the part worth sitting with. The explicit chain-of-thought we are nostalgic for was never faithful either. A large interpretability literature has shown that written reasoning traces are frequently post-hoc rationalizations that do not reflect the computation that produced the answer. Wang's 2026 analysis makes this its thesis: reasoning is primarily mediated by latent-state trajectories, and the surface chain-of-thought is "only a partial interface." So when latent reasoning "loses the readable trace," it is mostly giving up the appearance of an audit, not a real one. The trade is auditable surface for efficiency and trainability. It is not truth for speed, because we did not have the truth to begin with. That reframing is the whole point, and it should make you more skeptical of explicit traces, not just of latent ones. It connects directly to the evaluation-honesty problem: a plausible-looking reasoning trace is evidence of fluency, not of a correct process.

The second gap is scale, and it is the one that most cleanly disqualifies latent reasoning from production today. Look again at the base-model column: GPT-2 for Coconut and CODI, roughly 7B for Abstract CoT, 1B to 8B for SWITCH. The vertical family is not much higher; Huginn is a few-billion-parameter model. Every result in this field lives at or below the 8B mark. There is no published demonstration that continuous or abstract-token reasoning holds up at frontier scale, where explicit chain-of-thought was validated and where the RL post-training machinery that makes reasoning models good actually runs.

This is not a pedantic objection. Small-scale reasoning results have a long history of not surviving the trip to frontier scale, and the reason is specific to what latent reasoning claims to add. The gains here come from giving a weak model a better way to reason: more search, more deferred commitment, more compute per token. A frontier base model already reasons well in explicit tokens because scale bought it the capability directly, so the marginal room a latent method has to improve on a strong explicit-CoT baseline shrinks exactly as the base model gets better. It is entirely possible that latent reasoning is a technique whose benefit is largest precisely where it matters least, on small models, and vanishes into the noise on the models people actually deploy. Nobody has shown otherwise. Until someone demonstrates a latent method beating a strong explicit-CoT baseline at, say, 70B or more, at iso-compute, on tasks that were not chosen to favor search, the correct prior is that this is unproven where it counts.

There is a third, quieter gap worth naming: the training cost and complexity. Every method here needs a multi-stage curriculum, a distillation teacher, an RL loop, or some combination, layered on top of an already-trained base. That is real engineering and real compute spent to buy an efficiency win at inference. For the technique to pay for itself in production, the inference savings have to outweigh a nontrivial training investment across the deployment's lifetime, and none of the papers report that break-even because it is not the question they were asking. It is, however, the first question a practitioner should ask, and the absence of an answer is itself a signal about maturity.

Should you use any of this yet?

Almost certainly not in production, but the decision is worth making deliberately rather than by default. Here is the procedure I would run before reaching for a latent reasoning method today.

1. Check your scale. If you are working below roughly 8B parameters and on math or logical-deduction tasks, you are in the regime where these methods have actually been demonstrated. Above that, you are extrapolating past all published evidence. Stop here and use explicit chain-of-thought unless you are doing research.
2. Check whether you have an exact verifier. The trainable methods (Abstract CoT's RL stage, SWITCH's Switch-GRPO) need a checkable reward, so the same rule from test-time compute applies: cheap exact check (math, code, deduction) makes RL-based latent reasoning viable; no verifier means you are limited to the distillation-only variants and their weaker guarantees.
3. Check whether you need an auditable trace. If a human or a downstream safety system must inspect the reasoning (regulated decisions, agent oversight, debugging), do not go latent. The trace you would get is not readable, and even a boundary-token foothold is not an explanation. Stay explicit, and treat that explicit trace as a partial interface, not ground truth.
4. Check whether tokens are actually your bottleneck. The efficiency win is in the reasoning segment. If your chains are short, or if answer generation dominates your latency, the payoff is small. It matters most for long-chain, deliberation-heavy workloads.
5. If you are experimenting, start from SWITCH's design, not Coconut's. Discrete boundary tokens give you RL compatibility and a probe point for free, which is worth more in practice than pure continuity, and the released Qwen3-8B variant is the most realistic starting point available.

The short version: this is a technique to track, prototype against at small scale, and read the papers on. It is not one to put in front of users in 2026.

Key takeaways

1. Latent reasoning is an active research direction, not a production technique. Every published result lives at or below 8B parameters, and there is no demonstration that it survives at frontier scale, where explicit chain-of-thought and its RL machinery are validated.
2. The premise is that visible reasoning is mostly narration. Abstract CoT compressed reasoning to up to 11.6 times fewer tokens at flat accuracy, which is strong evidence that most chain-of-thought tokens were not doing the reasoning.
3. The field is retreating from pure continuity toward discreteness. Coconut's continuous vectors are the most expressive and least tractable; Abstract CoT's reserved tokens buy compressibility and trainability; SWITCH's boundary tokens buy RL compatibility and a probe point. Discreteness is what makes latent reasoning workable.
4. Coconut left two debts: hard to train with RL, hard to read. A continuous thought has no token to assign a probability to (breaking on-policy RL) and no clean mapping back to language (breaking interpretability). The 2026 work is largely about paying these down.
5. SWITCH's boundary tokens are the most complete answer so far. They make the latent block RL-optimizable via Switch-GRPO and give a mechanistic foothold, where analysis found the useful computation concentrated at a single hidden-state transition on entry.
6. Discreteness restores a foothold, not readability. 2026 studies find latent models are not easily interpretable and stay unfaithful across training, where a correct answer can coexist with an internal process that did not produce it.
7. Explicit chain-of-thought was never faithful either. Written traces are often post-hoc rationalizations, so latent reasoning gives up the appearance of an audit, not a real one. Treat any reasoning trace, latent or explicit, as a partial interface rather than ground truth.
8. The decision rule for today: below 8B, with an exact verifier, on long deliberation chains, and no audit requirement, it is worth prototyping. Otherwise, stay explicit.

References

1. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. (2024). Training Large Language Models to Reason in a Continuous Latent Space (Coconut). COLM 2025. arXiv:2412.06769
2. Ramji, K., Naseem, T., and Fernandez Astudillo, R. (2026). Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought. arXiv:2604.22709
3. Yang, J., Chen, C., Wu, S., Liu, Y., Fan, Y., Li, L., Lai, S., Qin, C., and Guo, Z. (2026). Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning (SWITCH). arXiv:2606.13106
4. Shen, Z., Yan, H., Zhang, L., Hu, Z., Du, Y., and He, Y. (2025). CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. arXiv:2502.21074
5. Zhu, R.-J., et al. (2025). A Survey on Latent Reasoning. arXiv:2507.06203
6. Dilgren, C., and Wiegreffe, S. (2026). Are Latent Reasoning Models Easily Interpretable? arXiv:2604.04902
7. Jin, H., Yang, S., and Wang, D. (2026). Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories. arXiv:2607.06648
8. Wang, W. (2026). LLM Reasoning Is Latent, Not the Chain of Thought. arXiv:2604.15726
9. Geiping, J., et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn). arXiv:2502.05171