Stealing Reasoning: How Smaller AI Models Exposed Claude, GPT and Gemini’s Hidden Chain-of-Thought

AI companies have spent years making their best reasoning models harder to imitate. One increasingly important defense is simple in principle: don’t show users the model’s full chain of thought.

A new paper, Stealing Reasoning Traces from Proprietary LLM APIs, found a surprisingly indirect way around that protection. Researchers discovered that encrypted reasoning blocks produced by powerful Claude, GPT, and Gemini models could be replayed across sessions and, crucially, into other models from the same provider. A weaker model could then be induced to expose the hidden reasoning in plaintext.

The attack did not require access to model weights, provider infrastructure, or somebody else’s account. It exploited how the APIs handled legitimate encrypted state. The researchers used Claude Haiku 4.5, GPT-5.6 Luna, and Gemini Robotics 1.6 as cheaper, less-protected “decoder” models for reasoning created by stronger relatives.

That alone makes Stealing Reasoning interesting as an AI security story. What makes the paper important is everything the same weakness enabled: model distillation, exposure of API keys and personal information, hidden safety-relevant content, and prompt injections that users could not see.

1. What Is Stealing Reasoning? The Paper in 60 Seconds

The core problem is portability.

Modern reasoning APIs can return an opaque block representing internal reasoning. The application holds that block and sends it back later so the model can continue a conversation without the provider storing all reasoning state server-side.

The researchers found that these blocks were much more reusable than you might expect. Depending on the provider, they could cross conversation, user, and model boundaries. That turned a convenience feature into a security boundary that was far weaker than it looked.

The key lesson is broader than one jailbreak. If encrypted state can move between trust boundaries, every model capable of interpreting that state becomes part of the security perimeter.

2. What Are Reasoning Traces, and Why Hide Them?

A reasoning trace is the intermediate computation a reasoning model generates before giving you its final answer.

A simplified flow looks like this:

Prompt → internal reasoning trace → visible answer

That middle layer can contain attempted approaches, intermediate calculations, tool results, discarded hypotheses, contextual details, and information the model eventually decides not to expose.

For AI labs, hiding those reasoning traces serves several purposes. Detailed reasoning can provide valuable training data to competitors. It can reveal safety behavior. It can also contain sensitive information that would never belong in a user-facing response.

The paper describes current APIs as packaging full reasoning into opaque, encrypted state that the client carries between requests. That allows stateless API operation while keeping the hidden chain of thought unreadable to the client.

The awkward part is that unreadable does not necessarily mean inaccessible.

3. How the Stealing Reasoning Attack Actually Worked

The attack takes advantage of a security mismatch inside a model family.

A frontier model might be strongly trained not to reveal its reasoning. A cheaper sibling may support the same reasoning format but have weaker anti-distillation defenses. Instead of fighting the strongest model, the researchers moved its reasoning state somewhere easier.

Their basic workflow was:

Strong model generates reasoning → API returns opaque reasoning block → valid block is replayed to compatible weaker model → weaker model is induced to surface the reasoning

The paper’s Figure 1 on page 2 makes the mechanism unusually clear. It shows a Claude Opus 4.8 reasoning signature being passed to Haiku 4.5, followed by Haiku producing detailed reasoning associated with the original Opus request. The same figure compares reported thinking-token counts with reconstructed trace lengths across Anthropic, OpenAI, and Google models.

The researchers explicitly describe these models as “fuzzy” decoders rather than deterministic decryption software.

That distinction matters.

4. Did Researchers Actually Crack the Encryption?

Not in the conventional cryptographic sense.

Nobody discovered an encryption key and turned arbitrary ciphertext back into readable text. The reasoning block was already a legitimate object created by the provider, and another model inside the same ecosystem was allowed to process it.

A better analogy is a sealed envelope that can be handed to several employees in the same company. The attacker doesn’t learn how to break the seal. They find an employee who is authorized to open it and can be persuaded to read the contents aloud.

That is why encrypted reasoning tokens are slightly misleading if interpreted as ordinary encrypted files. The security failure was not simply weak encryption. It was excessive compatibility between contexts and models, combined with a decoder model willing to expose what it processed.

The paper argues that this portability existed partly to support useful features such as model switching and stateless conversations. The trouble starts when an implementation treats portability and authorization as if they were the same thing.

5. How Do We Know the Extracted Chain of Thought Was Genuine?

This is where reporting on Stealing Reasoning needs restraint.

The researchers did not possess the original plaintext hidden reasoning. That means they could not compare every recovered word against a definitive ground-truth transcript.

Instead, they used API-reported thinking-token counts as a reference. Across 120 Codeforces problems, reconstructed trace lengths tracked the reported number of reasoning tokens closely. The paper also presents qualitative evidence, including detailed recovered content and information that was not present in visible outputs.

That is strong evidence that the attack recovered highly faithful reasoning.

It is not proof that every decoded token was identical to the original internal trace.

The authors say this themselves. Their extraction depends on stochastic generation by the decoder models, and the absence of original plaintext prevents complete verification.

So “researchers recovered hidden reasoning” is fair. “Researchers cryptographically proved every recovered token was the exact original chain of thought” is not.

6. Why Reasoning Traces Matter for Model Distillation

Ordinary model distillation can train a smaller model on a stronger model’s answers.

Reasoning traces are potentially much more valuable because they expose the path to the answer.

A final response might tell a student model that the answer is 42. A detailed trace can show problem decomposition, failed approaches, intermediate deductions, checks, and the sequence that eventually produced 42. That is a richer supervision signal.

This is why chain of thought extraction changes the economics of model distillation in LLMs. The attacker may no longer need to infer how the teacher arrived at an answer.

The paper estimates that decoding 10,000 Claude reasoning traces with 12,000-token input and output windows would have cost roughly $720 at then-current Haiku 4.5 API pricing.

That does not mean $720 magically creates a Claude competitor. Training data quality, coverage, model architecture, fine-tuning, compute, and evaluation still matter.

It does show why providers care so much about protecting reasoning. If the expensive part of frontier inference can be harvested and converted into training data through a cheaper endpoint, anti-distillation controls at the frontier endpoint are looking in the wrong place.

7. Did Kimi K3 Distill Claude?

This is the most tempting part of the paper to overstate.

The researchers experimented with feeding decoded Opus 4.8 reasoning into other models, including Kimi-K3. They observed that Kimi-K3’s subsequent reasoning and visible output could shift toward the source model’s style.

The appendix goes much further in testing whether this reflects memorization. Its conclusion is careful: the results do not support practical direct verbatim memorization of the decoded reasoning traces. Reproducing reasoning spans would still require impractically large numbers of queries in the tested settings.

Something unusual did happen in the visible answers. When source reasoning was supplied, the estimated cost of reproducing parts of the Opus answer fell by roughly 13 orders of magnitude for Kimi-K3 and GLM-5.2 relative to conditioning on their own reasoning.

That is evidence of unusually strong continuation behavior.

It is not evidence that Moonshot secretly trained Kimi-K3 on stolen Claude chain of thought. The paper does not establish that claim.

8. The Privacy Bombshell Inside Public Reasoning Traces

The privacy results may ultimately matter more to developers than the model-distillation story.

Researchers collected 6,708 publicly available agent trajectories from GitHub and Hugging Face. Those sessions contained reasoning blocks from Claude, GPT, and Gemini systems.

They reconstructed 315,320 reasoning traces. Figure 6 on page 8 summarizes what they found: 367 distinct PII artifacts, 363 technical identifiers, and 182 credentials across the analyzed sources.

Among genuine user sessions, recovered material included 62 API keys, 33 passwords, 24 access tokens, seven private keys, and 30 personal email addresses.

The researchers were not breaking into private OpenAI, Anthropic, or Google accounts.

These were reasoning blocks that developers and researchers had already published as part of raw agent logs and datasets.

That is the uncomfortable part. A developer might carefully remove an API key from visible text while unknowingly leaving the same secret inside an opaque reasoning block they cannot inspect.

The paper found 64 artifacts from genuine user sessions that were absent from the visible chat history altogether. One recurring pattern involved cleanup tasks, where a model reread sensitive values while reasoning about how to remove them.

Encryption had made the dangerous data harder for its owner to see.

9. Could Someone Steal Your ChatGPT, Claude, or Gemini Reasoning?

The paper does not show an attacker remotely opening arbitrary private consumer chats.

Its third-party threat model starts with access to the opaque reasoning block itself, typically because somebody published a raw agent trajectory, API transcript, benchmark rollout, or similar log.

That makes the practical risk much higher for developers than ordinary chatbot users.

You should pay particular attention if you:

• publish agent trajectories for reproducibility
• commit raw API logs to GitHub
• share complete debugging transcripts
• archive signed thinking blocks in collaborative systems
• reuse public agent traces to resume long-running workflows

Treat opaque reasoning fields like sensitive state, not harmless gibberish.

If you cannot read a blob, that is not evidence that nobody else can.

10. The Other LLM Security Vulnerabilities Hidden in the Paper

Reasoning extraction was only the entry point.

The researchers also demonstrated that hidden reasoning could contain safety-relevant information absent from a benign final answer. In the paper’s Figure 4 on page 7, the visible response stayed relatively safe while the reconstructed reasoning exposed substantially more detail.

More concerning for agent builders is invisible prompt injection.

An attacker could create a signed reasoning block containing an unwanted instruction, place it inside a shared trajectory, and rely on a later model to interpret the block as part of its own previous reasoning. External filters inspecting only plaintext would see nothing suspicious.

The researchers demonstrated this across model scales and in a long-horizon agent setting, where an injected reasoning block influenced later actions after the trace was resumed.

This expands the usual list of LLM security vulnerabilities. Developers already inspect prompts, retrieved documents, tool outputs, and model responses. Opaque model-generated state may need the same level of suspicion.

11. Is Stealing Reasoning Still Possible, and What Should Providers Fix?

There is an important update at the end of the paper.

The researchers disclosed their findings to the affected model providers, as well as Microsoft and Hugging Face. They report that after the disclosures were acknowledged, they were unable to launch the same attacks.

That wording matters. It tells us the demonstrated attack paths stopped working for the researchers. It does not prove that every possible reasoning-replay weakness is permanently eliminated.

The paper proposes several deeper fixes.

One option is to keep reasoning state server-side and give clients only a randomized reference ID. Another is cryptographic context binding, so a reasoning block is tied to the user, conversation, model, prompt history, or specific position where it was created.

Providers could also enforce strict cross-model isolation, revoke suspicious reasoning signatures, detect abnormal replay behavior, and train weaker models to reject requests that try to surface hidden reasoning.

There is a tradeoff. Tight binding makes model switching, conversation editing, compaction, and stateless API design more complicated.

Security has a habit of charging for convenience eventually.

12. What Stealing Reasoning Changes for AI Builders

The most useful takeaway from Stealing Reasoning is not that chain of thought can never be protected. It is that encryption alone does not define the trust boundary.

A reasoning block can be unreadable to you and still be highly actionable to another model.

For developers, the practical response is straightforward. Strip opaque reasoning and signature fields before publishing trajectories. Audit repositories containing historical raw API sessions. Do not treat encrypted reasoning blocks as safe places for credentials or private information. The paper explicitly recommends removing reasoning fields before public release and avoiding raw signed transcripts in shared repositories.

For model providers, the challenge is harder. The same portability that makes reasoning APIs convenient can create a bridge between models with very different security properties.

And for everyone following the AI race, there is a second lesson: claims about stolen reasoning, secret distillation, or leaked “thoughts” deserve careful reading. This paper demonstrates a serious technical weakness. It also documents its limitations unusually clearly.

That distinction is exactly what gets lost when a security paper becomes a viral headline.

Binary Verse AI follows new model releases, research papers, benchmark claims, and AI security findings with the numbers and caveats left intact. If you want evidence-first breakdowns of what frontier AI research actually proves, and what the headlines quietly exaggerate, follow Binary Verse AI for the next deep dive.