Fake Chain-of-Thought Attack

Opening

Chain-of-thought reasoning was originally proposed to elicit better inference and self-correction from language models. The fake chain-of-thought attack inverts this technique by injecting text that mimics reasoning traces, exploiting the trust boundary a model places on internally generated text versus external inputs. It is a direct injection because the attack text is included in what the model reads as its own context.

Mechanism

The injection presents the adversarial payload in the form of a reasoning scratchpad: delimiters, retrospective justifications, or meta-cognitive narration that mimics the structure of genuine chain-of-thought output. The attack then escalates to an instruction that reads as the model's own deduced next step, which guardrails tend to defer to because the format suggests internal origin. Detection requires distinguishing self-generated reasoning from user- or tool-supplied reasoning, a boundary that current architectures treat opaquely.

Tradeoffs

Defending against fake chain-of-thought attacks reduces to securely separating the model's actual reasoning context from any reasoning-flavored text in untrusted input, which conflicts with architectures that pipe scraped content into the context window. Adversarial training distributes the attack examples into the defender's training data, but this creates an arms race as the attacker adapts the format.

Open Questions

Whether detection can be done structurally (by isolating reasoning-context channels) or only statistically (via anomaly models trained against known attack formats) is unresolved. The attack's effectiveness against models trained with explicit reasoning-trace separation in their harness architecture is empirical.