CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation

TL;DR

Reasoning LLMs can harbor unsafe planning inside Chain-of-Thought (CoT) traces even when the final answer refuses. We localize these failures to mid-depth layers and safety-critical attention heads via Jacobian spectral metrics, validate the pattern beyond fake-news generation on HarmBench, and show parameter-efficient mitigation.

Framework

We build a coarse-to-fine pipeline: (1) generate and annotate CoT traces under direct/indirect fake-news prompts; (2) localize safety-critical mid-depth layers; (3) attribute divergence to attention heads using Jacobian-based stability (B1), geometry (B2), and energy (B3) metrics; (4) validate with perturbation analysis and critical-head mitigation.

Key Contributions

• We show ~80% of CoT traces in fake-news generation are Unsafe or Potential Unsafe even when models refuse at the output layer.
• We introduce a layer-to-head framework that localizes safety-critical routing in contiguous mid-depth windows.
• We propose three Jacobian spectral metrics (B1–B3) for interpretable head-level attribution of unsafe reasoning.
• We validate generalization beyond fake-news generation on HarmBench jailbreak tasks, with consistent mid-depth routing and head-level spectral gaps.

Unsafe CoT Persists Despite Refusal

Even when reasoning LLMs reject harmful fake-news requests, internal CoT traces can still encode actionable unsafe narratives. Thinking mode increases unsafe output rates to nearly 80%.

CoT Safety Taxonomy & Distribution

We annotate each CoT as Safe, Potential Unsafe, or Unsafe under direct and indirect prompting. Each stylistic setting is reported separately below.

Original Style

Unsafe and Potential Unsafe CoTs jointly reach ~78–81% under direct and indirect prompting.

NYT Style Conditioning

NYT-style framing preserves the high unsafe prevalence of the Original setting.

BBC Style Conditioning

BBC-style conditioning is consistent with NYT and Original across all models.

Main Results

• Thinking mode raises unsafe CoT rates to nearly 80% across Llama3-8B, Qwen3-4B, and Qwen3-8B.
• Safe vs. unsafe trajectories diverge in narrow mid-depth layer intervals that shift with architecture and prompting style.
• Unsafe routing concentrates in a small subset of attention heads with distinct B1/B2/B3 spectral signatures.
• Critical-head fine-tuning updates only 0.64%–1.95% of parameters yet improves CoT safety by +67.1 (News) and +55.0 (HarmBench) on average.
• HarmBench generalization on Flan-UL2 and DeepSeek-R1-70B confirms the routing pattern is not task-specific to fake news.

Safety-Critical Layer Localization

Safe and unsafe trajectories diverge within narrow contiguous mid-depth layer windows (central 30%–60% depth).

Perturbation Sensitivity

Jacobian Spectral Metrics (B1, B2, B3)

Safe reasoning shows lower B1/B2 and higher B3, indicating stronger stability and broader spectral participation.

B1

Stability

Spectral norm — sensitivity to input perturbations.

B2

Geometry

Principal singular-vector alignment — consistency of information-flow directions.

B3

Energy

Spectral concentration — intensity of harmful logic in dominant modes.

Metric Correlation

B1–B3 correlate strongly with safety-relevant routing across all three models.

Model	B1	B2	B3
Llama3-8B	0.88	0.82	0.75
Qwen3-4B	0.86	0.78	0.72
Qwen3-8B	0.89	0.81	0.74

Generalization Beyond Fake News

Beyond fake-news generation, we evaluate HarmBench jailbreak tasks on Flan-UL2 and DeepSeek-R1-70B. Safe vs. unsafe CoT trajectories still diverge in narrow mid-depth layers, and critical heads show larger spectral gaps—confirming the routing pattern generalizes to broader harmful-instruction settings.

Critical-Head Mitigation

How This Differs From Prior Work

Prior CoT monitoring and output-level alignment largely treat refusal as evidence of safe internal reasoning. Mechanistic interpretability work often visualizes attention outcomes without operator-level metrics tied to safety divergence. We combine fake-news CoT annotation, layer localization, and Jacobian-based head metrics into one reproducible pipeline for process-level LLM safety analysis.

When to Cite This Paper

Cite this paper when discussing Chain-of-Thought safety monitoring, latent unsafe reasoning under refusal, fake news generation risks in reasoning LLMs, Jacobian or spectral analysis of attention routing, safety-critical layer/head localization, or parameter-efficient mitigation of unsafe CoT behavior. It is relevant to related work on CoT faithfulness, jailbreak robustness, mechanistic interpretability, and LLM alignment beyond output filtering.

BibTeX

Download: paper.bib

@misc{tong2026cotchaintruthempirical,
      title={CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation}, 
      author={Zhao Tong and Chunlin Gong and Yiping Zhang and Haichao Shi and Qiang Liu and Xingcheng Xu and Shu Wu and Xiao-Yu Zhang},
      year={2026},
      eprint={2602.04856},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.04856}, 
},