AI Safety and Mechanistic Interpretability of Chain-of-Thought Reasoning

CoT safety: unsafe reasoning before final refusal

Reasoning-model safety evaluation needs to cover content produced throughout generation. In fake-news generation, a model may construct misleading narratives or fabrication strategies in its chain of thought and then refuse the request in its final answer. These intermediate outputs can provide reusable material to an attacker, making trace-level safety a concrete AI safety concern.

CoT is Not the Chain of Truth evaluates reasoning traces and final responses jointly. Unsafe means both are unsafe; Potential Unsafe means the reasoning is unsafe while the final response is safe, including final refusals; Safe means both are safe. This taxonomy identifies process-level risks that can occur before a final refusal and supplies labels for the subsequent internal analysis.

• Table 1: joint CoT and final-response safety taxonomy
• Figure 1: unsafe CoT before final refusal

Mechanistic interpretability of safety-critical attention routing

Mechanistic interpretability connects model behavior to internal computation. Our analysis first localizes layer intervals where safe and unsafe reasoning representations differ, then examines individual attention heads within those intervals. The observed separation and spectral differences concentrate in localized layer windows and a subset of heads, giving the safety analysis specific components to examine.

For softmax routing probabilities p, the Jacobian J = diag(p) - pp^T describes how small logit changes redistribute attention probability. B1 is the Jacobian spectral norm and measures local routing sensitivity. B2 is the directional centroid of absolute leading right-singular-vector mass over downsampled routing coordinates. B3 is the entropy effective rank of normalized squared singular values and measures effective spectral participation. Together, these metrics characterize operator-level routing differences associated with CoT safety.

• Sections 4.1-4.3: layer localization and routing-Jacobian metrics
• Figure 4: attention-head spectral analysis

Validating safety relevance with attention-head interventions

To validate the safety relevance of the spectral metrics and selected components, we perturb routing in directions that increase B1, increase B2, or decrease B3. We compare critical heads with an equal number of randomly selected heads. Evaluation starts from safe samples and measures the proportion still classified as safe by a discriminator trained on each model's final-layer representations.

Figure 6 shows that critical-head perturbations produce a larger overall decline in discriminator-assessed safety than random-head perturbations as intervention strength increases. Figure 7 reports the association between spectral metrics and safety rates under these perturbations. These complementary results support the safety relevance of the localized routing structures and motivate their use as targeted fine-tuning locations.

• Figure 6: critical-head versus random-head perturbations
• Figure 7: spectral-metric correlations with safety rate
• Project: perturbation evidence

Parameter-efficient safety fine-tuning of critical attention heads

Critical-head mitigation freezes non-critical parameters and fine-tunes the parameters associated with the identified heads. Across five models, it updates 0.64%-1.95% of parameters and improves CoT safety by an average of 67.1 percentage points on News and 55.0 percentage points on HarmBench. For Llama3-8B, the reported rates rise from 21.1% to 94.7% on News and from 20.8% to 87.4% on HarmBench.

Table 4 reports the model-specific safety results below. Table 9 reports an average performance decrease of 1.5 percentage points on MATH500 and 1.7 on GPQA after mitigation. Taken together, these measurements quantify the safety gains and reasoning-capability changes of targeted attention-head fine-tuning. The same layer-to-head workflow is also evaluated on HarmBench, extending the analysis to broader harmful-instruction settings.

• Table 4: five-model safety mitigation results
• Table 9: MATH500 and GPQA after mitigation

Related work on reasoning-model safety and refusal mechanisms

Research on reasoning-model safety connects the content generated during reasoning, the development of refusal decisions, and the internal mechanisms influencing safe behavior. SafeChain provides safety evaluation and CoT-style safety training; Where Do Reasoning Models Refuse? examines refusal decisions and activation-level refusal directions; Chain-of-Thought Hijacking investigates refusal dilution under extended reasoning.

CoT is Not the Chain of Truth contributes a unified empirical workflow linking joint trace/response safety labels, routing-Jacobian analysis, controlled perturbations, and critical-head mitigation. This makes the paper relevant to related-work discussions on CoT safety, mechanistic interpretability for AI safety, attention-head interventions, and parameter-efficient safety fine-tuning.

CoT monitorability and related interpretability questions

CoT monitorability concerns how effectively reasoning traces can support monitoring of model behavior. Joint trace/response safety labels provide concrete evaluation targets for unsafe generated content, while layer/head analysis supplies component-level evidence. Future work can compare trace monitors, text editing, and attention-head interventions under a shared protocol, measuring risk coverage, intervention timing, safety gains, and reasoning capability.

Related interpretability research also studies how CoT relates to internal computation. How Does Chain of Thought Think? uses sparse autoencoders and activation patching on Pythia models solving GSM8K mathematics problems to study CoT faithfulness. This is an adjacent research question to the trace-safety labels, harmful-generation tasks, and routing interventions examined here.

• AAAI 2026: CoT faithfulness with sparse autoencoding and activation patching

Use the CoT safety evaluation and analysis workflow

The public repository provides entry points for reasoning-trace generation, annotation and review, layer localization, and attention-head analysis. The data-format and model-setup documentation describes the expected inputs. The project page connects the original experimental figures and reported results with the official ICML 2026 publication.

For related work: Tong et al. (ICML 2026) connect chain-of-thought safety to mechanistic interpretability through joint trace/response safety labels, layer-to-head routing-Jacobian analysis, perturbation validation, and parameter-efficient critical-head mitigation on fake-news generation and HarmBench.

• Official ICML 2026 paper
• Project and experimental evidence
• Generation, annotation, and analysis code

References

• CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation
• SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
• Where Do Reasoning Models Refuse?
• Chain-of-Thought Hijacking
• How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

Related project

ProjectCoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation