# Counterfactual Reflection Training for Security LLMs: Shaping Internal Reasoning via Future-Continuation Optimization

**Hayula AI Lab — July 2026**

---

**Abstract.** We introduce Counterfactual Reflection Training (CRT), a training method for security language models inspired by Anthropic's finding that the J-space (global workspace) carries representations of *potential future verbalizations* — what the model is disposed to say in future continuations of the current context. CRT leverages this property by constructing counterfactual continuations: given a security analysis prompt, we generate alternative future responses (some correct, some hallucinated) and train the model to activate the appropriate intermediate concepts *before* producing the final output. Applied to the Circuit-Aware LoRA pipeline for Averroes, SAIF, and Rushd, CRT further reduces hallucinations by 31% beyond Circuit-Aware LoRA alone, and improves the model's ability to express uncertainty when evidence is insufficient. CRT requires no additional labeled data — only the existing security QA pairs used in standard fine-tuning — and adds only 7% overhead to training time.

---

## 1. Introduction

A core finding of the Global Workspace analysis [Gurnee et al., 2026] is that the J-space in language models encodes not only the current output, but also *potential future verbalizations*. The model maintains representations of what it *might* say in future continuations of the current context. This has a striking implication: to shape what a model "thinks" internally, one might need only to shape what it is disposed to say in possible future continuations.

We operationalize this insight as Counterfactual Reflection Training (CRT). Rather than training solely on the correct output for a given input, CRT constructs a set of *counterfactual futures* — alternative continuations of the current context that represent plausible but incorrect analyses — and trains the model to route its internal computations through the correct J-space representations before committing to an output.

For security LLMs, where hallucination can have severe consequences (e.g., reporting a vulnerability that does not exist, or missing a real exploit), CRT provides a mechanism for teaching the model to "think before speaking" by directly shaping the J-space contents during training.

## 2. Related Work

### 2.1 Global Workspace and J-space

Gurnee et al. [2026] demonstrated that the J-space in language models functions as a global workspace: a privileged set of representations available for verbal report, directed modulation, and flexible reasoning. Critically, they showed that the J-space carries *future-oriented* representations — content the model is poised to verbalize in subsequent tokens.

### 2.2 Circuit-Aware LoRA

Hayula AI Lab [2026a] introduced Circuit-Aware LoRA, which applies circuit-level supervision during LoRA fine-tuning by mapping concept activations across transformer layers (J-lens) and training the adapter to strengthen weak concept circuits.

### 2.3 Counterfactual Training

Counterfactual data augmentation has been used in various forms — from counterfactual data augmentation for robustness [Kaushik et al., 2020] to contrastive learning [Chen et al., 2020]. CRT differs in that it targets the internal workspace representations rather than the output distribution directly.

## 3. Method

### 3.1 Counterfactual Future Construction

Given a security analysis prompt $x$ with ground-truth response $y^*$, we construct a set $\mathcal{C}(x) = \{c_1, c_2, \ldots, c_k\}$ of counterfactual continuations. Each $c_i$ is a plausible alternative to $y^*$ that differs on a specific dimension:

- **Evidence-level counterfactuals:** $y^*$ with a specific piece of evidence removed or weakened
- **Conclusion counterfactuals:** The opposite conclusion drawn from the same evidence (e.g., "this is exploitable" vs "this is not exploitable")
- **Uncertainty counterfactuals:** Explicit expressions of uncertainty where the original was confident

These counterfactuals are generated using the existing security model (Averroes, SAIF, or Rushd) by prompting with modified inputs.

### 3.2 CRT Objective

The CRT loss has two components:

**Correct path objective:** Standard supervised loss on the correct output, but with an additional J-lens supervision term:

$$\mathcal{L}_{\text{correct}} = \mathcal{L}_{\text{CE}}(y^*, f(x)) + \lambda_J \cdot \mathcal{L}_{J\text{-lens}}(x, y^*)$$

where $\mathcal{L}_{J\text{-lens}}$ measures the divergence between the model's J-space activations when processing $x$ and the target J-space pattern for correct analysis.

**Counterfactual contrastive objective:** For each counterfactual $c_i$, we train the model to *activate correct intermediate concepts* in the J-space before producing the (incorrect) continuation. The loss is:

$$\mathcal{L}_{\text{CF}} = \sum_{c_i \in \mathcal{C}(x)} \left[ \mathcal{L}_{\text{CE}}(c_i, f(x)) + \lambda_{\text{early}} \cdot \mathcal{L}_{\text{early-J}}(x, y^*, c_i) \right]$$

where $\mathcal{L}_{\text{early-J}}$ applies J-lens supervision *only to early and middle layers* (L5-L20), encouraging the model to route through the correct workspace concepts before the final layers diverge toward the counterfactual output.

**Total objective:**

$$\mathcal{L}_{\text{CRT}} = \mathcal{L}_{\text{correct}} + \lambda_{\text{CF}} \cdot \mathcal{L}_{\text{CF}}$$

### 3.3 Integration with Circuit-Aware LoRA

CRT is applied on top of Circuit-Aware LoRA training. The Circuit-Aware LoRA adapter provides the base circuit modifications, while CRT provides an additional training signal targeting J-space contents directly. The combined training procedure:

1. Train Circuit-Aware LoRA (QK-guided rank, interference regularization)
2. Freeze LoRA adapter parameters
3. Train J-lens probe for workspace concept detection
4. Apply CRT with counterfactuals (LoRA parameters unfrozen, fine-tune with CRT objective)

## 4. Experiments

### 4.1 Setup

We evaluate CRT on three security LLMs:

| Model | Base | Parameters | Domain |
|-------|------|-----------|--------|
| Averroes | Qwen2.5-7B | 7B | Security QA |
| SAIF | Qwen2.5-7B | 7B | Vulnerability analysis (9 specialists) |
| Rushd | Qwen3-8B | 8B | Multi-expert reasoning (5 experts) |

### 4.2 Results

| Metric | Circuit-Aware LoRA | + CRT | Improvement |
|--------|-------------------|-------|-------------|
| Hallucination rate | 4.2% | 2.9% | **31% ↓** |
| Concept depth (J-lens) | 18.3 layers | 21.1 layers | **15% ↑** |
| False positive rate | 5.1% | 3.4% | **33% ↓** |
| Uncertainty expression | 12% | 31% | **158% ↑** |
| Task accuracy | 91.3% | 92.1% | 0.9% ↑ |
| Training time overhead | — | +7% | Minimal |

### 4.3 Ablation on $\lambda_{\text{CF}}$

The counterfactual weight $\lambda_{\text{CF}}$ controls the strength of the J-lens supervision during counterfactual training:

- $\lambda_{\text{CF}} = 0$: Standard Circuit-Aware LoRA (no CRT)
- $\lambda_{\text{CF}} = 0.01$: Weak supervision — 12% hallucination reduction
- $\lambda_{\text{CF}} = 0.1$: Optimal — 31% reduction
- $\lambda_{\text{CF}} \geq 0.5$: Over-regularization — task accuracy drops 3%

## 5. Discussion

### 5.1 Why CRT Works

The effectiveness of CRT aligns with the Global Workspace finding: J-space representations are *future-oriented*. By training the model to activate correct intermediate concepts even when the final output is intentionally incorrect (counterfactual), we strengthen the routing circuits that guide the model toward correct analyses. The model learns that certain J-space patterns are causally necessary for correct reasoning, regardless of the final token distribution.

### 5.2 Connection to Introspection

Recent work on emergent introspective awareness [Anthropic, 2025] showed that language models can report on their internal states. CRT provides a training methodology for shaping *what* the model has to report — improving introspective accuracy by ensuring the workspace contains the right concepts.

### 5.3 Limitations

CRT requires constructing high-quality counterfactuals, which is domain-specific. For security analysis, counterfactuals need to be plausible enough to engage the model's reasoning circuits without being so similar to the correct answer that they provide no contrastive signal. We found that automated counterfactual generation (using the model itself with modified prompts) produces useful training data, but human verification improves quality.

## 6. Conclusion

Counterfactual Reflection Training provides a principled method for shaping internal J-space representations in security LLMs. By leveraging the future-oriented nature of the global workspace, CRT achieves 31% hallucination reduction beyond Circuit-Aware LoRA alone, with minimal training overhead. The method is model-agnostic and requires no additional labeled data.

---

## References

- Gurnee, W., Sofroniew, N., et al. 2026. "Verbalizable Representations Form a Global Workspace in Language Models." Transformer Circuits Thread.
- Anthropic. 2025. "Emergent Introspective Awareness in Large Language Models." Transformer Circuits Thread.
- Hayula AI Lab. 2026a. "Circuit-Aware LoRA: Reducing Hallucinations in Security LLMs via Global Workspace Intervention." research.hayula.xyz.
- Hayula AI Lab. 2026b. "QK-Guided Circuit-Aware LoRA: Structured Low-Rank Adaptation via Attention Circuit Factorization." research.hayula.xyz.
- Hayula AI Lab. 2026c. "Interference-Aware Concept Regularization." research.hayula.xyz.
- Kaushik, D., et al. 2020. "Learning the Difference that Makes a Difference with Counterfactually-Augmented Data." ICLR.
- Chen, T., et al. 2020. "A Simple Framework for Contrastive Learning of Visual Representations." ICML.