---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
tags:
  - latent-thoughts-tuning
  - gsm8k
  - math
datasets:
  - openai/gsm8k
language:
  - en
pipeline_tag: text-generation
library_name: transformers
---

# LT-Tuning Qwen2.5-1.5B-Instruct (GSM8K)

[Latent-Thoughts-Tuning](https://github.com/NeosKnight233/Latent-Thoughts-Tuning) applied to
`Qwen/Qwen2.5-1.5B-Instruct` on GSM8K.

A `<thinking>` token (id `151665`) is added to the vocabulary and inserted into the training
sequences at positions chosen by the model's own token confidence, so the model learns to spend
latent computation there before emitting an answer.

## Status

**Not evaluated.** No GSM8K accuracy was measured for this checkpoint, so there is no evidence
here that it improves over the base model. Benchmark it before relying on it.

## Usage

Loads as a plain `Qwen2ForCausalLM` — the run used `thinking_use_mlp: false`, so the checkpoint
contains no modules beyond the base architecture.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "robinzixuan/LT-Tuning-Qwen2.5-1.5B-Instruct-GSM8K"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Natalia sold clips to 48 friends in April, "
                                        "and then she sold half as many clips in May. "
                                        "How many clips did she sell altogether?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(inputs, max_new_tokens=512)[0], skip_special_tokens=True))
```

## Training

Three stages of 1 epoch each, run back-to-back with the optimizer reset between stages:

| Stage | Name | Mode | Thinking insertion prob |
|---|---|---|---|
| 0 | `stage0-cot` | `common` | 0.0 |
| 1 | `stage1-hidden-state` | `hidden_state` | 0.85 |
| 2 | `stage2-soft-fusion` | `soft_fusion` | 0.95 |

Between stages the dataset is regenerated: every training example is scored with a forward pass
(`thinking_strategy: confidence`) to pick where the `<thinking>` tokens go.

| | |
|---|---|
| Hardware | 8× H100 80GB |
| Precision | bf16, FlashAttention-2 |
| Global batch | 32 (4 per device × 8 GPUs × 1 grad accum) |
| Steps | 702 (234 per stage) |
| LR | 5e-5, warmup ratio 0.05, weight decay 0.01 |
| Grad clipping | 1.0 |
| Wall clock | 4h08m |
| Train data | GSM8K socratic, 7473 train / 1319 val |

### Loss

Final `stage2-soft-fusion` loss settled around **0.36** (grad_norm ~1.7, stable throughout).

The reported `train_loss` of 1.297 is the mean across all three stages and is not a meaningful
convergence figure — the stages optimize different objectives, so their losses are not comparable
to each other.

## Limitations

- Trained only on GSM8K grade-school word problems; nothing here transfers to other domains by default.
- No held-out evaluation was run, on GSM8K or anything else.
- Inherits the limitations and license terms of the `Qwen2.5-1.5B-Instruct` base model.