Qwen3.8-27B-Claude-Opus-Reasoning-Distilled

A LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, targeting the deep, structured chain-of-thought style Opus is known for — mapped onto an open-weight dense 27B you can run and fine-tune yourself.

Trained text-only (vision tower untouched, not targeted by LoRA) with Unsloth on a single H100.

🔗 Quantized GGUF versions (Q8_0 → Q4_K_M) with verified MTP speculative decoding available at rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF.

🧠 What this is

Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers). This fine-tune targets only the text reasoning path — both attention types via LoRA, MLP layers, vision tower left frozen and unused.

• Base model: Qwen/Qwen3.8-27B (dense, 27B, hybrid linear+full attention, MTP head)
• Method: LoRA (r=32, alpha=32) via Unsloth, 4-bit QLoRA base
• Response-only masking: loss computed only on assistant turns (train_on_responses_only)
• Hardware: 1× NVIDIA H100 NVL (95GB), Vast.ai
• Training data: ~21.5K examples combining real Opus extended-thinking traces and reconstructed-reasoning traces over genuine Opus outputs (see Dataset Composition below)

📊 Training Details

Steps trained	150 (partial run — pipeline validation, not full convergence)
Epochs	0.126 (~12.6% of one epoch, ~2,700 of 21,490 examples)
Effective batch size	18 (per-device 6 × grad accum 3)
Final train loss (avg)	0.728
Final eval loss	not computed this run (eval_strategy="no" to save time)
Wall-clock time	53 min on 1× H100 NVL
Sequence length	8192 (99.7% of dataset uncut; 0.3% longest examples excluded rather than truncated)
Trainable params	233,455,616 (0.85% of 27.6B)
Adapter size on disk	954 MB

Status: pipeline-validation run, not a fully converged model. 150 steps at batch 18 covers ~12.6% of one epoch — enough to confirm the training pipeline (LoRA on hybrid attention, masking, chat template, checkpointing) works end-to-end, not enough for the model to have generalized across the full dataset. A full run (1+ epoch, ~1,194+ steps) is needed before this should be treated as a finished distillation. Loss trended downward over the run (high ~0.95-1.1 early, several sub-0.6 readings by step 70-90) but with a cosine schedule calibrated for exactly 150 steps — this checkpoint cannot simply be resumed for a longer run; a full run needs a fresh schedule sized to the full dataset.

📚 Dataset Composition

Source	Examples	What it is
lordx64/reasoning-distill-claude-opus-4-7-max	8,124	Genuine Claude Opus 4.7 extended-thinking traces (verified model/usage fields per row)
Jackrong/Claude-opus-4.7-TraceInversion-5000x	4,800	Real Opus 4.7 outputs (teacher_model field), reasoning trace reconstructed post-hoc by a smaller "trace inversion" model — plausible CoT matching a genuine answer, not captured original thinking
Jackrong/Claude-opus-4.6-TraceInversion-9000x	8,700	Same trace-inversion method, Opus 4.6

Transparency note: 62% of the dataset (the two TraceInversion sets) has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. This is disclosed so downstream users know the reasoning style is a plausible approximation for roughly two-thirds of the data, not a verbatim mirror throughout. The lordx64 subset (38%) is the closest to authentic Opus thinking style.

We deliberately excluded several other "Claude-distilled" datasets found on Hugging Face after inspection — some contained self-referential synthetic text describing itself as "exemplifying" a model's style rather than real model output, and others were low-effort duplicate repos farming downloads. Always verify teacher_model/usage provenance fields before trusting a distillation dataset.

📈 Base Model Benchmarks (Qwen/Qwen3.8-27B, not this fine-tune)

These are official Qwen3.8-27B numbers for the base model, included for reference — this fine-tune has not been independently re-benchmarked yet.

Text Performance

	Qwen3.8-27B	Qwen3.6-27B	Qwen3.7-Plus
Terminal-Bench 2.1 (agentic coding)	73.0	63.4	64.0
SWE-bench Pro	61.7	53.5	57.6
QwenSWEBench	79.0	49.3	59.2
CoWorkBench (long-horizon office work)	70.7	61.0	65.1
IFBench (instruction following)	79.5	69.1	79.1
GPQA Diamond (scientific reasoning)	89.2	87.8	90.3
HLE (multidisciplinary reasoning)	30.8	24.0	34.7
LiveCodeBench v6	90.3	83.9	89.6

VL Performance (base model only — this fine-tune does not target vision)

	Qwen3.8-27B	Qwen3.6-27B	Qwen3.7-Plus
OSWorld-Verified (computer use)	84.3	63.9	73.3
AndroidWorld (mobile use)	81.9	70.3	81.0
MathVision (with CI)	94.6	85.1	90.3
OmniDocBench 1.5	91.1	89.4	91.4

Full base model benchmark tables: Qwen/Qwen3.8-27B model card.

🚀 Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled",
    max_seq_length = 8192,
    dtype = None,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Explain the difference between TCP and UDP."}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(inputs, max_new_tokens=1024, temperature=1.0, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0]))

Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template. Use reasoning_content as a separate message field if constructing multi-turn history manually — do not pack the thinking block into content, the chat template expects it separate.

⚠️ Known Limitations

• This checkpoint is a pipeline-validation run (150 steps, ~12.6% of one epoch), not a fully converged fine-tune. Expect it to show the target <think> format and some stylistic shift, but not robust generalization across task domains. A full training run is needed for production use — see status note above.
• Dataset partially composed of reconstructed (not captured) reasoning traces — see Dataset Composition above
• Text-only fine-tune; vision tower is unmodified base weights, not evaluated

🙏 Acknowledgments

Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets.

---

Maintained by rico03