---
license: apache-2.0
base_model: Qwen/Qwen3-8B-Base
tags:
  - code
  - reasoning
  - sft
  - chain-of-thought
datasets:
  - a-m-team/AM-Thinking-v1-Distilled
language:
  - en
pipeline_tag: text-generation
---

# Qwen3-8B-Base-SFT-AM-Thinking-v1-Distilled-Code-1800steps

SFT of [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base) on the
**code subset** of [AM-Thinking-v1-Distilled](https://huggingface.co/datasets/a-m-team/AM-Thinking-v1-Distilled)
(verify_score ≥ 0.9), using the standard Qwen3 chat template and `<think>...</think>`
reasoning protocol.

## Training

- **Base**: Qwen/Qwen3-8B-Base
- **Data**: AM-Thinking-v1-Distilled (code subset, ~300K samples)
- **Hardware**: 4 nodes × 8 × H20-96G (32 GPUs)
- **Framework**: TRL SFTTrainer + FSDP FULL_SHARD + Liger Kernel + FlashAttention-2 + packing
- **LR**: 5e-5, cosine schedule, warmup_ratio=0.1
- **Global batch size**: 128 (32 GPUs × bsz=4 × accum=1)
- **Max seq len**: 32768
- **Steps**: 1800

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "LumosJiang/Qwen3-8B-Base-SFT-AM-Thinking-v1-Distilled-Code-1800steps"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Write a Python function to compute Fibonacci(n)."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

The model emits `<think>...reasoning...</think>` followed by a fenced ` ```python ``` ` code block.

## Recommended sampling

Aligned with the official Qwen3 sampling protocol:
`temperature=0.6, top_p=0.95, top_k=20`