Qwen3.6-34B-80L-Fable-5-Heretic

Fable-5 distilled Qwen3.6 80-layer deep reasoning model.

Built on the Qwen3.6-27B (64-layer, original) base, extended to 80 layers (~34B parameters) and distilled from Fable-5 Agentic Traces. Fine-tuned via QLoRA (4-bit NF4, rank=64) on 4,665 Fable-5 Chain-of-Thought trajectories, then merged with MTP speculative decoding heads.

Model Card

Property	Value
Base model	Qwen3.6-27B (64-layer, original) → Qwen3.6-34B-80L-BF16-v2
Architecture	Qwen3_5ForConditionalGeneration
Parameters	~34B (80 layers, hidden=5120)
Attention heads	24 (4 KV Grouped)
Context length	262,144 (256K)
Vocabulary size	248,320
Attention mode	Hybrid Linear/Full (full every 4 layers)
MTP speculative decoding	✅ (15 mtp weights)
Precision	BF16

Distillation Data

Item	Value
Data source	Glint-Research/fable-5-traces
Samples	4,665 entries
Content	Fable-5 Agentic CoT trajectories (context + completion)
Training config	QLoRA rank=64, α=64, 2 epochs, lr=2e⁻⁴, max_len=4096

Training Details

• Quantization: bitsandbytes 4-bit NF4 + double quantization (QQ)
• LoRA targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, and MoE projection layers
• Optimizer: paged_adamw_8bit, cosine schedule, 5% warmup
• Gradient checkpointing: enabled (use_reentrant=False)
• Effective batch size: 4 × 4 = 16

Key Features

1. Fable-5 deep reasoning — Inherits Fable-5's CoT chain-of-thought reasoning and tool-use patterns.
2. 80-layer efficient architecture — Stronger expressiveness at 34B parameters with 80 layers.
3. Hybrid attention — Alternating Linear/Full design for higher inference efficiency on long sequences.
4. MTP speculative decoding — Native MTP weights; vLLM deployment achieves ~78% acceptance rate, 2× throughput.
5. BF16 full precision — MergeLoRA merged without quantization loss, ready for deployment and re-fine-tuning.

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hiebo/Qwen3.6-34B-80L-Fable-5-Heretic"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "What is quantum computing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM Launch Configuration

vllm serve hiebo/Qwen3.6-34B-80L-Fable-5-Heretic \
    --dtype bfloat16 \
    --max-model-len 262144 \
    --enforce-eager \
    --gpu-memory-utilization 0.95 \
    --swap-space 16 \
    --disable-log-stats

Supported Parameters

Parameter	Default	Description
--model	(required)	Model ID or local path
--dtype	auto	Data type: bfloat16, float16, float32, auto
--max-model-len	4096	Maximum context length (supports 256K)
--max-num-seqs	256	Maximum number of concurrent sequences
--gpu-memory-utilization	0.90	Fraction of GPU memory to use (0–1)
--swap-space	4	CPU swap space in GB
--tensor-parallel-size	1	Tensor parallelism degree (multi-GPU)
--pipeline-parallel-size	1	Pipeline parallelism degree
--enforce-eager	False	Force PyTorch eager mode (recommended for BF16 models)
--disable-log-stats	False	Disable statistics logging
--served-model-name	None	Override served model name
--port	8000	API server port
--host	0.0.0.0	API server host
--tokenizer-mode	auto	Tokenizer mode: auto, slow, mistral
--quantization	None	Quantization: fp8, awq, gptq, squeezellm
--load-format	auto	Load format: auto, sharded_state, safetensors, npcache
--kv-cache-dtype	auto	KV cache data type: auto, float16, bfloat16
--enable-chunked-prefill	False	Enable chunked prefill for long sequences
--max-num-batched-tokens	None	Max batched tokens per step
--seed	0	Random seed for reproducibility
--distributed-executor-backend	ray	Distributed backend: ray, mp, mp_rank
--trust-tokenizers	False	Trust remote tokenizer files
--revision	main	Model revision (branch/tag/commit)