Replace stub README with full model card (training details, dataset links, pending eval table)

README.md

@@ -1,21 +1,157 @@
 ---
-base_model: unsloth/Qwen3.6-35B-A3B
-tags:
-- text-generation-inference
-- transformers
-- unsloth
-- qwen3_5_moe
 license: apache-2.0
 language:
 - en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: Qwen/Qwen3.6-35B-A3B
+datasets:
+- lordx64/reasoning-distill-kimi-k2-6-max-sft
+tags:
+- text-generation
+- reasoning
+- distillation
+- chain-of-thought
+- qwen
+- qwen3.6
+- kimi
+- kimi-k2
+- mixture-of-experts
+- moe
+- lora
+- unsloth
+model-index:
+- name: Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled
+  results: []
 ---
 
-# Uploaded finetuned  model
+# Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled
+
+A reasoning-distilled variant of **Qwen3.6-35B-A3B** taught to imitate the chain-of-thought style of **Kimi K2.6**, the frontier reasoning model from Moonshot AI. The goal: port Kimi-grade reasoning behavior into a permissively-licensed Mixture-of-Experts model that an individual can actually run.
+
+This is the **second model in the same lineup** as [`lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled`](https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled). Same base model, same training pipeline, same Unsloth + LoRA recipe — only the teacher differs. The two are designed to be **directly compared**, so users can see how reasoning style transfers from different upstream teachers into the same student architecture.
+
+## Why this model
+
+- **Kimi-style reasoning, open weights.** Kimi K2.6 is one of the strongest open-style reasoning models available, but only via the Moonshot API. This model has been fine-tuned on ~7.8k high-quality reasoning traces produced by Kimi K2.6, teaching the base to *think* before answering — with explicit `<think>…</think>` blocks — in Kimi's structure and cadence.
+- **Verbose, deliberate reasoning.** Empirically, Kimi K2.6 produces **~3.4× longer reasoning chains than Claude Opus 4.7** at "max" effort (mean 2,933 tokens/row vs 849, p95 9,764 vs 2,404 — measured on this dataset's tokenized output). The student model trained here inherits that verbosity. If you want long, careful, deliberate chains of thought, this is the variant of the lineup to use.
+- **Sparse activation, dense knowledge.** The base is a 35B-parameter MoE with **256 experts, 8 routed + 1 shared**, of which only about **3B parameters are active** per token. You get the capacity of a 35B model at the inference cost of a small dense model. Full-quality bf16 inference runs on 2× 80GB A100, 1× H200, or any 96GB+ single GPU. Quantized variants fit smaller setups (see below).
+- **Long thinking supported.** 64k token context. The model routinely emits 5–30k tokens of `<think>` reasoning on hard problems before giving the final answer — which is the whole point of reasoning models, and why this one was specifically trained end-to-end with an upstream teacher that also reasons explicitly. Note: 21% of training examples had reasoning chains exceeding the 4,096-token training context, and were truncated; for the next iteration we plan `MAX_SEQ_LENGTH=8192` to fully capture Kimi's longer chains.
+- **Companion to the Claude variant.** Use this when you want Kimi's longer/more deliberate reasoning. Use the Claude variant when you want shorter/tighter chains. Same base, same conversational interface, fully interchangeable for serving.
+
+## Intended use
+
+Built for hard reasoning: graduate-level STEM, competition math (AIME / MATH), code reasoning with explicit walk-through, multi-step logic puzzles, and agentic planning where explicit `<think>` helps correctness.
+
+For short-turn conversational latency-sensitive workloads the thinking budget can be large (longer than the Claude variant); cap `max_new_tokens` or post-process to strip `<think>…</think>` blocks if you only want final answers in production.
+
+## How to use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+repo = "lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled"
+tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    repo, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
+)
+
+messages = [{"role": "user", "content": "How many positive integers less than 1000 have digits that sum to 20?"}]
+inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(inputs, max_new_tokens=32768, do_sample=False)
+print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+### Serving with vLLM
+
+Recommended backend: **vLLM** for serving — the MoE routing + KV cache benefit significantly from continuous batching.
+
+```bash
+vllm serve lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled \
+  --dtype bfloat16 \
+  --max-model-len 65536 \
+  --gpu-memory-utilization 0.9 \
+  --trust-remote-code
+```
+
+The `--trust-remote-code` flag is required: the Qwen3.6 tokenizer ships custom code that vLLM and `transformers` need explicit permission to execute.
+
+### GGUF (LM Studio / llama.cpp)
+
+Quantized GGUF weights are coming. Watch this card or the [author's HF profile](https://huggingface.co/lordx64) for `*-IQ4_XS-GGUF`, `*-Q5_K_M-GGUF`, `*-Q8_0-GGUF` companion repos — and an [APEX-GGUF](https://github.com/mudler/apex-quant) variant once the community quanters pick it up (the [Claude variant's APEX quant](https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF) is the precedent for this lineup).
+
+## Training
+
+| | |
+|---|---|
+| Base model | `Qwen/Qwen3.6-35B-A3B` (loaded via `unsloth/Qwen3.6-35B-A3B` for faster finetuning) |
+| Teacher | Kimi K2.6 (Moonshot AI), accessed via OpenRouter |
+| Training dataset | [`lordx64/reasoning-distill-kimi-k2-6-max-sft`](https://huggingface.co/datasets/lordx64/reasoning-distill-kimi-k2-6-max-sft) — reasoning traces from Kimi K2.6 reformatted into SFT conversations (ChatML + `<think>…</think>`) |
+| Source dataset | [`lordx64/reasoning-distill-kimi-k2-6-max`](https://huggingface.co/datasets/lordx64/reasoning-distill-kimi-k2-6-max) — raw teacher traces (pre-SFT formatting) |
+| Dataset size | 7,836 full conversations, assistant side trained including `<think>…</think>` |
+| Source prompts | Drawn from `Delta-Vector/Tauri-Physical-Reasoning`, multiple `TeichAI` Claude reasoning sets, and `Crownelius Opus-4.6-Reasoning-2100x` — same prompt distribution as the Claude variant for direct teacher-comparability |
+| Method | SFT with Unsloth + TRL `SFTTrainer` + `train_on_responses_only` (loss only on assistant tokens) |
+| LoRA config | `r=16, alpha=16, dropout=0.0, targets=["q_proj","k_proj","v_proj","o_proj"]` (attention-only) |
+| Hyperparameters | `lr=2e-5`, cosine schedule, `warmup_ratio=0.03`, `weight_decay=0.01`, optimizer `adamw_8bit` |
+| Batch | `per_device=1, grad_accum=16, effective=16`, 2 epochs = 980 steps |
+| Sequence | 4,096 tokens during training (64k usable at inference — base supports it natively) |
+| Precision | bf16 on 1× H200 141GB (HF Inference Endpoint, custom container) |
+| Trainable | 3.44M params out of 35.1B (0.01%) |
+| Wall-clock | ~21 hours on H200 |
+
+### Training-time observations
+
+- **Loss curve**: descended cleanly from ~0.95 (warmup) → ~0.83 (mid-training), gradient norms steady at ~0.005, no instability throughout 980 steps. Cosine LR decayed from peak 2e-5 to ~0 by the final step.
+- **FLA fast-path disabled**: Unsloth's runtime check rejected the compiled `causal-conv1d==1.6.1` binary on H200/cc-9.0 as ABI-incompatible, forcing the Gated DeltaNet linear-attention layers to run on the slower torch fallback. This is a known issue for this stack and added an estimated ~30–50% to per-step time. Future runs in this lineup will pin `causal-conv1d` to a binary-compatible version.
+- **Token verbosity**: Kimi K2.6 traces averaged 2,933 tokens (mean) and 9,764 tokens (p95) versus 849 / 2,404 for the matched Opus 4.7 dataset — an **effective ~2.5× compute multiplier** for distillation runs at fixed `MAX_SEQ_LENGTH`. Treat this as a budgeting prior when scoping future verbose-teacher distillations.
+
+### Why attention-only LoRA on a MoE
+
+The initial plan was full LoRA including the MoE expert FFNs (`gate_proj/up_proj/down_proj`). In the course of the sister Claude project I filed and upstreamed a shape-mismatch fix to unsloth-zoo's MoE+LoRA grouped-mm path — [unslothai/unsloth-zoo#601](https://github.com/unslothai/unsloth-zoo/pull/601) — without which the expert-LoRA forward crashes on Qwen3.6's 256-expert layout. Even with that fix, single-GPU memory made expert-LoRA impractical for this run. Attention-only captures most of the signal on *style* distillation at a fraction of the trainable parameter count and memory footprint, and matches the recipe used for the Claude sibling so the two student runs are directly comparable.
+
+## Evaluation
+
+Formal benchmarks are pending. Numbers will land here once the [`training/eval.py`](https://github.com/lordx64/distillation/blob/main/training/eval.py) sweep (vLLM + lm-eval-harness, with `<think>` block stripping) finishes.
+
+| Benchmark | Setup | Score | Status |
+|---|---|---|---|
+| GSM8K | 8-shot CoT, 300 examples | _pending_ | 🟡 in queue |
+| MMLU-Pro | 5-shot, 500 examples | _pending_ | 🟡 in queue |
+| GPQA Diamond | 0-shot CoT zero-shot, 198 problems | _pending_ | 🟡 in queue |
+| AIME 2024 | 0-shot, 30 problems | _pending_ | 🟡 in queue |
+| AIME 2025 | 0-shot, 30 problems | _pending_ | 🟡 in queue |
+| MATH-500 | 0-shot, 100 problems | _pending_ | 🟡 in queue |
+
+Comparison baselines will include:
+
+- `Qwen/Qwen3.6-35B-A3B` (base, untuned)
+- `lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled` (sibling distillation, different teacher)
+
+The point of the lineup is the comparison, not the absolute score. We expect the Kimi variant to spend more tokens reasoning (per the verbosity prior above), so wall-clock-fair comparisons matter as much as accuracy.
+
+## Limitations and caveats
+
+- **Inherits base limitations.** Anything `Qwen/Qwen3.6-35B-A3B` is bad at, this model is also bad at. Distillation transfers reasoning style; it does not add factual knowledge.
+- **Not safety-tuned beyond the base.** No additional RLHF or safety alignment pass was performed. The model will reason out loud about anything it's asked to. Add your own guardrails before exposing to end users.
+- **Long generations.** As noted, Kimi-style reasoning is verbose. Plan tokens accordingly; default `max_new_tokens=32768` is recommended for hard problems, lower for shorter Q&A.
+- **Truncated training reasoning.** ~21% of training examples were truncated at the 4,096-token training context, so on questions that elicit very long teacher chains, the model has been partially trained on cut-off thoughts. Future iterations will use `MAX_SEQ_LENGTH=8192` to cover the p95.
+- **Apache-2.0 license** matches the base. Use freely for commercial and research work; attribution appreciated but not required.
+
+## Citation
 
-- **Developed by:** lordx64
-- **License:** apache-2.0
-- **Finetuned from model :** unsloth/Qwen3.6-35B-A3B
+```bibtex
+@misc{lordx64_qwen36_kimi_distill_2026,
+  author       = {lordx64},
+  title        = {Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled},
+  year         = {2026},
+  url          = {https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled},
+}
+```
 
-This qwen3_5_moe model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
+## Acknowledgements
 
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+- **Moonshot AI** for Kimi K2.6, the teacher whose reasoning style this model emulates.
+- **Qwen team** for the strong open-weights MoE base.
+- **Unsloth** for the fast-finetuning stack that made this run tractable.
+- The wider open-weights reasoning-distillation community whose prompt sets and methodology informed the dataset construction.