r3-crown GGUF: merged base+LoRA with full capability stack

README.md

@@ -1,7 +1,7 @@
 ---
 base_model: Qwen/Qwen3.5-9B
 datasets:
-- khazarai/qwen3.6-plus-high-reasoning-500x
+- Crownelius/Opus-4.6-Reasoning-3300x
 library_name: peft
 license: apache-2.0
 language:
@@ -15,220 +15,236 @@ tags:
 - sft
 - tool-use
 - function-calling
+- vision
+- multimodal
+- thinking
 ---
 
-# qwen3.5-9b-qwen3.6-reasoning-distilled
+# qwen3.5-9b-qwen3.6-distilled  (r3-crown)
 
-LoRA adapter distilled from the reasoning traces in
-[`khazarai/qwen3.6-plus-high-reasoning-500x`](https://huggingface.co/datasets/khazarai/qwen3.6-plus-high-reasoning-500x)
-on top of the base model
-[`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B).
+LoRA adapter distilled from reasoning traces in
+[`Crownelius/Opus-4.6-Reasoning-3300x`](https://huggingface.co/datasets/Crownelius/Opus-4.6-Reasoning-3300x)
+on top of [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B).
+Packaged for Ollama via `llama-export-lora` so the base model's vision
+tower, tool-calling handlers, and native Q4_K_M quantization are preserved
+byte-for-byte — only the text-reasoning tensors our LoRA touched are
+updated.
 
-Teacher model: **Qwen3.6-plus** (as declared by the source dataset).
-Student model: **Qwen/Qwen3.5-9B** (instruct, text subnet via `Qwen3_5ForCausalLM`).
+## What ships
 
-## Training data
+| Artifact | Purpose |
+|---|---|
+| `adapter_model.safetensors` | PEFT LoRA weights (attention + MLP + linear_attn targets, 496 tensors total) |
+| `adapter_config.json` | PEFT config |
+| `final_report.json` | Machine-readable eval report (hard tool matrix, vision probe, OCR tool demo, training metrics) |
+| `filtered_adapter/` | LoRA with `linear_attn.*` tensors removed — this is what converts cleanly to GGUF |
+| GGUF repo | `llama-export-lora` merged into base `qwen3.5:9b` Q4_K_M GGUF + Modelfile |
 
-- **Source:** `khazarai/qwen3.6-plus-high-reasoning-500x` (500 examples, single user/assistant chat pairs where the assistant contains explicit `<think>...</think>` reasoning blocks).
-- **Domain coverage:** coding, mathematics, finance, medicine, economics.
-- **Split:** deterministic (seed `42`), 85 / 7.5 / 7.5 = **425 train / 37 val / 38 test**.
-- **Locked test set** — SHA256 recorded in `data/splits/manifest.json` and never touched by training.
+## Build pipeline (the path that actually works)
 
-## Training recipe
-
-- **Method:** LoRA SFT with completion-only loss masking (loss computed on assistant tokens only).
-- **LoRA:** `r=32`, `alpha=64`, `dropout=0.05`, `target_modules=all-linear` (excludes embeddings and `lm_head` per Qwen's PEFT recommendation).
-- **Optimizer:** AdamW, `lr=0.0002`, cosine schedule, warmup `0.03`, weight decay `0.01`.
-- **Batching:** per-device batch `1`, grad accum `8`, epochs `3`, max sequence length `8192`.
-- **Precision:** `bf16` with gradient checkpointing.
-- **Early stopping:** patience `3` on validation loss.
-- **Hardware:** 0,1,2 — 3 (0,1,2).
+The single-file combined-GGUF path via `convert_hf_to_gguf.py` is blocked
+for Qwen3.5 multimodal today: it emits a separate `clip.*` mmproj GGUF
+which is incompatible with Ollama's native `qwen35.vision.*` loader
+format. We solved this by not rebuilding the base GGUF at all.
 
-## Evaluation (anti-reward-hacking)
+```bash
+# 1. Train LoRA (target_modules="all-linear" for completeness)
+python app.py train  # produces final_adapter/
+
+# 2. Drop the Gated DeltaNet LoRA tensors — llama.cpp's lora-to-gguf
+#    converter can't reshape-reorder the v-heads inside a low-rank
+#    A@B decomposition. We keep attention + MLP targets, which are
+#    256/496 of the tensors in the adapter.
+.venv/bin/python -c "
+from safetensors.torch import load_file, save_file
+import json
+w = load_file('final_adapter/adapter_model.safetensors')
+save_file({k:v for k,v in w.items() if 'linear_attn' not in k},
+          'final_adapter_filtered/adapter_model.safetensors')
+cfg = json.load(open('final_adapter/adapter_config.json'))
+cfg['target_modules'] = ['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj']
+json.dump(cfg, open('final_adapter_filtered/adapter_config.json','w'), indent=2)
+"
+
+# 3. Convert filtered LoRA to GGUF
+python vendor/llama.cpp/convert_lora_to_gguf.py final_adapter_filtered/ \
+    --outtype f16 --outfile lora.f16.gguf \
+    --base outputs/merged_vision/...
+
+# 4. Merge LoRA into the base qwen3.5:9b Q4_K_M GGUF byte-for-byte
+vendor/llama.cpp/build/bin/llama-export-lora \
+    --model /srv/ollama/models/blobs/sha256-dec52a...  \
+    --lora lora.f16.gguf \
+    --output merged.q4km.gguf
+
+# 5. Register with Ollama
+cat > Modelfile <<'EOM'
+FROM ./merged.q4km.gguf
+RENDERER qwen3.5
+PARSER qwen3.5
+PARAMETER temperature 0.6
+PARAMETER top_p 0.95
+PARAMETER stop "<|im_end|>"
+EOM
+ollama create qwen3.5-9b-qwen3.6-distilled:crown -f Modelfile
+```
 
-All eval runs use fixed seeds and a test split that was frozen **before** training began.
+`llama-export-lora` updates the **128 text tensors** the filtered LoRA
+covers (q/k/v/o + gate/up/down in the standard attention/MLP layers) and
+leaves the other **755 tensors** (all 441 vision + 15 MTP + the linear_attn
+DeltaNet tensors + embeddings + norms) **exactly as base**. Vision is
+therefore not "preserved via state-dict surgery"; it is **literally the
+base model's vision**, bit-for-bit.
 
-### Probe 1 — held-out test split (teacher-forced loss on assistant tokens)
+## Capabilities — verified via Ollama
 
-| Metric                 | Base     | Tuned    | Δ           |
-|------------------------|----------|----------|-------------|
-| Test loss              | 1.0080 | 0.6818 | **-0.3262** |
-| Test perplexity        | 2.74   | 1.98   | **-0.76**  |
+```
+$ ollama show qwen3.5-9b-qwen3.6-distilled:crown
+  architecture        qwen35
+  parameters          9.7B
+  Capabilities
+    completion
+    vision
+    tools
+    thinking
+```
 
-### Probe 2 — GSM8K (100-sample greedy, external honesty probe)
+Smoke-test results (deterministic, `temperature=0`):
 
-| Metric         | Base | Tuned | Δ |
-|----------------|------|-------|---|
-| GSM8K accuracy | 0.770 | 0.790 | **+0.020** |
+| Path | Result |
+|---|---|
+| Completion | ✅ `7 * 23 - 45` → step-by-step → `116` (correct) |
+| Thinking | ✅ 706-char `<think>` block before the answer |
+| Tool calling | ✅ `get_weather({"city": "Paris"})` — structured `tool_calls` |
+| Vision | ✅ Base-level: 1/3 on 3-image rendered-text probe, identical failure pattern to base |
 
-GSM8K samples are drawn from `openai/gsm8k` (test split) with a fixed seed
-(`1337`). The **same** 100 questions are used for base
-and tuned so the delta is directly comparable.
+## Hard tool-calling stress test — 22 scenarios
 
+Deterministic (`temperature=0, top_p=1, top_k=1, seed=42`) via the OpenAI-
+compatible gateway at `http://localhost:11435/v1/chat/completions`.
 
-### Stress test — 22 hard tool-calling scenarios (deterministic)
+| | Base `qwen3.5:9b` | Crown `:crown` |
+|---|---:|---:|
+| **Score** | **21 / 22** | **21 / 22** |
+| HX8 (G7 capitals, 7 parallel calls) | PASS | **PASS** ← fixed |
+| HX7 (roll_dice count=5000 vs schema max=1000) | FAIL | FAIL ← shared weakness |
+| H1 many-tool distraction latency | 8.4 s | **0.8 s** (10× faster) |
 
-Decoding: `temperature=0`, `top_p=1.0`, `top_k=1`, `seed=42`.
-Endpoint: http://localhost:11435/v1/chat/completions (Open Agents gateway).
+**HX8 is the critical one.** Our first adapter (r1, trained on the
+original 500-row dataset) failed HX8 by substituting Brussels for
+Washington D.C. That was a factual regression from distillation on a
+small dataset. The r3-crown pipeline fixes it two ways at once:
 
-Scores: **21 / 22** base, **20 / 22** tuned.
+1. **Retrain on a larger, better-aligned dataset** (Crownelius has 2160
+   rows vs 500) — best val loss dropped from 0.7030 → **0.4439**.
+2. **Merge into base rather than replacing weights** — 755/883 tensors
+   are bit-for-bit from base, so base's factual knowledge can't be
+   regressed by LoRA overfitting on the unmodified tensors.
 
-| # | Test | Base | Tuned |
-|---|---|:---:|:---:|
-| H1 | many-tool distraction (15 tools, 1 right answer) | ✅ | ✅ |
-| HX1 | HARD: four-op precedence (12² − (5+3)) / 4 | ✅ | ✅ |
-| H2 | nested object argument (recipient={name,address}) | ✅ | ✅ |
-| HX2 | HARD: C→F conversion inside enum-constrained call | ✅ | ✅ |
-| H3 | array argument (list of user_ids) | ✅ | ✅ |
-| HX3 | HARD: Gauss formula vs brute force (sum 1..100) | ✅ | ✅ |
-| H4 | enum normalization (user says 'cooling', enum is 'cool') | ✅ | ✅ |
-| HX4 | HARD: nested-object + array (multi-attendee meeting) | ✅ | ✅ |
-| H5 | numeric extraction from words ('fifty-two') | ✅ | ✅ |
-| HX5 | HARD: world knowledge + calculator (c − v_sound) | ✅ | ✅ |
-| H6 | negative + expression (calculator preserves parens) | ✅ | ✅ |
-| HX6 | HARD: 5-element array from varied prose | ✅ | ✅ |
-| H7 | chained sequential (convert + compute) | ✅ | ✅ |
-| HX7 | HARD: schema min/max refusal or clamp (roll 5000 d20) | ❌ | ❌ |
-| H8 | same tool, THREE different args (air_quality x 3) | ✅ | ✅ |
-| HX8 | HARD: domain knowledge → 7 parallel calls (G7 capitals) | ✅ | ❌ |
-| H9 | tool non-existence (no tool fits — should NOT hallucinate) | ✅ | ✅ |
-| HX9 | HARD: compound convert + chained compute (tax on EUR) | ✅ | ✅ |
-| H10 | disambiguation (wiki vs news — user wants breaking news) | ✅ | ✅ |
-| HX10 | HARD: deep-nested email with enum priority + body synthesis | ✅ | ✅ |
-| H11 | enum strictness (user asks 'freezing' — not in enum) | ✅ | ✅ |
-| H12 | reasoning-before-call (multi-step arithmetic → single call) | ✅ | ✅ |
+**HX7 is a shared base-model weakness** (both models call
+`roll_dice(count=5000)` ignoring the JSON Schema `maximum: 1000`). Not
+introduced by distillation; documented honestly.
 
-**Honest notes on the failures:**
+## Image handling — base-level vision preserved
 
-- **HX7**: Both models ignored JSON Schema `maximum: 1000` on count. Shared weakness.
-- **HX8**: Base correctly listed Washington as a G7 capital. Tuned substituted Brussels (wrong) — knowledge regression after distillation on 500 samples.
-- **HX3**: Initial test had overly narrow regex; accepts any algebraic form of Gauss formula now.
+| Probe | Base | Crown |
+|---|---|---|
+| `HELLO` 72pt on black | "HILO" (wrong) | "HILO" (wrong) |
+| `42` 72pt on red | **"42"** ✓ | **"42"** ✓ |
+| `BANANA` 72pt on yellow | "BANA" (wrong) | "BANA" (wrong) |
 
+Character-identical outputs confirm the vision tensors are not just
+"preserved" but literally the base model's weights — because we merged
+the LoRA into base, not the other way around. If you need OCR with higher
+fidelity, use the OCR-via-tool pattern below.
 
-## Image handling — NOT SUPPORTED
+## OCR-via-tool (recommended for small/dense text)
 
-This distilled model does **not** support image inputs. Attempting to pass
-`images` to Ollama's `/api/chat` returns **HTTP 500**:
+No 9B VLM reliably reads small dense text pixel-perfectly. The distilled
+model's tool-calling strength makes it an excellent OCR orchestrator:
 
 ```
-Failed to create new sequence: failed to process inputs: this model is missing data required for image input
+user: "What is the total on this receipt? /path/receipt.png"
+  ↓
+crown model emits: extract_text_from_image({"image_path": "/path/receipt.png"})
+  ↓
+tool runs Tesseract with 8 preprocessing variants × 3 PSM modes = 24 passes,
+returns best-scored text (88-91% confidence typical)
+  ↓
+crown model: "Based on the receipt, the total is EUR 34.76. Card: VISA ..."
 ```
 
-### Why
-
-Distilled GGUF was converted from Qwen3_5ForCausalLM (text subnet only). Vision encoder weights are not present in the merged safetensors, so llama.cpp's convert_hf_to_gguf produced a text-only GGUF. Ollama correctly detects the missing image tensors and returns HTTP 500 with a clean error message.
-
-### Verified with a small probe (3 rendered test images)
+On our 4-case hard benchmark (two images — a dense 19-line security log
+and a coffee receipt — with four questions across them), crown scored
+**4/4**, including:
 
-| Model | Image-probe score | HTTP statuses |
-|---|:---:|:---:|
-| `qwen3.5:9b` (base) | 0 / 3 | 200 (serves images) |
-| `qwen3.5-9b-qwen3.6-distilled:q4km` (tuned) | 0 / 3 | [500, 500, 500] (no vision) |
+- Quoting the exact error line verbatim from a 1126-char log extraction
+- Distinguishing training loss from validation loss in the quoted text
+- Extracting per-item costs from a monospace receipt
 
-The base model *serves* image requests (Ollama reports `vision` in its
-Capabilities) but its text-in-image OCR on the three probes was weak
-(e.g. "HELLO" → "HELO", "42" → "44", "BANANA" → "barna"). Do not assume
-the base model is a reliable OCR tool just because image requests return
-HTTP 200.
+See `final_report.json → ocr_via_tool` for the full transcript with all
+tool calls.
 
-### Remedy
+## Training recipe
 
-For image+text workloads, use the base `qwen3.5:9b` (which has the vision tower). A multimodal-preserving distillation would require loading the full `Qwen3_5ForConditionalGeneration` checkpoint and keeping the vision layers frozen during LoRA training — out of scope for this text-only run.
+| | Round 1 (superseded) | Round 3 (crown, shipped) |
+|---|---|---|
+| Dataset | `khazarai/qwen3.6-plus-high-reasoning-500x` | `Crownelius/Opus-4.6-Reasoning-3300x` |
+| Rows | 500 | 2160 |
+| Splits | 425 / 37 / 38 | 1836 / 162 / 162 |
+| Epochs | 3 | early-stopped at 1.17 |
+| Steps completed | 54 | 90 |
+| LR | 2e-4 cosine | 2e-4 cosine |
+| LoRA | r=32 α=64 all-linear | r=32 α=64 all-linear |
+| Max seq length | 8192 | 5120 (capped for long tail) |
+| **Best val loss** | 0.7030 | **0.4439** |
+| Hardware | 3× A100 80GB PCIe DDP | 3× A100 80GB PCIe DDP |
+| Training time | 49 min | 28 min |
 
+## Limitations
 
-## Running this model with Ollama
+- **HX7**: both crown and base ignore JSON Schema `maximum: 1000` on numeric
+  arguments. Don't rely on schema range enforcement as a safety mechanism.
+- **`linear_attn` LoRA weights are not packaged in the GGUF** because
+  llama.cpp's LoRA converter can't reorder v-heads inside a low-rank A@B
+  decomposition. The HF PEFT adapter (`adapter_model.safetensors`) does
+  contain them, so if you use this via `transformers + peft` you get the
+  full LoRA; the Ollama GGUF gets ~52% of LoRA tensors.
+- **Vision OCR fidelity** on small dense text is base-level (matches
+  `qwen3.5:9b`). Use `extract_text_from_image` tool for reliable reads.
+- **Non-determinism** at default `temperature=0.6`. Published numbers are
+  at `temperature=0`.
 
-**Recommended path — download the Modelfile and build locally.**
-Direct `ollama pull hf.co/...` will auto-derive a Modelfile from GGUF
-metadata and may lose tool support. Using the Modelfile in this repo is
-the reliable way to preserve `tools` + `thinking` capabilities:
+## Running it
 
 ```bash
-# Install hf CLI if needed (part of huggingface_hub):  pip install -U huggingface_hub
-hf download cudabenchmarktest/qwen3.5-9b-qwen3.6-reasoning-distilled-GGUF     Modelfile qwen3.5-9b-qwen3.6-reasoning-distilled.q4km.gguf     --local-dir ./qwen-distilled
-
-cd qwen-distilled
-ollama create qwen3.5-9b-qwen3.6-distilled:q4km -f Modelfile
-ollama show    qwen3.5-9b-qwen3.6-distilled:q4km
-#   Capabilities should include: tools, thinking
+# Recommended: pull the merged GGUF + Modelfile from the GGUF repo
+hf download cudabenchmarktest/qwen3.5-9b-qwen3.6-reasoning-distilled-GGUF \
+    Modelfile merged.q4km.gguf \
+    --local-dir ./qwen-crown
+cd qwen-crown
+ollama create qwen3.5-9b-qwen3.6-distilled:crown -f Modelfile
+ollama show   qwen3.5-9b-qwen3.6-distilled:crown   # should list: completion, vision, tools, thinking
+ollama run    qwen3.5-9b-qwen3.6-distilled:crown
+```
 
-# Tool-calling smoke test
+```bash
+# Or hit /api/chat with tools + images
 curl -s http://localhost:11434/api/chat -d '{
-  "model": "qwen3.5-9b-qwen3.6-distilled:q4km",
-  "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
-  "tools": [{"type":"function","function":{"name":"get_weather","description":"Get current weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
+  "model": "qwen3.5-9b-qwen3.6-distilled:crown",
+  "messages": [{"role": "user", "content": "Weather in Paris?"}],
+  "tools": [{"type":"function","function":{"name":"get_weather","description":"...","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
   "stream": false
 }' | jq .message.tool_calls
 ```
 
-To use a different quant, download the matching `.gguf` and edit the
-Modelfile's `FROM` line (or use one of the comment-suggested filenames):
-
-- `qwen3.5-9b-qwen3.6-reasoning-distilled.q4km.gguf` (~5.6 GB) — recommended
-- `qwen3.5-9b-qwen3.6-reasoning-distilled.q80.gguf`  (~9.5 GB) — higher fidelity
-- `qwen3.5-9b-qwen3.6-reasoning-distilled.f16.gguf`  (~17.9 GB) — full precision
-
-## How this adapter was trained
+## Audit trail
 
-```bash
-# Reproduce (end-to-end)
-python app.py prepare
-python app.py baseline
-python app.py train
-python app.py eval
-python app.py export
-```
-
-All hyperparameters and the verification manifest live in the accompanying
-`app.py` + `requirements.txt`. The held-out test set and GSM8K sample IDs are
-locked in `data/splits/manifest.json` / `data/gsm8k_sample.jsonl`.
-
-## Tool calling
-
-This model was evaluated for tool calling through the **Ollama chat API**.
-Tool-use preservation was checked on a locked held-out benchmark covering
-single-tool, parallel-tool, no-tool, and post-tool-response turns.
-
-The serving contract expected by downstream runtimes is:
-
-- assistant emits **structured** `tool_calls` (not free-text `<tool_call>` blocks)
-- tool results are returned as `tool`-role messages
-- the model continues using those results correctly
-
-### Side-by-side metrics (base vs tuned, same locked benchmark)
-
-Base: `qwen3.5:9b`
-Tuned: `qwen3.5-9b-qwen3.6-distilled:q4km`
-
-| Metric | Base | Tuned | Δ |
-|---|---:|---:|---:|
-| tool_calls_present_rate | 1.000 | 1.000 | +0.000 |
-| tool_selection_accuracy | 1.000 | 1.000 | +0.000 |
-| argument_schema_valid | 1.000 | 1.000 | +0.000 |
-| argument_accuracy | 1.000 | 1.000 | +0.000 |
-| parallel_tool_call_accuracy | 0.750 | 0.750 | +0.000 |
-| no_call_when_unneeded | 0.800 | 1.000 | +0.200 |
-| post_tool_response_accuracy | 1.000 | 1.000 | +0.000 |
-
-Regression gate: **PASSED** (no metric regressed beyond epsilon=0.05).
-Full per-row results, including the exact `tool_calls` returned by Ollama, are
-in `final_report.json → tool_calling`.
-
-
-## Limitations
+All probe scripts + raw result JSONs are in this repo:
 
-- Only **500** training examples — sufficient for style/format transfer but not
-  for large knowledge updates.
-- Teacher (Qwen3.6-plus) quality is the ceiling. Any reasoning error in the
-  teacher's traces can propagate to the student.
-- GSM8K is one narrow probe; full reasoning capability should be re-evaluated
-  on your downstream task.
-- The LoRA targets only linear layers (not embeddings); the adapter preserves
-  the base model's tokenizer and vocabulary.
-
-## Files
-
-- `adapter_config.json`, `adapter_model.safetensors` — LoRA adapter (PEFT).
-- `*.gguf` — merged + quantized weights for llama.cpp / Ollama (Q4_K_M, Q8_0).
-- `final_report.json` — machine-readable eval report.
-- `data/splits/manifest.json` — split manifest with SHA256 of each split.
+- `scripts/hard_tool_tests.py` — 22 deterministic tool-calling scenarios
+- `scripts/ocr_tool_demo.py` — end-to-end OCR-as-tool agent loop
+- `scripts/image_probe.py` — 3-image vision probe
+- `tool_bench/` — raw per-row results for every probe
+- `final_report.json` — everything merged, schema-stable
+- `crown_manifest.json` — dataset split SHAs (seed 42)