docs: add upstream Qwen3-4B-Instruct-2507 official evals
Browse files
README.md
CHANGED
|
@@ -87,6 +87,49 @@ on 7 hardware-design tasks against the base `gemma-e4b-eu-kiki-base` model.
|
|
| 87 |
See full scoreboard:
|
| 88 |
[ailiance-bench Phase 6 README](https://github.com/ailiance/ailiance-bench#scoreboard-lora-phase-6--2026-05-11)
|
| 89 |
· interactive viewer: [Ailiance-fr/playground](https://huggingface.co/spaces/Ailiance-fr/playground).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
## Citations
|
| 91 |
|
| 92 |
|
|
|
|
| 87 |
See full scoreboard:
|
| 88 |
[ailiance-bench Phase 6 README](https://github.com/ailiance/ailiance-bench#scoreboard-lora-phase-6--2026-05-11)
|
| 89 |
· interactive viewer: [Ailiance-fr/playground](https://huggingface.co/spaces/Ailiance-fr/playground).
|
| 90 |
+
|
| 91 |
+
## Upstream base model — official evaluations
|
| 92 |
+
|
| 93 |
+
These are the **official scores** for the unmodified base model
|
| 94 |
+
[`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
|
| 95 |
+
reported by Alibaba Qwen team. They represent the floor of capability that this
|
| 96 |
+
LoRA inherits before the hardware-domain fine-tune adapts behavior.
|
| 97 |
+
|
| 98 |
+
| Category | Benchmark | Qwen3-4B-Instruct-2507 |
|
| 99 |
+
|---------------|-------------------------|-----------------------:|
|
| 100 |
+
| Knowledge | MMLU-Pro | 69.6 |
|
| 101 |
+
| Knowledge | MMLU-Redux | 84.2 |
|
| 102 |
+
| Knowledge | GPQA | 62.0 |
|
| 103 |
+
| Knowledge | SuperGPQA | 42.8 |
|
| 104 |
+
| Reasoning | AIME25 | 47.4 |
|
| 105 |
+
| Reasoning | HMMT25 | 31.0 |
|
| 106 |
+
| Reasoning | ZebraLogic | 80.2 |
|
| 107 |
+
| Reasoning | LiveBench 2024-11-25 | 63.0 |
|
| 108 |
+
| Coding | LiveCodeBench v6 | 35.1 |
|
| 109 |
+
| Coding | MultiPL-E | 76.8 |
|
| 110 |
+
| Coding | Aider-Polyglot | 12.9 |
|
| 111 |
+
| Alignment | IFEval | 83.4 |
|
| 112 |
+
| Alignment | Arena-Hard v2 | 43.4 |
|
| 113 |
+
| Alignment | Creative Writing v3 | 83.5 |
|
| 114 |
+
| Alignment | WritingBench | 83.4 |
|
| 115 |
+
| Agent | BFCL-v3 | 61.9 |
|
| 116 |
+
| Agent | TAU1-Retail | 48.7 |
|
| 117 |
+
| Agent | TAU1-Airline | 32.0 |
|
| 118 |
+
| Agent | TAU2-Retail | 40.4 |
|
| 119 |
+
| Multilingual | MultiIF | 69.0 |
|
| 120 |
+
| Multilingual | MMLU-ProX | 61.6 |
|
| 121 |
+
| Multilingual | INCLUDE | 60.1 |
|
| 122 |
+
| Multilingual | PolyMATH | 31.1 |
|
| 123 |
+
|
| 124 |
+
Source: [official Qwen3-4B-Instruct-2507 model card](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).
|
| 125 |
+
|
| 126 |
+
> **Reading these numbers alongside the Phase 6 bench above:** the upstream
|
| 127 |
+
> scores measure *general* capability (knowledge, reasoning, coding, alignment).
|
| 128 |
+
> The Phase 6 deltas measure *hardware-domain* specialization (KiCad, SPICE,
|
| 129 |
+
> schematic extraction). A rank-16 LoRA adapter modifies less than 1% of base
|
| 130 |
+
> weights, so the upstream scores remain approximately the floor — this LoRA
|
| 131 |
+
> adds the Phase 6 deltas on top of these inherited capabilities.
|
| 132 |
+
|
| 133 |
## Citations
|
| 134 |
|
| 135 |
|