clemsail commited on
Commit
2e2fe96
·
verified ·
1 Parent(s): a519003

docs: Phase 7 CUDA bench results (dsp)

Browse files
Files changed (1) hide show
  1. README.md +9 -28
README.md CHANGED
@@ -59,34 +59,15 @@ Track progress: [ailiance-bench issues](https://github.com/ailiance/ailiance-ben
59
  For reference benchmarks on the `gemma-4-E4B` base, see the
60
  [base-vs-LoRA matrix](https://github.com/ailiance/ailiance-bench/blob/main/bench-results/compare_base_vs_lora.md).
61
 
62
- ## Bench results — `ailiance-bench` Phase 6 (2026-05-11)
63
-
64
- Evaluated as part of the `mascarade` adapter family in
65
- [`ailiance/ailiance-bench`](https://github.com/ailiance/ailiance-bench)
66
- commit [`46801af`](https://github.com/ailiance/ailiance-bench/commit/46801af),
67
- on 7 hardware-design tasks against the base `gemma-e4b-eu-kiki-base` model.
68
-
69
- | Phase | Task | base | +mascarade | Δ vs base |
70
- |-------|---------------------|------:|-----------:|------------:|
71
- | P1 | kicad-dsl | 0.090 | 0.090 | 0 |
72
- | P1 | kicad-pcb | 0.010 | 0.010 | 0 |
73
- | P1 | spice-sim | 0.425 | 0.176 | **−25** |
74
- | P2 | kicad-sch-gen | 0.420 | 0.400 | −2 |
75
- | P3 | **kicad-sch-extract** | 0.308 | **0.785** | **+48** 🥇 |
76
- | P4 | kicad-erc-abs | 0.060 | 0.060 | 0 |
77
- | P5 | kicad-erc-delta | 0.060 | 0.060 | 0 |
78
-
79
- **Verdict:** champion on **P3 KiCad schematic extraction (+48 pts) 🥇**. Recommended use: **DSP code analysis** (filter design review, algorithm understanding). Regresses on SPICE simulation reasoning (−25); prefer `eu-kiki` (the Gemma-4 curriculum adapter) for SPICE-sim and KiCad DSL/PCB generation.
80
-
81
- > **Cross-base disclaimer:** Phase 6 benchmarked the mascarade family on a
82
- > `gemma-e4b-eu-kiki-base` reference model for cross-adapter comparison.
83
- > This adapter is trained on `Qwen/Qwen3-4B-Instruct-2507` for production
84
- > deployment via Tower Ollama `:8004`. Domain semantics transfer; absolute
85
- > numbers may differ on the production base.
86
-
87
- See full scoreboard:
88
- [ailiance-bench Phase 6 README](https://github.com/ailiance/ailiance-bench#scoreboard-lora-phase-6--2026-05-11)
89
- · interactive viewer: [Ailiance-fr/playground](https://huggingface.co/spaces/Ailiance-fr/playground).
90
 
91
  ## Upstream base model — official evaluations
92
 
 
59
  For reference benchmarks on the `gemma-4-E4B` base, see the
60
  [base-vs-LoRA matrix](https://github.com/ailiance/ailiance-bench/blob/main/bench-results/compare_base_vs_lora.md).
61
 
62
+ ## Bench results — ailiance-bench Phase 7 (CUDA, 2026-05-11)
63
+
64
+ Functional eval via the parsers/scorers from [`ailiance/ailiance-bench`](https://github.com/ailiance/ailiance-bench) Phase 1 (`bench_kicad_functional`), ported to CUDA / transformers + PEFT for the Qwen3-4B-Instruct-2507 base.
65
+
66
+ | Dataset | n | Composite score | Duration |
67
+ |---|---:|---:|---:|
68
+ | `emc-dsp-power` | 10 | **0.619** | 1137.1s |
69
+
70
+ _Composite score combines structural-parse-ok, component-count match, ground-node presence, etc. — see `bench_kicad_functional.score_*` for the exact formula. Greedy decoding, max_tokens per `GEN_PARAMS`._
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
 
72
  ## Upstream base model — official evaluations
73