README.md

---
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: transformers
model_name: qwen3-4b-mascarade-dsp-lora
tags:
- generated_from_trainer
- trl
- sft
licence: license
---

# Model Card for qwen3-4b-mascarade-dsp-lora

This model is a fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).
It has been trained using [TRL](https://github.com/huggingface/trl).

## Quick start

```python
from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="Ailiance-fr/qwen3-4b-mascarade-dsp-lora", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])
```

## Training procedure

 



This model was trained with SFT.

### Framework versions

- TRL: 1.4.0
- Transformers: 5.8.0
- Pytorch: 2.11.0
- Datasets: 4.8.5
- Tokenizers: 0.22.2

## Benchmark roadmap

This LoRA has **not yet been evaluated** through `electron-bench` (the current
pipeline supports `gemma-4-E4B` base only). Training was completed with the
standard `mlx-lm` LoRA trainer (rank 16, alpha 32, scale 2.0, AdamW
LR 1e-5, 500 iters) — full hyperparameters are in the `Training` table above.

Planned evaluations:

- Perplexity on the validation split of the training data
- Functional benchmark on **qwen3**-specific tasks
- Comparison vs base `Qwen/Qwen3-4B`

Track progress: [ailiance-bench issues](https://github.com/ailiance/ailiance-bench/issues).

For reference benchmarks on the `gemma-4-E4B` base, see the
[base-vs-LoRA matrix](https://github.com/ailiance/ailiance-bench/blob/main/bench-results/compare_base_vs_lora.md).

## Bench results — ailiance-bench Phase 7 (CUDA, 2026-05-11)

Functional eval via the parsers/scorers from [`ailiance/ailiance-bench`](https://github.com/ailiance/ailiance-bench) Phase 1 (`bench_kicad_functional`), ported to CUDA / transformers + PEFT for the Qwen3-4B-Instruct-2507 base.

| Dataset | n | Composite score | Duration |
|---|---:|---:|---:|
| `emc-dsp-power` | 10 | **0.619** | 1137.1s |

_Composite score combines structural-parse-ok, component-count match, ground-node presence, etc. — see `bench_kicad_functional.score_*` for the exact formula. Greedy decoding, max_tokens per `GEN_PARAMS`._

## Upstream base model — official evaluations

These are the **official scores** for the unmodified base model
[`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
reported by Alibaba Qwen team. They represent the floor of capability that this
LoRA inherits before the hardware-domain fine-tune adapts behavior.

| Category      | Benchmark               | Qwen3-4B-Instruct-2507 |
|---------------|-------------------------|-----------------------:|
| Knowledge     | MMLU-Pro                | 69.6                   |
| Knowledge     | MMLU-Redux              | 84.2                   |
| Knowledge     | GPQA                    | 62.0                   |
| Knowledge     | SuperGPQA               | 42.8                   |
| Reasoning     | AIME25                  | 47.4                   |
| Reasoning     | HMMT25                  | 31.0                   |
| Reasoning     | ZebraLogic              | 80.2                   |
| Reasoning     | LiveBench 2024-11-25    | 63.0                   |
| Coding        | LiveCodeBench v6        | 35.1                   |
| Coding        | MultiPL-E               | 76.8                   |
| Coding        | Aider-Polyglot          | 12.9                   |
| Alignment     | IFEval                  | 83.4                   |
| Alignment     | Arena-Hard v2           | 43.4                   |
| Alignment     | Creative Writing v3     | 83.5                   |
| Alignment     | WritingBench            | 83.4                   |
| Agent         | BFCL-v3                 | 61.9                   |
| Agent         | TAU1-Retail             | 48.7                   |
| Agent         | TAU1-Airline            | 32.0                   |
| Agent         | TAU2-Retail             | 40.4                   |
| Multilingual  | MultiIF                 | 69.0                   |
| Multilingual  | MMLU-ProX               | 61.6                   |
| Multilingual  | INCLUDE                 | 60.1                   |
| Multilingual  | PolyMATH                | 31.1                   |

Source: [official Qwen3-4B-Instruct-2507 model card](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).

> **Reading these numbers alongside the Phase 6 bench above:** the upstream
> scores measure *general* capability (knowledge, reasoning, coding, alignment).
> The Phase 6 deltas measure *hardware-domain* specialization (KiCad, SPICE,
> schematic extraction). A rank-16 LoRA adapter modifies less than 1% of base
> weights, so the upstream scores remain approximately the floor — this LoRA
> adds the Phase 6 deltas on top of these inherited capabilities.

## Citations



Cite TRL as:
    
```bibtex
@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}
```


## Bench comparison (2026-05-11)

### Base model (Qwen3-4B-Instruct-2507) capability

No baseline bench yet for Qwen3-4B-Instruct-2507 in our pipeline.

### This LoRA (tuned) — bench PENDING

Will include iact-bench Docker validators per domain + N3 5-axis kicad-cli.
Production usage: compiled to Tower Ollama Modelfile, served via gateway alias
`ailiance-<domain>` (kicad/spice/stm32/emc/embedded/platformio/freecad/dsp/iot/power).

## Cross-domain forgetting check (Phase 9, 2026-05-11)

For each domain's eval set (seed=101, n samples held-out), compare this LoRA's Jaccard token-overlap vs the Qwen3-4B-Instruct-2507 **baseline (no adapter)** on the SAME prompts. Negative Δ = the LoRA degrades base behaviour on that domain.

| Eval domain | LoRA Jaccard | Δ vs base |
|---|---:|---:|
| `kicad` | 0.088 | +0.001 |
| `spice` | 0.01 | +0.005 |
| `stm32` | 0.054 | +0.004 |
| `emc` | 0.067 | +0.001 |
| `embedded` | 0.075 | +0.001 |
| `platformio` | 0.038 | -0.004 |
| `freecad` | 0.028 | +0.007 |
| `dsp` | 0.094 | -0.007 ⬅ in-domain |
| `iot` | 0.065 | -0.003 |
| `power` | 0.069 | +0.001 |

**In-domain Δ**: -0.007  **Out-of-domain mean Δ**: 0.001