Model Card for qwen3-4b-mascarade-dsp-lora

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507. It has been trained using TRL.

Quick start

from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="Ailiance-fr/qwen3-4b-mascarade-dsp-lora", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training procedure

This model was trained with SFT.

Framework versions

• TRL: 1.4.0
• Transformers: 5.8.0
• Pytorch: 2.11.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Benchmark roadmap

This LoRA has not yet been evaluated through electron-bench (the current pipeline supports gemma-4-E4B base only). Training was completed with the standard mlx-lm LoRA trainer (rank 16, alpha 32, scale 2.0, AdamW LR 1e-5, 500 iters) — full hyperparameters are in the Training table above.

Planned evaluations:

• Perplexity on the validation split of the training data
• Functional benchmark on qwen3-specific tasks
• Comparison vs base Qwen/Qwen3-4B

Track progress: ailiance-bench issues.

For reference benchmarks on the gemma-4-E4B base, see the base-vs-LoRA matrix.

Bench results — ailiance-bench Phase 7 (CUDA, 2026-05-11)

Functional eval via the parsers/scorers from ailiance/ailiance-bench Phase 1 (bench_kicad_functional), ported to CUDA / transformers + PEFT for the Qwen3-4B-Instruct-2507 base.

Dataset	n	Composite score	Duration
emc-dsp-power	10	0.619	1137.1s

Composite score combines structural-parse-ok, component-count match, ground-node presence, etc. — see bench_kicad_functional.score_* for the exact formula. Greedy decoding, max_tokens per GEN_PARAMS.

Upstream base model — official evaluations

These are the official scores for the unmodified base model Qwen/Qwen3-4B-Instruct-2507, reported by Alibaba Qwen team. They represent the floor of capability that this LoRA inherits before the hardware-domain fine-tune adapts behavior.

Category	Benchmark	Qwen3-4B-Instruct-2507
Knowledge	MMLU-Pro	69.6
Knowledge	MMLU-Redux	84.2
Knowledge	GPQA	62.0
Knowledge	SuperGPQA	42.8
Reasoning	AIME25	47.4
Reasoning	HMMT25	31.0
Reasoning	ZebraLogic	80.2
Reasoning	LiveBench 2024-11-25	63.0
Coding	LiveCodeBench v6	35.1
Coding	MultiPL-E	76.8
Coding	Aider-Polyglot	12.9
Alignment	IFEval	83.4
Alignment	Arena-Hard v2	43.4
Alignment	Creative Writing v3	83.5
Alignment	WritingBench	83.4
Agent	BFCL-v3	61.9
Agent	TAU1-Retail	48.7
Agent	TAU1-Airline	32.0
Agent	TAU2-Retail	40.4
Multilingual	MultiIF	69.0
Multilingual	MMLU-ProX	61.6
Multilingual	INCLUDE	60.1
Multilingual	PolyMATH	31.1

Source: official Qwen3-4B-Instruct-2507 model card.

Reading these numbers alongside the Phase 6 bench above: the upstream scores measure general capability (knowledge, reasoning, coding, alignment). The Phase 6 deltas measure hardware-domain specialization (KiCad, SPICE, schematic extraction). A rank-16 LoRA adapter modifies less than 1% of base weights, so the upstream scores remain approximately the floor — this LoRA adds the Phase 6 deltas on top of these inherited capabilities.

Citations

Cite TRL as:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}

Bench comparison (2026-05-11)

Base model (Qwen3-4B-Instruct-2507) capability

No baseline bench yet for Qwen3-4B-Instruct-2507 in our pipeline.

This LoRA (tuned) — bench PENDING

Will include iact-bench Docker validators per domain + N3 5-axis kicad-cli. Production usage: compiled to Tower Ollama Modelfile, served via gateway alias ailiance-<domain> (kicad/spice/stm32/emc/embedded/platformio/freecad/dsp/iot/power).

Cross-domain forgetting check (Phase 9, 2026-05-11)

For each domain's eval set (seed=101, n samples held-out), compare this LoRA's Jaccard token-overlap vs the Qwen3-4B-Instruct-2507 baseline (no adapter) on the SAME prompts. Negative Δ = the LoRA degrades base behaviour on that domain.

Eval domain	LoRA Jaccard	Δ vs base
kicad	0.088	+0.001
spice	0.01	+0.005
stm32	0.054	+0.004
emc	0.067	+0.001
embedded	0.075	+0.001
platformio	0.038	-0.004
freecad	0.028	+0.007
dsp	0.094	-0.007 ⬅ in-domain
iot	0.065	-0.003
power	0.069	+0.001

In-domain Δ: -0.007 Out-of-domain mean Δ: 0.001