--- base_model: Qwen/Qwen3-4B-Instruct-2507 library_name: transformers model_name: qwen3-4b-mascarade-dsp-lora tags: - generated_from_trainer - trl - sft licence: license --- # Model Card for qwen3-4b-mascarade-dsp-lora This model is a fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507). It has been trained using [TRL](https://github.com/huggingface/trl). ## Quick start ```python from transformers import pipeline question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?" generator = pipeline("text-generation", model="Ailiance-fr/qwen3-4b-mascarade-dsp-lora", device="cuda") output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0] print(output["generated_text"]) ``` ## Training procedure This model was trained with SFT. ### Framework versions - TRL: 1.4.0 - Transformers: 5.8.0 - Pytorch: 2.11.0 - Datasets: 4.8.5 - Tokenizers: 0.22.2 ## Benchmark roadmap This LoRA has **not yet been evaluated** through `electron-bench` (the current pipeline supports `gemma-4-E4B` base only). Training was completed with the standard `mlx-lm` LoRA trainer (rank 16, alpha 32, scale 2.0, AdamW LR 1e-5, 500 iters) — full hyperparameters are in the `Training` table above. Planned evaluations: - Perplexity on the validation split of the training data - Functional benchmark on **qwen3**-specific tasks - Comparison vs base `Qwen/Qwen3-4B` Track progress: [ailiance-bench issues](https://github.com/ailiance/ailiance-bench/issues). For reference benchmarks on the `gemma-4-E4B` base, see the [base-vs-LoRA matrix](https://github.com/ailiance/ailiance-bench/blob/main/bench-results/compare_base_vs_lora.md). ## Bench results — ailiance-bench Phase 7 (CUDA, 2026-05-11) Functional eval via the parsers/scorers from [`ailiance/ailiance-bench`](https://github.com/ailiance/ailiance-bench) Phase 1 (`bench_kicad_functional`), ported to CUDA / transformers + PEFT for the Qwen3-4B-Instruct-2507 base. | Dataset | n | Composite score | Duration | |---|---:|---:|---:| | `emc-dsp-power` | 10 | **0.619** | 1137.1s | _Composite score combines structural-parse-ok, component-count match, ground-node presence, etc. — see `bench_kicad_functional.score_*` for the exact formula. Greedy decoding, max_tokens per `GEN_PARAMS`._ ## Upstream base model — official evaluations These are the **official scores** for the unmodified base model [`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507), reported by Alibaba Qwen team. They represent the floor of capability that this LoRA inherits before the hardware-domain fine-tune adapts behavior. | Category | Benchmark | Qwen3-4B-Instruct-2507 | |---------------|-------------------------|-----------------------:| | Knowledge | MMLU-Pro | 69.6 | | Knowledge | MMLU-Redux | 84.2 | | Knowledge | GPQA | 62.0 | | Knowledge | SuperGPQA | 42.8 | | Reasoning | AIME25 | 47.4 | | Reasoning | HMMT25 | 31.0 | | Reasoning | ZebraLogic | 80.2 | | Reasoning | LiveBench 2024-11-25 | 63.0 | | Coding | LiveCodeBench v6 | 35.1 | | Coding | MultiPL-E | 76.8 | | Coding | Aider-Polyglot | 12.9 | | Alignment | IFEval | 83.4 | | Alignment | Arena-Hard v2 | 43.4 | | Alignment | Creative Writing v3 | 83.5 | | Alignment | WritingBench | 83.4 | | Agent | BFCL-v3 | 61.9 | | Agent | TAU1-Retail | 48.7 | | Agent | TAU1-Airline | 32.0 | | Agent | TAU2-Retail | 40.4 | | Multilingual | MultiIF | 69.0 | | Multilingual | MMLU-ProX | 61.6 | | Multilingual | INCLUDE | 60.1 | | Multilingual | PolyMATH | 31.1 | Source: [official Qwen3-4B-Instruct-2507 model card](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507). > **Reading these numbers alongside the Phase 6 bench above:** the upstream > scores measure *general* capability (knowledge, reasoning, coding, alignment). > The Phase 6 deltas measure *hardware-domain* specialization (KiCad, SPICE, > schematic extraction). A rank-16 LoRA adapter modifies less than 1% of base > weights, so the upstream scores remain approximately the floor — this LoRA > adds the Phase 6 deltas on top of these inherited capabilities. ## Citations Cite TRL as: ```bibtex @software{vonwerra2020trl, title = {{TRL: Transformers Reinforcement Learning}}, author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin}, license = {Apache-2.0}, url = {https://github.com/huggingface/trl}, year = {2020} } ``` ## Bench comparison (2026-05-11) ### Base model (Qwen3-4B-Instruct-2507) capability No baseline bench yet for Qwen3-4B-Instruct-2507 in our pipeline. ### This LoRA (tuned) — bench PENDING Will include iact-bench Docker validators per domain + N3 5-axis kicad-cli. Production usage: compiled to Tower Ollama Modelfile, served via gateway alias `ailiance-` (kicad/spice/stm32/emc/embedded/platformio/freecad/dsp/iot/power). ## Cross-domain forgetting check (Phase 9, 2026-05-11) For each domain's eval set (seed=101, n samples held-out), compare this LoRA's Jaccard token-overlap vs the Qwen3-4B-Instruct-2507 **baseline (no adapter)** on the SAME prompts. Negative Δ = the LoRA degrades base behaviour on that domain. | Eval domain | LoRA Jaccard | Δ vs base | |---|---:|---:| | `kicad` | 0.088 | +0.001 | | `spice` | 0.01 | +0.005 | | `stm32` | 0.054 | +0.004 | | `emc` | 0.067 | +0.001 | | `embedded` | 0.075 | +0.001 | | `platformio` | 0.038 | -0.004 | | `freecad` | 0.028 | +0.007 | | `dsp` | 0.094 | -0.007 ⬅ in-domain | | `iot` | 0.065 | -0.003 | | `power` | 0.069 | +0.001 | **In-domain Δ**: -0.007 **Out-of-domain mean Δ**: 0.001