---
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: transformers
model_name: qwen3-4b-mascarade-iot-lora
tags:
- generated_from_trainer
- sft
- trl
licence: license
---

# Model Card for qwen3-4b-mascarade-iot-lora

This model is a fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).
It has been trained using [TRL](https://github.com/huggingface/trl).

## Quick start

```python
from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="Ailiance-fr/qwen3-4b-mascarade-iot-lora", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])
```

## Training procedure

 



This model was trained with SFT.

### Framework versions

- TRL: 1.4.0
- Transformers: 5.8.0
- Pytorch: 2.11.0
- Datasets: 4.8.5
- Tokenizers: 0.22.2

## Bench results — iact-bench Phase 8 (Docker validators, 2026-05-11)

Functional eval via the [`iact-bench`](https://github.com/electron-rare/iact-bench) v0.2 Docker sandbox validators (no-network, read-only rootfs, uid 1000 dropped caps, `--tmpfs /tmp:size=256m`). Source LoRA: `Ailiance-fr/qwen3-4b-mascarade-iot-lora`. Eval samples drawn with seed=101 from `Ailiance-fr/mascarade-iot-dataset`.

| Metric | Value |
|---|---:|
| Validator | `judge-only` |
| Tmpfs cap | `256m` |
| Samples | 10 |
| **Pass rate** | **0.0** |
| Total validator wall-clock | 0.0s |

_Pass = sandboxed compile/exec succeeds (exit_code=0). See `iact-bench/configs/domain_validators.yaml` for the exact toolchain invocation (arm-none-eabi-gcc / arduino-cli / freecadcmd depending on domain)._

## Upstream base model — official evaluations

These are the **official scores** for the unmodified base model
[`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
reported by Alibaba Qwen team. They represent the floor of capability that this
LoRA inherits before the hardware-domain fine-tune adapts behavior.

| Category      | Benchmark               | Qwen3-4B-Instruct-2507 |
|---------------|-------------------------|-----------------------:|
| Knowledge     | MMLU-Pro                | 69.6                   |
| Knowledge     | MMLU-Redux              | 84.2                   |
| Knowledge     | GPQA                    | 62.0                   |
| Knowledge     | SuperGPQA               | 42.8                   |
| Reasoning     | AIME25                  | 47.4                   |
| Reasoning     | HMMT25                  | 31.0                   |
| Reasoning     | ZebraLogic              | 80.2                   |
| Reasoning     | LiveBench 2024-11-25    | 63.0                   |
| Coding        | LiveCodeBench v6        | 35.1                   |
| Coding        | MultiPL-E               | 76.8                   |
| Coding        | Aider-Polyglot          | 12.9                   |
| Alignment     | IFEval                  | 83.4                   |
| Alignment     | Arena-Hard v2           | 43.4                   |
| Alignment     | Creative Writing v3     | 83.5                   |
| Alignment     | WritingBench            | 83.4                   |
| Agent         | BFCL-v3                 | 61.9                   |
| Agent         | TAU1-Retail             | 48.7                   |
| Agent         | TAU1-Airline            | 32.0                   |
| Agent         | TAU2-Retail             | 40.4                   |
| Multilingual  | MultiIF                 | 69.0                   |
| Multilingual  | MMLU-ProX               | 61.6                   |
| Multilingual  | INCLUDE                 | 60.1                   |
| Multilingual  | PolyMATH                | 31.1                   |

Source: [official Qwen3-4B-Instruct-2507 model card](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).

> **Reading these numbers alongside the Phase 6 bench above:** the upstream
> scores measure *general* capability (knowledge, reasoning, coding, alignment).
> The Phase 6 deltas measure *hardware-domain* specialization (KiCad, SPICE,
> schematic extraction). A rank-16 LoRA adapter modifies less than 1% of base
> weights, so the upstream scores remain approximately the floor — this LoRA
> adds the Phase 6 deltas on top of these inherited capabilities.

## Citations



Cite TRL as:
    
```bibtex
@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}
```


## Bench comparison (2026-05-11)

### Base model (Qwen3-4B-Instruct-2507) capability

No baseline bench yet for Qwen3-4B-Instruct-2507 in our pipeline.

### This LoRA (tuned) — bench PENDING

Will include iact-bench Docker validators per domain + N3 5-axis kicad-cli.
Production usage: compiled to Tower Ollama Modelfile, served via gateway alias
`ailiance-<domain>` (kicad/spice/stm32/emc/embedded/platformio/freecad/dsp/iot/power).

## Cross-domain forgetting check (Phase 9, 2026-05-11)

For each domain's eval set (seed=101, n samples held-out), compare this LoRA's Jaccard token-overlap vs the Qwen3-4B-Instruct-2507 **baseline (no adapter)** on the SAME prompts. Negative Δ = the LoRA degrades base behaviour on that domain.

| Eval domain | LoRA Jaccard | Δ vs base |
|---|---:|---:|
| `kicad` | 0.083 | -0.004 |
| `spice` | 0.006 | +0.001 |
| `stm32` | 0.075 | +0.025 |
| `emc` | 0.069 | +0.003 |
| `embedded` | 0.085 | +0.011 |
| `platformio` | 0.057 | +0.015 |
| `freecad` | 0.035 | +0.014 |
| `dsp` | 0.109 | +0.008 |
| `iot` | 0.06 | -0.008 ⬅ in-domain |
| `power` | 0.076 | +0.008 |

**In-domain Δ**: -0.008  **Out-of-domain mean Δ**: 0.009