---
language:
- zh
- en
- ja
- ko
- fr
- de
- es
- pt
- ar
- ru
- vi
- th
- id
- it
- tr
- hi
- ms
- nl
- sv
- da
- fi
- pl
- cs
license: apache-2.0
tags:
- mlx
- asr
- speech-recognition
- quantized
- qwen3
- audio
base_model: mlx-community/Qwen3-ASR-1.7B-bf16
---

# Qwen3-ASR-1.7B — Mixed-Precision MLX (audio-8bit / text-4bit)

A mixed-precision quantized version of [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) for Apple Silicon (MLX framework).

| Component | Precision | group_size |
|-----------|-----------|------------|
| `audio_tower.*` | **8-bit** affine | 64 |
| `model.layers.*` | **4-bit** affine | 64 |
| `model.embed_tokens` | **4-bit** affine | 64 |

This configuration was identified as optimal after a systematic ablation study across 16 quantization configurations on Common Voice 22.0 (zh-TW / en / ja / ko).

---

## Performance (n=1000, Common Voice 22.0)

Evaluated against [`mlx-community/Qwen3-ASR-1.7B-4bit`](https://huggingface.co/mlx-community/Qwen3-ASR-1.7B-4bit) on the same benchmark under identical conditions.

| Metric | This model | mlx-community-4bit | Delta |
|--------|-----------|-------------------|-------|
| **Disk size** | **1,253 MB** | 1,529 MB | **−18.0%** |
| zh-TW CER (n=1000) | 3.670% | 3.644% | +0.026pp (within noise) |
| en WER (n=1000) | 6.458% | 6.482% | −0.024pp |
| Speed — zh-TW | **0.244 s/sample** | 0.268 s/sample | **−9.9%** |
| Speed — en | **0.294 s/sample** | 0.318 s/sample | **−7.5%** |
| RAM delta on load | **~0 MB** | +1,095 MB | load-from-weights |

**Verdict**: Accuracy is statistically equivalent to `mlx-community/Qwen3-ASR-1.7B-4bit`. The model is **18% smaller on disk** and **~10% faster** at inference.

> Note: Japanese (ja) and other languages with embed_tokens quantized to 4-bit exhibit hallucination (~172% CER). This is the same behaviour as `mlx-community/Qwen3-ASR-1.7B-4bit` (174% CER) and is an inherent limitation of 4-bit embed_tokens quantization, not specific to this model.

---

## Why a custom loader is needed

`mlx-audio`'s `load_model` hardcodes a `model_quant_predicate` inside `Qwen3ASRForConditionalGeneration` that unconditionally prevents the `audio_tower` from being quantized:

```python
# mlx_audio/stt/models/qwen3_asr.py (~line 779)
def model_quant_predicate(self, path, module, group_size, bits):
    return not path.startswith("audio_tower")
```

Because of this guard, calling `load_model(repo_id)` on this repo would silently load the 8-bit audio weights into un-quantized `Linear` layers, causing a shape mismatch and producing empty transcriptions.

**The workaround** is to load the bf16 base model, apply quantization manually (bypassing `model_quant_predicate`), then replace the weights with the saved ones. The complete loader is shown below.

---

## Usage

### Requirements

```bash
pip install mlx-audio>=0.3.1 mlx>=0.22
```

### Transcription

```python
import mlx.core as mx
import mlx.nn as nn
from mlx_audio.stt.utils import load_model
from mlx_audio.stt.generate import generate_transcription

# ── Step 1: load bf16 base ────────────────────────────────────────────────────
bundle = load_model("mlx-community/Qwen3-ASR-1.7B-bf16")
model  = bundle._model

# ── Step 2: apply the same mixed-precision quantization ──────────────────────
def pred_audio(path, m):
    if not isinstance(m, (nn.Linear, nn.Embedding)):
        return False
    return path.startswith("audio_tower.")

def pred_text(path, m):
    if not isinstance(m, (nn.Linear, nn.Embedding)):
        return False
    return path.startswith("model.") and not path.startswith("model.audio_tower")

nn.quantize(model, group_size=64, bits=8, class_predicate=pred_audio)
nn.quantize(model, group_size=64, bits=4, class_predicate=pred_text)

# ── Step 3: load the saved weights from this repo ────────────────────────────
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download("Alkd/Qwen3-ASR-1.7B-audio8-text4-mlx", "model.safetensors")
saved = list(mx.load(weights_path).items())
model.load_weights(saved, strict=True)
mx.eval(model.parameters())

# ── Step 4: transcribe ────────────────────────────────────────────────────────
# language=None  → auto-detect (recommended for zh-TW / Chinese)
# language="English", "Japanese", "Korean", etc. → force language
result = generate_transcription(bundle, "audio.wav", language=None)

# generate_transcription returns either a str or an STTOutput object
if isinstance(result, str):
    text = result
elif hasattr(result, "text"):
    text = result.text or ""
else:
    text = str(result)

print(text)
```

### Language hints

| Language | `language` argument |
|----------|---------------------|
| Chinese (Mandarin / zh-TW / zh-CN) | `None` (auto-detect) |
| English | `"English"` |
| Japanese | `"Japanese"` ⚠ hallucination risk |
| Korean | `"Korean"` |

---

## How this model was created

### Quantization procedure

```python
import mlx.nn as nn

def pred_audio(path, m):
    return isinstance(m, (nn.Linear, nn.Embedding)) and path.startswith("audio_tower.")

def pred_text(path, m):
    return (isinstance(m, (nn.Linear, nn.Embedding))
            and path.startswith("model.")
            and not path.startswith("model.audio_tower"))

nn.quantize(model, group_size=64, bits=8, class_predicate=pred_audio)  # audio tower → 8-bit
nn.quantize(model, group_size=64, bits=4, class_predicate=pred_text)   # LM layers + embed → 4-bit
```

### Architecture equivalence

The text+embed component (`model.layers` + `model.embed_tokens`) is quantized identically to `mlx-community/Qwen3-ASR-1.7B-4bit`. Running our `pred_text` predicate alone on the bf16 base model reproduces the mlx-community weights **digit-for-digit** (verified across all four languages on the same benchmark).

The only difference is the `audio_tower`: mlx-community keeps it in bf16 (3,580 MB → 1,529 MB), while this model quantizes it to 8-bit (→ 1,253 MB), saving an additional **276 MB**.

### Component sizes

| Component | bf16 | This model | mlx-community-4bit |
|-----------|------|------------|--------------------|
| audio_tower | ~635 MB | **~341 MB (8-bit)** | ~635 MB (bf16) |
| model.layers | ~793 MB | ~268 MB (4-bit) | ~268 MB (4-bit) |
| model.embed_tokens | ~623 MB | ~175 MB (4-bit) | ~175 MB (4-bit) |
| **Total (on-disk)** | ~4,076 MB | **~1,253 MB** | ~1,529 MB |

---

## Ablation study summary (n=250 per language)

The configuration was selected from 16 candidates evaluated on Common Voice 22.0 (zh-TW, en, ja, ko):

| Config | Size | zh-TW CER | en WER | ja CER | ko CER |
|--------|------|-----------|--------|--------|--------|
| bf16-FULL (baseline) | 4,076 MB | 4.57% | 7.69% | 31.91% | 6.33% |
| mlx-community-4bit (anchor) | 1,603 MB | 4.569% | 8.559% | 174.175% ⚠ | 8.077% |
| **AUDIO-8-EMBED-TEXT-4 (this model)** | **1,310 MB** | **4.467%** | **8.472%** | 172.353% ⚠ | **7.955%** |

> ⚠ CER > 100% indicates hallucination (model output longer than reference). This is caused by 4-bit quantization of `embed_tokens` and affects both this model and `mlx-community/Qwen3-ASR-1.7B-4bit` equally for Japanese.

---

## Limitations

- **Japanese / CJK hallucination**: 4-bit `embed_tokens` causes hallucinations for Japanese (CER ~172%). Same issue exists in `mlx-community/Qwen3-ASR-1.7B-4bit`.
- **Custom loader required**: See [Why a custom loader is needed](#why-a-custom-loader-is-needed) above.
- **MLX only**: Weights are in MLX safetensors format and are not compatible with PyTorch/Transformers.

---

## Citation

If you use this model, please cite the original Qwen3-ASR work and mlx-audio:

```bibtex
@misc{qwen3asr2025,
  title  = {Qwen3-ASR},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen3-ASR-1.7B}
}
```

## License

Apache 2.0 — same as the base model.