Audex-30B-A3B-W4A16-v2

⚠️ Experimental / needs more VRAM headroom than a single 24GB GPU provides. See Known limitations before using this in production. If you just want something proven-stable on a 24GB card, use Audex-30B-A3B-W4A16 instead (no shared-expert/embedding quantization, but rock solid).

A second GPTQ quantization pass on top of the v1 recipe, for nvidia/Nemotron-Labs-Audex-30B-A3B. Adds two more quantized groups beyond v1's routed-MoE-only INT4:

Component	Precision	BF16 size → quantized
Routed MoE experts (same as v1)	INT4, group_size=64	58.75GB → ~14GB
Shared expert (new in v2)	INT4, group_size=64	0.92GB → ~0.23GB
Embeddings + lm_head (new in v2)	INT8, channel-wise	2.21GB → stored dense (see below)

Attention/Mamba mixer projections, audio encoder, and router are left untouched at BF16, same reasoning as v1 (most quality-sensitive, least structurally redundant).

Goal of this pass: free enough VRAM to also fit the 2.4GB speech decoder on the same 24GB GPU as the LLM weights, so TTS synthesis doesn't have to run on CPU (CPU decoder measured at 2.7–3.2x slower than real-time — not viable for interactive use). Net saving from the new groups alone is modest (0.69GB from the shared expert; the embed/lm_head group did not actually compress — see below), which is why this remains tight on 24GB.

Known limitations

• Not stable for sustained real-world inference on a single 24GB GPU. It loads and passes lightweight smoke tests at --gpu-memory-utilization values around 0.88, but under real multi-request traffic it OOMs inside the Mamba mixer's _chunk_state_fwd (SSD scan) — this was reproduced across a GMU sweep from 0.83 to 0.90 and is a hard memory-margin problem, not a config-tuning one. The math: ~20GB quantized weights + 2.4GB decoder leaves too little consistent margin for KV cache + Mamba scan activation memory on a 24GB card.
• The embeddings/lm_head group is not actually compressed. vLLM's VocabParallelEmbedding/lm_head layer types for this architecture aren't wired to consume a quant_method, so despite being INT8-quantized during the GPTQ pass, the resulting weights are stored in compressed-tensors' "dense" format (full-size tensors, quantization scale/zero-point metadata computed but unused at inference time). In practice this group contributes close to zero VRAM savings over BF16 today. It's included here in case a future vLLM version wires up that layer type, or if you use a different inference stack that respects the stored quantization scheme.
• Given the above, this checkpoint's real usable savings vs v1 are ~0.69GB (shared expert only) — likely not enough headroom to justify the added complexity over v1 on a single 24GB GPU. It may be more useful on a GPU with more headroom (32GB+) where you want the decoder on-GPU with comfortable margin, or as a base for further work (e.g. wiring up embed/lm_head compression properly, or additionally quantizing the decoder itself).
• Not independently evaluated for quality regression vs BF16 or vs v1.

Quantization recipe

GPTQModifier(
    config_groups={
        "group_0": QuantizationScheme(  # routed experts
            targets=[r"re:.*mixer\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
            weights=QuantizationArgs(num_bits=4, type="int", symmetric=True,
                                      strategy="group", group_size=64),
        ),
        "group_1": QuantizationScheme(  # shared experts
            targets=[r"re:.*mixer\.shared_experts\.(gate_proj|up_proj|down_proj)$"],
            weights=QuantizationArgs(num_bits=4, type="int", symmetric=True,
                                      strategy="group", group_size=64),
        ),
        "group_2": QuantizationScheme(  # embeddings + lm_head
            targets=[r"re:^backbone\.embeddings$", r"re:^lm_head$"],
            weights=QuantizationArgs(num_bits=8, type="int", symmetric=True,
                                      strategy="channel"),
        ),
    },
    ignore=[r"re:.*audio_encoder.*", r"re:.*sound.*", r"re:.*projector.*", r"re:.*mixer\.gate.*"],
)

Calibrated on 512 samples from HuggingFaceH4/ultrachat_200k, max sequence length 2048. Run on Modal (A100-80GB, 96GB host RAM) — the local machine's 32GB RAM was insufficient (severe swap thrashing during weight loading).

The gate_proj pattern is included in the routed/shared expert target regexes even though it isn't separately calibrated/quantized as its own tensor for MoE experts in this architecture — it's required so vLLM's FusedMoE scheme lookup (which checks all three projection names) succeeds; the regex match alone satisfies the "all MoE projections need matching schemes" requirement.

Usage (vLLM)

Same as v1 — requires the audex_30b_a3b_vllm out-of-tree plugin bundled with the original model repo. Drop-in weights replacement for the checkpoint_folder_full/ subfolder layout.

python cascaded_s2s_web_server.py \
  --model /path/to/Audex-30B-A3B-W4A16-v2 \
  --gpu-memory-utilization 0.88 \
  --decoder-device cuda:0   # fits at this GMU on a 24GB card, but not stable under load — see caveats

If you have more VRAM headroom (32GB+, or a second GPU dedicated to the decoder), this checkpoint's weights are still a valid ~1GB-smaller alternative to v1.