majentik/Gemma-4-E4B-MERaLiON-Speech-LoRA-MNSC-MLX

Gemma-4-E4B-MERaLiON Speech LoRA for Singapore English (MLX)

A speech-language model for Singapore English ASR, built by composing:

• Decoder: Gemma-4-E4B (Google) — quantized to MLX 8-bit via RotorQuant
• Speech encoder + adaptor: MERaLiON-3 speech tower — Whisper-large-v3-encoder-based acoustic features
• Projector: trained from scratch (bridges 3584-d speech → 2560-d Gemma embedding space)
• LoRA adapter: trained for Singapore English ASR alignment

The whole bundle runs natively on Apple Silicon via mlx. The LoRA is switchable — disable it for plain text generation, enable it for speech transcription. No merging required.

---

Results

Evaluated on a held-out Singapore English ASR test split (3000 clips):

Model	WER ↓
MERaLiON-2 (baseline, end-to-end speech-LM)	25.78%
This model (Gemma-4-E4B + MERaLiON speech tower + ours)	18.86%
Improvement	−6.92pp absolute (−26.8% relative)

WER is computed with jiwer's standard normalization: lowercase, punctuation removed, whitespace collapsed. <SpeakerN>: dialogue prefixes (occasionally emitted by the decoder) are stripped before scoring — this happens natively inside transcribe_with_pipeline, not as a post-hoc evaluation hack.

---

Provenance

This is a composite work. See PROVENANCE.md for the full chain-of-custody on every weight file.

Component	Source	License
Decoder	majentik/gemma-4-E4B-RotorQuant-MLX-8bit ← google/gemma-4-E4B-it	Gemma Terms
Speech encoder + adaptor	majentik/MERaLiON-3-10B-MLX-4bit ← MERaLiON/MERaLiON-3-10B	MERaLiON Public License v2
Projector	Trained by majentik for Singapore English ASR alignment	MIT
LoRA adapter	Trained by majentik for Singapore English ASR alignment	MIT
Composite work (this repo)	—	MERaLiON Public License v2 (most restrictive in chain)

The composite is bound by the MERaLiON Public License v2 because we include MERaLiON-3 weights. Gemma's terms also apply to the decoder.

---

Precision summary

Component	Storage dtype
Decoder weights	MLX 8-bit affine (uint32-packed) + bf16 norms/embeds
Speech encoder	fp16
Adaptor	fp16
Projector (ours)	fp32
LoRA adapters (ours)	fp32

The decoder is the only quantized component. The speech tower stays at fp16 (it came that way from the MERaLiON-3-10B-MLX-4bit repo, where only the decoder shards are 4-bit). Our trained additions are at fp32 for maximum fidelity at minimal cost (208 MB combined).

---

Bundle layout

.
├── config.json                # composition manifest
├── PROVENANCE.md              # full chain-of-custody
├── README.md                  # this file
├── decoder/                   # Gemma-4-E4B 8bit MLX (~8.4 GB)
├── speech_encoder/            # MERaLiON-3 encoder + adaptor (~1.4 GB, fp16)
├── projector/                 # our trained 3584→3072→2560 projector (75 MB, fp32)
└── lora/                      # speech-alignment LoRA adapter (133 MB, fp32)

Total: ~9.9 GB.

---

Usage

Install the inference code:

pip install git+https://github.com/ajentik/elderwise-mlx.git
# or clone and pip install -e .

Then:

from elderwise.inference import load_pipeline, transcribe_with_pipeline
from huggingface_hub import snapshot_download

# Download the bundle
local = snapshot_download("majentik/Gemma-4-E4B-MERaLiON-Speech-LoRA-MNSC-MLX")

pipeline = load_pipeline(
    meralion_dir=f"{local}/speech_encoder",
    gemma_id=f"{local}/decoder",        # local path, not HF id
    projector_path=f"{local}/projector",
    lora_path=f"{local}/lora",
    lora_rank=16,
    lora_target_names=(
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ),
)

text = transcribe_with_pipeline(pipeline, "your_audio.wav")
print(text)

The pipeline:

1. Resamples audio to 16 kHz.
2. Runs the MERaLiON encoder → adaptor to get 3584-d speech features.
3. Projects them into Gemma's embedding space.
4. Prompts Gemma with "Transcribe the following audio: " + projected speech embeddings + the PLE bypass (see Alignment notes).
5. Greedy-decodes up to 128 tokens, strips any <SpeakerN>: prefix, returns the result.

---

Alignment notes

What is included

• Projector: from scratch, 7 layers worth of params: LayerNorm(3584) → Linear(3584→3072) → SiLU → Linear(3072→2560) → RMSNorm(2560). The final RMSNorm matters — Gemma-4 normalizes its embeddings to ‖x‖ ≈ 1.0 and projector outputs need to live on that manifold.
• LoRA adapter: switchable speech-alignment adapter. Speech mode enables the adapter; text mode disables it and falls back to base decoder behavior.

Data

The alignment adapter was trained on Singapore English ASR data. The public model card intentionally keeps internal optimization details high-level.

The non-obvious bit: Gemma-4's PLE

Gemma-4 uses Per-Layer Embeddings (PLE) as a side-channel into every transformer layer. When you call the model with inputs=None and only input_embeddings (as you must for speech, where there's no clean token id), mlx_lm runs nearest-neighbor recovery — it argmins your embeddings against the vocab table and uses the resulting fake token ids to populate PLE.

If your speech embeddings are off-manifold (they will be without the final RMSNorm), the nearest-neighbor returns garbage rare tokens, and Gemma reverts to its strongest priors (PII redaction templates, in our case — we measured 592% WER before fixing this).

The fix lives in elderwise/per_layer.py: we bypass nearest-neighbor by building per_layer_inputs explicitly — zeros for speech positions (silences the PLE side-channel where it has no signal anyway), and real embed_tokens_per_layer(token_ids) * scale for prompt/transcript positions.

This + three other compounding fixes (projector RMSNorm, prompt tokenization match, LoRA scale restore) is what made the model actually condition on speech. See the elderwise repo for the full diagnostic trail.

---

Limitations

• Singapore English only. Not evaluated on other accents/locales.
• Substitution errors on rare proper nouns. The model phonetically captures unusual names but substitutes (e.g., "Tendon" → "Tender", "Vindaloo" survives but "Fallon" → "Felten"). This is a data scale problem, not a capacity problem.
• 8-bit decoder may show occasional artifacts vs. the full bf16 base.
• No streaming. Audio is encoded in one shot, then transcribed.
• No timestamps, no diarization. Plain transcription only. (The <SpeakerN>: artifacts are stripped, not used.)

---

Citation

If you use this model, please cite:

@misc{gemma4_meralion_speech_lora_mnsc_mlx,
  title  = {Gemma-4-E4B-MERaLiON Speech LoRA for Singapore English},
  author = {majentik},
  year   = {2026},
  url    = {https://huggingface.co/majentik/Gemma-4-E4B-MERaLiON-Speech-LoRA-MNSC-MLX}
}

And the upstream sources:

• Google Gemma
• MERaLiON-3
• Whisper

---

Acknowledgements

• Google DeepMind — Gemma-4 base model
• MERaLiON team — MERaLiON-3 speech tower
• OpenAI — Whisper encoder architecture
• Apple ML Research — MLX framework