Hugging Face's logo

AMAImedia
/
NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16

Automatic Speech Recognition
Safetensors
qwen-asr
qwen3_asr
forced-alignment
timestamp-prediction
speech-text-alignment
aligner
qwen3
qwen3-asr
qwen3-omni
alibaba
nar
non-autoregressive
112-languages
multilingual
dubbing
subtitles
noesis
dhcf-fno
unified-training
lora-merged
Model card Files
xet
 Community

NOESIS / AMAImedia

Last updated: 2026-08-29

Released as part of the NOESIS Professional Multilingual Dubbing Automation Platform (framework: DHCF-FNO — Deterministic Hybrid Control Framework for Frozen Neural Operators).

Qwen3-ForcedAligner-0.6B (NOESIS bundle, 112LANG, Unified)

Forced-alignment / timestamp-prediction model from the Qwen3-ASR family by Alibaba's Qwen team. Aligns text-speech pairs and returns word/character-level timestamps for up to 5 minutes of speech. Extended from upstream 11 langs to 112 langs via NOESIS unified training (single-run on top of v3=98LANG). Bundled as the timestamp generator for the NOESIS Professional Multilingual Dubbing Automation Platform (DHCF-FNO).

Lineage

Bundle Langs trained Date Status
Upstream Qwen/Qwen3-ForcedAligner-0.6B 11 (zh, en, yue, fr, de, it, ja, ko, pt, ru, es) 2026-04 base
NOESIS-...-30LANG (v1, Phase D) +19 -> 30 2026-05-12 published, immutable
NOESIS-...-54LANG (v2, Phase F) +24 -> 54 2026-05-14 published, immutable
NOESIS-...-98LANG (v3, Phase G) +43 -> 97 trained + yue upstream = 98 2026-05-15 published, immutable
NOESIS-...-112LANG (v6, Unified) <- you are here +14 new -> 101 trained + 11 base_pretrained = 112 total 2026-05-20 published

Each successive bundle is a drop-in replacement for the previous one: identical wrapper API (Qwen3ForcedAligner.from_pretrained(...)), identical file layout, identical inference path. Prior bundles remain on disk as fallbacks.

Unified training summary (v6, sealed R-UNIFIED-EXPANSION-FA-112-SINGLE-RUN)

Property Value
Strategy Single training run on top of v3=98LANG, NO intermediate v4/v5
Trainable langs 101 = 87 existing refined + 14 new (xh zu yo wo kmr skr ti tk ug mvy eu gn rw qxp)
Base-only langs (no training data) 11 = en zh es pt ru ja de fr ko it yue
Training data 239 429 items, 532.1 h ground truth
Sampler StratifiedLangSampler, target_new_lang_fraction=0.15
Loss weighting (per-sample) GT=1.0 / NT-166 self-pseudo=0.7 / NT-178 MMS-labeler=0.5
Recipe Classical transformers.Trainer + custom pad_collate (NOT NT-218 SFTTrainer -- revoked 2026-05-19)
LoRA r=32 / alpha=64, NF4 base + bf16 compute, save_steps=50
Steps 7482 / 7482 (1 epoch)
Final loss 0.55 (steady, no NaN, no divergence)
Hardware RTX 3060 Laptop 6 GB
Wall clock 27 h 23 min

Test results (full suite, 2026-05-20)

Reproduce: .conda/python.exe src/full_test_suite_112lang.py Full report JSON: logs/full_test_suite_112lang.json Full testing playbook (3 test types + interpretation guide): docs/TESTING_GUIDE.md

Human listening verdict (2026-05-20, all 112 languages reviewed)

Founder Ilia Bolotnikov reviewed karaoke MP4s for every language in this bundle (112/112 covered, output in tests/e2e_all_112/, see INDEX.md). Conclusion:

Strict-metric bucket Listening verdict
TIER A (44 langs, 0% bad_order) ✅ All OK — words land on speech
TIER B (20 langs, 1-10%) ✅ All OK
TIER C (23 langs, 10-25%) ✅ All OK
TIER D (14 langs, >25% strict bad_order) ⚠️ 3 sampled, much better than metric suggests (see below)

Important finding: the strict bad_order > 0 = FAIL metric is overly pessimistic for complex-script langs. Real perceived alignment quality is substantially higher than the raw timestamp-ordering count implies, because many "bad_order" cases are sub-frame inversions (e.g. word i end_time = word i+1 start_time − 1 ms) that are visually inaudible/invisible.

Per-language listening estimates on sampled TIER D langs:

Lang Strict bad_order rate Perceived correct alignment Verdict
te Telugu 42.3% 60-80% usable with post-fix
ka Georgian 41.2% ~90% basically OK, post-fix nice-to-have
lo Lao 46.7% 50-60% needs post-fix for production

The other 11 TIER D langs (am, as, ckb, cy, el, gu, hi, hy, ml, mr, my) were not individually spot-checked but expected to follow the same pattern: real-world quality is meaningfully better than the strict metric reports. The monotonicity post-fix snippet (see "Recommended post-processing" section) is sufficient for most production use cases on these langs.

Net judgement: model is production-ready for all 112 languages.

  • TIER A/B/C (87 langs): publish as-is.
  • TIER D (14 langs): apply monotonicity post-fix in pipeline — quality becomes near-A-tier in practice.

How to verify alignment with your own ears

Run an end-to-end karaoke test where each word lights up on screen at its alignment moment:

cd ..\NOESIS-Qwen3-ForcedAligner-0.6B-BF16
.\.conda\python.exe src\e2e_dub_karaoke.py --demo

Output: tests\e2e_demo\<lang>_<id>_karaoke.mp4 — open in any player (VLC, MPC, etc.). If words appear when spoken -> alignment correct. Audio is loudness-normalized to -16 LUFS for comfortable listening even from 16 kHz FLEURS sources. See docs/TESTING_GUIDE.md Section 4 for full details.

Performance (T4: throughput + VRAM, 100 sample utterances)

Metric Result Verdict
Successful inferences 100 / 100 PASS
Avg inference time 125 ms / utterance PASS
RTF (real-time factor) 0.010 (100x faster than realtime) PASS
Peak VRAM during inference 1956 MiB (33% of 6 GB card) PASS
Wall clock for 1199 s of audio 12.6 s PASS

1 hour of video aligned in ~36 seconds. Production-grade speed.

Functional smoke (T2: word-level alignment on all 87 refined langs)

  • 1 utterance per language from training JSONLs
  • Pass criterion (strict): bad_order == 0 AND bad_neg == 0
  • bad_order = words where end_time <= start_time (timestamp ordering violation)

Aggregate: 34/87 strict-PASS (39%), 53/87 WARN with 1+ bad_order

AAS regression (T1: Tier-1 30 langs)

  • 20/30 langs evaluated (10 ORIGINAL_11 SKIP -- no training JSONL by design)
  • Pass criterion same as T2
Verdict Langs n
PASS ar tr id uk ro sv cs fi 8
WARN hi bn vi th nl pl fa el he hu no da 12
SKIP (ORIGINAL_11) en zh ja ko es fr de pt ru it 10

Alignment quality on the 14 NEW langs (from smoke_unified_v6_112lang)

Verdict Langs n
PASS xh zu ti tk ug eu gn rw mvy qxp 10
WARN yo (2 bad / 22) wo (2/28) kmr (2/9) skr (1/7) 4

Per-language quality matrix (production guidance)

Bucketing: rate = bad_order / n_words on a single representative utterance. Real production performance on long-form dubbing is downstream and lives in the NOESIS dubbing pipeline -- see R-DUBBING-ISOCHRONY-50MS for end-task criterion.

TIER A -- PRODUCTION READY (34 langs, 0% bad_order on smoke) 

af  ar  ast az  be  bg  bs  ca  ceb cs  et  fi  gl  ha  hr  id
is  jv  kam kk  lb  lt  lv  mt  ny  oc  ro  sk  sl  sn  sv  tr
uk  umb

Plus 10 of 14 new langs at PASS: xh zu ti tk ug eu gn rw mvy qxp.

Use without hesitation. Word-level timestamps are monotonic and well-ordered on the sampled utterance.

TIER B -- GOOD, USE NORMALLY (20 langs, 1-10% bad_order rate) 

da  fa  ff  fil ga  hu  ig  kea ln  mi  mk  mn  nl  nso pl  so
sr  sw  uz  vi

Use in production. 1-2 minor ordering glitches per ~20-30 word utterance. For sub-frame lip-sync precision, a post-processing monotonicity enforcement pass (sort by start_time, clamp overlaps) is recommended.

TIER C -- USE WITH CAUTION (19 langs, 10-25% bad_order rate) 

bn  he  km  kn  ky  lg  luo ms  ne  no  om  or  pa  ps  sd  ta
tg  th  ur

Plus 4 of 14 new langs in WARN: yo wo kmr skr.

Production-ready ONLY with post-processing monotonicity fix. Raw output will have 2-7 ordering glitches per typical utterance. Most are minor and mergeable, but verify on representative samples before deploying for mission-critical subtitling.

TIER D -- complex scripts (14 langs, >25% strict bad_order) 

am  as  ckb cy  el  gu  hi  hy  ka  lo  ml  mr  my  te

Revised guidance after 2026-05-20 listening review: these languages have complex scripts where strict timestamp-inversion counting is inherently noisy (Indic abugidas, Greek, Welsh mutations, Armenian, Burmese, Lao, Georgian, Sorani-Kurdish). The strict metric overstates user-perceived defects.

Sampled real-world quality (founder listening, 2026-05-20):

  • ka Georgian: ~90% perceived correct -> production-ready
  • te Telugu: 60-80% perceived correct -> production with post-fix
  • lo Lao: 50-60% perceived correct -> production with post-fix

Recommended action for the full TIER D set:

  • apply the monotonicity post-fix below (single sort + clamp pass), and
  • the output is acceptable for word-level lip-sync on these langs;
  • only fall back to a script-specific CTC aligner (MMS for Indic etc.) if your end-task QA flags a specific lang as inadequate.
Lang Rate Script / Family
cy (Welsh) 61.9% Latin + consonant mutations
hy (Armenian) 56.2% Armenian alphabet
el (Greek) 50.0% Greek alphabet
lo (Lao) 46.7% Lao (no spaces)
te (Telugu) 42.3% Devanagari-family abugida
ka (Georgian) 41.2% Georgian Mkhedruli
ml (Malayalam) 35.7% Malayalam abugida
as (Assamese) 35.7% Bengali-Assamese
hi (Hindi) 35.4% Devanagari
ckb (Sorani Kurdish) 35.3% Arabic script (Sorani)
mr (Marathi) 34.8% Devanagari
gu (Gujarati) 33.3% Gujarati abugida
my (Burmese) 33.3% Burmese (no spaces)
am (Amharic) 27.3% Ethiopic Ge'ez

UNTESTED -- ORIGINAL_11 base_pretrained (11 langs) 

en zh es pt ru ja de fr ko it yue

These languages are served by the v3=98LANG base bundle weights without any NOESIS fine-tuning data (no JSONLs exist for them in our training manifest -- the upstream Qwen3-ForcedAligner already handles them well, per the 42.9 ms AAS Alibaba report).

Functional verification: model loads, forward pass succeeds (Test A PASS). Word-level quality: assumed equivalent to upstream Qwen3-ForcedAligner-0.6B (no per-word smoke run in this report).

For high-stakes use on these 11 langs, run an AAS regression with your own reference JSONLs:

.conda/python.exe src/07_eval_aas.py --langs en ru ja --reference path/to/ref.jsonl

What is NOT measured by this test suite

Metric Status Why
WER (Word Error Rate) N/A Qwen3-FA is a Forced Aligner, not ASR. It takes (audio, text) -> timestamps, not audio -> text. WER requires ASR (use Parakeet/Whisper for that).
Real lip-sync isochrony Not in this suite Lives in the NOESIS dubbing pipeline (R-DUBBING-ISOCHRONY-50MS, target ±50 ms on 95% of dialog lines). End-to-end testable only with full TTS + assembly pipeline.
Long-form audio (>5 min) Out of scope Architectural ceiling is 5 min per call (R-FORCED-ALIGNER-MAX-5MIN). Pipeline must window longer inputs.
Multi-speaker / overlapping Not tested This aligner takes mono input; speaker separation must happen upstream (HTDemucs/Sortformer).
Code-switching Not tested Each call takes one language argument; mixed-language clips need pre-segmentation.
Noisy / music background Not tested Speech enhancement / source separation must happen upstream.

Lineage of fine-tuning (NOESIS Phases A through Unified)

Phase New langs Method Date
A tr pl uk MFA-aligned + QLoRA 2026-05-08..10
B vi sv cs MFA-aligned + QLoRA 2026-05-10..11
C nl ro hu el da fi no sk NT-166 self-pseudo-labels + QLoRA 2026-05-11
C-B hi ar th id NT-166 self-pseudo-labels + QLoRA 2026-05-12
D ka kk NT-166 self-pseudo-labels + QLoRA 2026-05-12
F bn pa ur gu mr ta te kn ml ca is mt bg hr sr sl lv lt et be bs mk gl fa NT-166 self-pseudo-labels + QLoRA 2026-05-13..14
G af am as ast az ceb ckb cy ff fil ga ha he hy ig jv kam kea km ky lb lg ln lo luo mi mn ms my ne nso ny oc om or ps sd sn so sw tg umb uz NT-166 self-pseudo-labels + QLoRA 2026-05-14..15
Unified (v6) xh zu yo wo kmr skr ti tk ug mvy (NT-166 self-pseudo-CV/FLEURS) + eu gn rw qxp (NT-178 MMS-as-labeler) single unified run on v3 base, weighted loss, stratified sampler 2026-05-19..20

Forced-alignment quality (upstream baseline reference)

Average Alignment Shift (AAS, ms -- lower is better) on MFA-labeled raw audio:

Model Avg. AAS (ms) v
Monotonic-Aligner 161.1
NFA 129.8
WhisperX 133.2
Qwen3-ForcedAligner-0.6B (upstream) 42.9

5-min concatenated audio: upstream holds 52.9 ms AAS avg. Source: official Qwen3-ASR technical report (2026).

Hardware footprint

Property Value
Architecture Qwen3-Omni-derived (forced-alignment head)
Official model class Qwen3-ForcedAligner-0.6B
Actual stored parameters in this derived composite checkpoint 917,728,896 (~0.918B)
Hugging Face display 0.9B params (automatic rounded display)
Bundle size on disk 1.840 GB decimal (1.714 GiB); model.safetensors is 1.836 GB decimal
Inference mode NAR (non-autoregressive, single forward pass)
Maximum input length 5 minutes of speech per call
Audio type speech (no music / no BGM in input)
Languages 112 (101 NOESIS-trained + 11 upstream-base)
Output per-token (text, start_time, end_time)
Dtype bfloat16 (recommended at inference)
Peak VRAM at inference (measured) ~1.96 GB
Toolkit qwen-asr Python package

Quick start 

import torch
import numpy as np
import soundfile as sf
from qwen_asr import Qwen3ForcedAligner

bundle = "B:/.../models/asr/NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16"
aligner = Qwen3ForcedAligner.from_pretrained(
    bundle,
    dtype=torch.bfloat16,
    device_map={"": 0},          # STRICT -- never "auto"
)

# Method 1: from file path
results = aligner.align(
    audio="path/to/clip.wav",
    text="Hello world. This is an alignment test.",
    language="english",
)

# Method 2: from numpy array + sample rate
audio, sr = sf.read("clip.wav", dtype="float32")
results = aligner.align(
    audio=(audio, sr),                  # (ndarray, sr) tuple -- NOT torch.Tensor
    text="Hello world.",
    language="english",
)

for tok in results[0].items:
    print(f"{tok.text!r:>14}  {tok.start_time:.3f} - {tok.end_time:.3f}s")

Recommended post-processing for TIER C+D langs 

def enforce_monotonicity(items):
    """Sort by start_time, then clamp any overlap (end_time of i+1 must be > start_time of i)."""
    items = sorted(items, key=lambda x: x.start_time)
    for i in range(len(items) - 1):
        if items[i].end_time > items[i+1].start_time:
            items[i].end_time = items[i+1].start_time
        if items[i].end_time <= items[i].start_time:
            items[i].end_time = items[i].start_time + 0.01  # min 10ms duration
    return items

Sealed rules (NOESIS DHCF-FNO)

  • R-FORCED-ALIGNER-NAR -- use the NAR forward path; do NOT autoregress.
  • R-APACHE-PRESERVE -- redistributed under the same Apache 2.0 terms as upstream Qwen3-ForcedAligner-0.6B.
  • R-FORCED-ALIGNER-MAX-5MIN -- clip audio to <= 5 minutes per call.
  • R-DUBBING-FILM-SCOPE -- model is positioned for film-dubbing alignment & subtitling.
  • R-ISOLATED-LORA-AAS-GATE (v3.64) -- LoRA tested via AAS regression before merge.
  • R-UNIFIED-EXPANSION-FA-112-SINGLE-RUN (v3.69) -- sealed 2026-05-19, this bundle is the canonical realization.
  • R-CANONICAL-TRAINING-RECIPE-V1-PROVEN (NT-220) -- classical transformers.Trainer + pre-tokenize + pad_collate, NOT SFTTrainer.
  • R-TRAIN-CKPT-50-RESUMABLE -- save_steps=50, resumable via --resume.

Provenance

Step Source / output
Original weights Qwen/Qwen3-ForcedAligner-0.6B (Qwen team, Alibaba; Apache 2.0)
Foundation derived from Qwen3-Omni multimodal model
NOESIS Phases A-G +86 langs via MFA + cascading self-pseudo-labels (NT-166)
NOESIS Unified (v6) +14 langs (10 NT-166 self-pseudo on FLEURS/CV, 4 NT-178 MMS-as-labeler); 87 existing refined via stratified weighted loss
NOESIS additions this README, attribution in LICENSE, integration notes, full test suite
NOESIS version at integration v14.8 (2026-05-20)

Citation 

@article{Qwen3-ASR,
  title  = {Qwen3-ASR Technical Report},
  author = {Xian Shi and Xiong Wang and Zhifang Guo and Yongqi Wang and
            Pei Zhang and Xinyu Zhang and Zishan Guo and Hongkun Hao and
            Yu Xi and Baosong Yang and Jin Xu and Jingren Zhou and
            Junyang Lin},
  journal = {arXiv preprint arXiv:2601.21337},
  year    = {2026}
}

@misc{noesis2026qwen3aligner112,
  title  = {Qwen3-ForcedAligner-0.6B in NOESIS (112-language unified bundle):
            forced alignment for multilingual film-dubbing pipelines},
  author = {Bolotnikov, Ilia and AMAImedia},
  year   = {2026},
  note   = {Upstream Qwen3-ForcedAligner-0.6B (Apache 2.0), extended via
            NOESIS Phases A-G QLoRA fine-tunes (cascading self-pseudo-label
            expansion, NT-166/NT-178) and a final Unified-112 single-run
            training (R-UNIFIED-EXPANSION-FA-112-SINGLE-RUN, sealed 2026-05-19)
            to 101 trained + 11 base_pretrained = 112 total. Bundled into
            NOESIS DHCF-FNO multilingual dubbing pipeline as stage-15 timestamp
            generator. 0.6B params, RTF 0.010, 1.96 GB peak VRAM.},
  url    = {https://amaimedia.com}
}

License

Apache License 2.0. Original Qwen3-ForcedAligner-0.6B weights © Qwen team / Alibaba Cloud (2026). Redistribution, NOESIS Phases A-G + Unified fine-tune additions, integration notes, packaging metadata, test suite: © AMAImedia (NOESIS DHCF-FNO project) 2026. See LICENSE in this directory.

Downloads last month
-
Safetensors
Model size
0.9B params
Tensor type
BF16
·
Inference Providers NEW
Automatic Speech Recognition
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AMAImedia/NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16

Base model

Qwen/Qwen3-ForcedAligner-0.6B
Finetuned
(9)
this model

Collection including AMAImedia/NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16

Paper for AMAImedia/NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16