v45: SFT+chained GRPO with ITN — 95.9% number accuracy, 97.0% filler-free, deletion behavior matches v36

README.md

@@ -12,12 +12,13 @@ tags:
 - LiquidAI
 - mlx
 - mlx-5bit
+- inverse-text-normalization
 pipeline_tag: text-generation
 datasets:
 - juanquivilla/sotto-transcript-cleanup
 ---
 
-# SottoASR Transcript Cleanup — LFM2.5-350M MLX 5-bit (v36 + Preservation)
+# SottoASR Transcript Cleanup — LFM2.5-350M MLX 5-bit (v45 + Numbers)
 
 [sottoasr.app](https://sottoasr.app) · [Full precision (bf16)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) · [MLX 4-bit (smaller)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) · [Training Dataset](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)
 
@@ -25,17 +26,20 @@ datasets:
 
 **MLX 5-bit affine quantization** of [juanquivilla/sotto-cleanup-lfm25-350m](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m). The recommended variant for most Apple Silicon users — best size/quality trade-off.
 
-This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, and **— new in v36 — preserves substantive content reliably even on long inputs**, all locally with zero cloud dependency.
+This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, preserves substantive content reliably even on long inputs, and **— new in v45 — converts spoken-form numbers to digit form correctly** (inverse text normalization), all locally with zero cloud dependency.
 
-## What's new in v36
+## What's new in v45
 
-v36 fixes the **aggressive-edits failure mode** that earlier checkpoints occasionally exhibited: on long inputs the model would sometimes delete substantive content along with the fillers. v36 is a GRPO **full fine-tune** (all 354M params trainable, no LoRA) with a substantive-deletion-aware reward. Result: high-substantive-deletion incidence on long inputs drops from **3.85% → 0.64%** while filler-free rate climbs from **50.9% → 96.9%**.
+v45 adds **inverse text normalization (ITN)**: when users dictate compound spoken numbers like "talk about server three sixty," v45 reliably produces "Talk about server 360." Earlier versions (v36 and prior) either preserved the spoken form (looks unprofessional) or attempted the conversion incorrectly. v45 covers all common ITN categories — compound numbers, hundreds, four-digit years, times, decimals, percentages, currency, ordinals, dates — while continuing to preserve cardinals in idioms ("I'll be there in five" stays as written).
 
-| Capability | v23 baseline | **v36 (this model)** |
-|---|---|---|
-| Filler-Free rate | 50.9 % | **96.9 %** ⭐ |
-| Substantive-deletion >15% on long inputs | 3.85 % | **0.64 %** ⭐ |
-| ROUGE-L F1 on long inputs (>100 words) | 0.9242 | **0.9425** |
+| Capability | v36 (preservation) | **v45 (this model)** |
+|---|---:|---:|
+| Number accuracy (171-sample stratified set) | 12.9 % | **95.9 %** ⭐ |
+| Filler-Free rate | 96.9 % | **97.0 %** |
+| Substantive-deletion >15% on long inputs† | 13.3 % | 13.7 % (~tied) |
+| Word retention median | 0.884 | 0.922 |
+
+† Measured on all 241 long inputs (>100 words) from `data_v23_paragraphs/val.jsonl` — a stricter metric than v36's published 0.64 % (which was on a 350-sample mix). v45 inherits v36's deletion-aware behavior on the same eval.
 
 ## Key Specs
 
@@ -47,7 +51,7 @@ v36 fixes the **aggressive-edits failure mode** that earlier checkpoints occasio
 | **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
 | **Latency** | ~85 ms average per transcript (M-series) |
 
-Quality at this quantization tracks the bf16 model closely. See the [bf16 model card](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) for full benchmark numbers.
+Quality at this quantization tracks the bf16 model closely. See the [bf16 model card](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) for full benchmark numbers, training pipeline, and reward shape.
 
 ## Quantization Recipe
 
@@ -70,14 +74,14 @@ from mlx_lm.sample_utils import make_sampler
 model, tokenizer = load("juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit")
 sampler = make_sampler(temp=0.0)  # greedy
 
-text = "so uh basically we need to fix the deployment pipeline"
+text = "talk about server three sixty"
 prompt = f"### Input:\n{text}\n\n### Output:\n"
 
 output = generate(model, tokenizer, prompt=prompt, max_tokens=512, sampler=sampler)
 if "###" in output:
     output = output[:output.index("###")].strip()
 print(output)
-# → "We need to fix the deployment pipeline."
+# → "Talk about server 360."
 ```
 
 For long dictation that may need paragraph formatting, raise `max_tokens` to 1024–2048.
@@ -87,10 +91,12 @@ For long dictation that may need paragraph formatting, raise `max_tokens` to 102
 | Input (raw ASR) | Output (cleaned) |
 |-----------------|------------------|
 | so uh basically we need to fix the deployment pipeline | We need to fix the deployment pipeline. |
-| the deadline is friday no monday we have until monday | The deadline is Monday. |
-| what we what i wanted to say is the tests pass | What I wanted to say is the tests pass. |
-| okay so the thing is basically we're running out of disk space | We're running out of disk space. |
-| uh yes | Yes. |
+| talk about server three sixty | Talk about server 360. |
+| schedule it for three fifteen pm | Schedule it for 3:15 PM. |
+| we hit ninety eight percent uptime last month | We hit 98 % uptime last month. |
+| transfer fifty dollars to billing | Transfer $50 to billing. |
+| i'll be there in five | I'll be there in five. |
+| we run twenty four seven | We run 24/7. |
 
 ### Paragraph emission on long dictations (inherited from v23)
 

config.json

@@ -21,7 +21,6 @@
     "eos_token_id": [
         7
     ],
-    "full_attn_idxs": null,
     "hidden_size": 1024,
     "initializer_range": 0.02,
     "intermediate_size": 6656,
@@ -67,7 +66,7 @@
     },
     "rope_theta": 1000000.0,
     "tie_word_embeddings": true,
-    "transformers_version": "5.6.2",
+    "transformers_version": "5.3.0",
     "use_cache": false,
     "use_pos_enc": true,
     "vocab_size": 65536

generation_config.json

@@ -5,5 +5,5 @@
     7
   ],
   "pad_token_id": 0,
-  "transformers_version": "5.6.2"
+  "transformers_version": "5.3.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8fdff7017e6929cb5d3bb90e3136da4be6bfa0897109286532486dc92d57ab33
+oid sha256:07ae4ba3889eaf4e3d959b8f9f8ad5ac6f09a03a5bb9b2a20ec3165beaf91803
 size 243830312