v45: SFT+chained GRPO with ITN — 95.9% number accuracy, 97.0% filler-free, deletion behavior matches v36

README.md

@@ -12,13 +12,13 @@ tags:
 - LiquidAI
 - grpo
 - full-fine-tune
-- preservation
+- inverse-text-normalization
 pipeline_tag: text-generation
 datasets:
 - juanquivilla/sotto-transcript-cleanup
 ---
 
-# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v36 + Preservation)
+# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v45 + Numbers)
 
 [sottoasr.app](https://sottoasr.app) · [MLX 5-bit (recommended)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) · [MLX 4-bit (smaller)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) · [Training Dataset](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)
 
@@ -26,39 +26,61 @@ datasets:
 
 **Full-precision bf16** fine-tune of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base) for on-device speech-to-text transcript cleanup. This is the **training artifact** — for on-device deployment on Apple Silicon, use the [5-bit MLX variant](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) instead.
 
-This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, and **— new in v36 — preserves substantive content reliably even on long inputs**, all locally with zero cloud dependency.
+This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, preserves substantive content reliably even on long inputs, and **— new in v45 — converts spoken-form numbers to digit form correctly** (inverse text normalization), all locally with zero cloud dependency.
 
-## What's new in v36
+## What's new in v45
 
-v36 fixes the **aggressive-edits failure mode** that earlier checkpoints occasionally exhibited: on long inputs the model would sometimes delete substantive content along with the fillers. v36 is a GRPO **full fine-tune** (all 354M params trainable, no LoRA) with a substantive-deletion-aware reward that penalizes any deletion of non-filler words beyond a 10% threshold. Result: high-substantive-deletion incidence on long inputs drops from **3.85% → 0.64%** while filler-free rate climbs from **50.9% → 96.9%**.
+v45 fixes the **inverse text normalization (ITN) failure mode** that v36 exhibited: when users dictated compound spoken numbers like "talk about server three sixty," v36 would either preserve the spoken form (looks unprofessional) or attempt the conversion incorrectly ("server 315" instead of "server 360"). v45 reliably produces the correct digit form across all common ITN categories — compound numbers, hundreds, four-digit years, times, decimals, percentages, currency, ordinals, dates — while continuing to preserve cardinals in idioms ("I'll be there in five" stays as written; "twenty four seven" → "24/7").
 
-| Capability | v23 baseline | **v36 (this model)** |
-|---|---|---|
-| Filler-Free rate | 50.9 % | **96.9 %** ⭐ |
-| Substantive-deletion >15% on long inputs | 3.85 % | **0.64 %** ⭐ |
-| ROUGE-L F1 on long inputs (>100 words) | 0.9242 | **0.9425** |
-| Set-based deletion >40% (legacy metric) | 11.4 % | **3.4 %** |
-| Word retention median | 0.899 | 0.884 |
-| Composite winner score | 131.7 | **189.2** |
+### Two-stage pipeline
 
-The substantive-deletion metric is the one that maps most directly to the user-reported failure mode: it counts how often the model removes more than 15% of the *non-filler* words from a long input. v36 is roughly **6× rarer to do this than the v23 baseline**.
+1. **SFT (v39)** — supervised fine-tuning of v36 on a 19K-row mix: 12,757 number-conversion examples (programmatically generated with ground-truth digit forms, then augmented 4× via Qwen3.6-27B for diversity) + 5K random flat samples + 1K preservation samples. 2 epochs, full FT, lr 2e-5.
+2. **GRPO (v40 → v42 → v43 → v44 → v45)** — five GRPO refinement passes that progressively tightened the reward shaping. The final v45 reward combines v36's substantive-deletion penalty + a number-accuracy term (digit-signature exact match, ±5.0) + an over-conversion penalty (penalize digits when reference has none) + an amplified filler penalty (6× multiplier). Each pass trained on a 6K-row mix (3K numbers + 2K flat + 1K preservation), batch=48, num_gen=8, 1000 steps, lr=5e-6 cosine.
 
-v36 is the breakthrough run from a 28-run autonomous GRPO campaign. All 14 LoRA r=32 variants we tried converged to a ~7–8% sub-del-15-long floor regardless of reward shape, oversampling ratio, or warm-start checkpoint. The full fine-tune cleared the floor by a 12× margin — the conclusion: with 96 GB of VRAM available, full fine-tuning of a 354M model is the right tool for shaping fine-grained word-level behavior.
+### Headline results
+
+| Capability | v23 baseline | v36 (preservation) | **v45 (this model)** |
+|---|---:|---:|---:|
+| Number accuracy (171-sample stratified set) | — | 12.9 % | **95.9 %** ⭐ |
+| Filler-Free rate | 50.9 % | 96.9 % | **97.0 %** |
+| Substantive-deletion >15% on long inputs | 3.85 % | 0.64 %† | 13.7 %‡ |
+| Word retention median | 0.899 | 0.884 | 0.922 |
+
+† Measured on the v36-era mix of 200 v23 random + 150 preservation samples.
+‡ Measured on a stricter eval — all 241 long inputs (>100 words) from `data_v23_paragraphs/val.jsonl`. On that same stricter eval v36 measures 13.3 %, so v45 is essentially tied with v36 on deletion behavior while gaining 83pp on number accuracy.
+
+### Number conversion accuracy by category (171 stratified val samples)
+
+| Category | v36 | **v45** |
+|---|---:|---:|
+| compound_2digit ("three sixty" → "360") | 10.0 % | **96.7 %** |
+| sound_alike (sixteen vs sixty) | 0.0 % | **100.0 %** |
+| hundred_form ("two hundred fifty" → "250") | 5.0 % | **100.0 %** |
+| four_digit ("nineteen ninety nine" → "1999") | 6.7 % | **100.0 %** |
+| time ("three pm" → "3 PM") | 0.0 % | **100.0 %** |
+| decimal ("three point one four" → "3.14") | 10.0 % | 90.0 % |
+| percent ("sixty percent" → "60 %") | 0.0 % | **100.0 %** |
+| currency ("fifty dollars" → "$50") | 0.0 % | **100.0 %** |
+| negative ("minus five" → "-5") | 0.0 % | **100.0 %** |
+| digit_by_digit (phone) | 0.0 % | 60.0 % |
+| ordinal ("twenty third" → "23rd") | 0.0 % | **100.0 %** |
+| date ("july fourth" → "July 4") | 16.7 % | **100.0 %** |
+| **preserve** (idioms, small cardinals) | 100.0 % | **100.0 %** |
+
+12 of 13 categories at ≥90 %; preserve cases held at 100 % — v45 does not over-convert idiomatic uses of small cardinals.
 
 ## Key Specs
 
 | Property | Value |
 |----------|-------|
 | **Size** | **676 MB** |
-| **ROUGE-L F1 (long inputs)** | **0.9425** |
-| **Filler-Free rate** | **96.9 %** ⭐ |
-| **Substantive-deletion >15% (long)** | **0.64 %** ⭐ |
+| **Number accuracy (stratified val)** | **95.9 %** ⭐ |
+| **Filler-Free rate** | **97.0 %** ⭐ |
 | **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
 | **Precision** | bf16 |
-| **Training method** | Full fine-tune GRPO (no LoRA) |
-| **Training context** | 2,048 prompt + 1,024 completion |
+| **Training method** | SFT then chained GRPO (full fine-tune, no LoRA) |
 | **Native context** | 32,768 tokens (128K base) |
-| **Hardware** | 1× RTX PRO 6000 Blackwell (96 GB), ~46 min for 1000 GRPO steps |
+| **Hardware** | 1× RTX PRO 6000 Blackwell (96 GB) |
 
 ## What It Does
 
@@ -68,40 +90,16 @@ Takes raw, unpunctuated ASR output and produces clean, readable text:
 |-----------------|------------------|
 | so uh basically we need to fix the deployment pipeline | We need to fix the deployment pipeline. |
 | the deadline is friday no monday we have until monday | The deadline is Monday. |
-| what we what i wanted to say is the tests pass | What I wanted to say is the tests pass. |
-| okay so the thing is basically we're running out of disk space | We're running out of disk space. |
-| uh yes | Yes. |
+| talk about server three sixty | Talk about server 360. |
+| schedule it for three fifteen pm | Schedule it for 3:15 PM. |
+| we hit ninety eight percent uptime last month | We hit 98 % uptime last month. |
+| transfer fifty dollars to billing | Transfer $50 to billing. |
+| i'll be there in five | I'll be there in five. |
+| we run twenty four seven | We run 24/7. |
 
 ### Paragraph emission on long dictations (inherited from v23)
 
-Long, multi-topic input is restructured into paragraph-formatted prose with `\n\n` breaks at natural topic boundaries:
-
-**Input:**
-> okay so were having some real issues with the deployment pipeline and i want to walk through whats going wrong the main problem is that the redis cache is timing out during deploys we push a new version and then for about thirty seconds the connections hang and customers see errors so i think we need to add a graceful shutdown period before we kill the old pods now separately the elasticsearch cluster has been a pain we deployed a new svelte frontend last month and it caused index corruption we need to validate the schema before pushing anything live going forward i think the answer here is to add both of these checks to our standard deployment checklist
-
-**Output:**
-> We're having some real issues with the deployment pipeline, and I want to walk through what's going wrong. The main problem is that the Redis cache is timing out during deploys. We push a new version and then for about thirty seconds the connections hang, and customers see errors.
->
-> I think we need to add a graceful shutdown period before we kill the old pods. Now separately, the Elasticsearch cluster has been a pain. We deployed a new Svelte frontend last month and it caused index corruption. We need to validate the schema before pushing anything live. Going forward, I think the answer here is to add both of these checks to our standard deployment checklist.
-
-## Benchmark Results
-
-Evaluated on a 350-sample held-out mix: 200 random samples from `data_v23_paragraphs/val.jsonl` plus 150 held-out preservation samples (the last 150 of `preservation_samples.jsonl`, never seen during training).
-
-### Headline numbers vs baselines
-
-| Metric | v23 baseline | v24 (Apr) | **v36 (this model)** |
-|--------|---:|---:|---:|
-| ROUGE-L F1 (overall) | 0.9357 | 0.9300 | 0.9076 |
-| ROUGE-L F1 (long inputs >100 words) | 0.9242 | 0.9144 | **0.9425** |
-| Filler-Free rate | 50.9 % | 74.9 % | **96.9 %** ⭐ |
-| Set-based deletion >40% | 11.4 % | 12.9 % | **3.4 %** |
-| **Substantive-deletion >15% on long inputs** | **3.85 %** | — | **0.64 %** ⭐ |
-| Substantive-deletion >25% on long inputs | 0.0 % | — | 0.0 % |
-| Set-based deletion median | 0.107 | 0.175 | 0.092 |
-| Word retention median | 0.899 | 0.818 | 0.884 |
-
-v36 is **slightly more aggressive on short inputs** (overall ROUGE-L drops from 0.936 → 0.908 — the model more readily strips fillers and tightens punctuation) but **better on long inputs** (ROUGE-L 0.924 → 0.943) where preservation matters most. For workloads dominated by long-form dictation (the design target), v36 is strictly better than v23.
+Long, multi-topic input is restructured into paragraph-formatted prose with `\n\n` breaks at natural topic boundaries. See the [v36 model card archive](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m/tree/main) for a worked example.
 
 ## Usage
 
@@ -127,7 +125,7 @@ model = AutoModelForCausalLM.from_pretrained(
 )
 tokenizer = AutoTokenizer.from_pretrained("juanquivilla/sotto-cleanup-lfm25-350m")
 
-text = "so uh basically we need to fix the deployment pipeline"
+text = "talk about server three sixty"
 prompt = f"### Input:\n{text}\n\n### Output:\n"
 
 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
@@ -137,7 +135,7 @@ output = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_to
 if "###" in output:
     output = output[:output.index("###")]
 print(output.strip())
-# → "We need to fix the deployment pipeline."
+# → "Talk about server 360."
 ```
 
 For long dictation that may need paragraph formatting, use a higher `max_new_tokens` (1024–2048).
@@ -148,36 +146,38 @@ For long dictation that may need paragraph formatting, use a higher `max_new_tok
 
 ```
 LiquidAI/LFM2.5-350M-Base
-  → SFT (v23): 157,556 rows (v22 base + 4,012 paragraph_formatting),
-                LR 3e-5, β2=0.95, 3 epochs, bf16+tf32, packed 4,096 ctx
-  → GRPO R5/R6 (v23): LoRA r=32, paragraph emission tuned
-                      → main val ROUGE-L 0.9499, Filler-Free 91.0 %, paragraph rate 89 %
-  → GRPO v36 (this model): FULL fine-tune (no LoRA), all 354M params,
-                            substantive-deletion-aware reward, 6,000 examples
-                            (3,000 flat + 3,000 oversampled preservation, 50/50),
-                            batch=48, num_gen=8, 1000 steps, LR 5e-6, ~46 min
-                            → 350-sample eval: 96.9 % filler-free, 0.64 % sub-del-15-long
+  → SFT (v23): 157,556 rows, LR 3e-5, 3 epochs, packed 4,096 ctx
+  → GRPO R5/R6 (v23): paragraph emission tuned
+  → GRPO v36: full FT, substantive-deletion-aware reward (no LoRA)
+  → SFT v39: v36 + 12,757 number examples + 5K flat + 1K preservation, 2 epochs
+  → GRPO v40-v45: chained refinement passes with combined reward
+                  (ROUGE×5 - filler×6 + format + sub-del-pen + pres + num_term ±5)
 ```
 
-### v36 reward function
+### v45 reward function
 
-The v36 reward is a sum of:
+A sum of:
 
 1. **Quality**: `+5.0 × ROUGE-L F1` against the reference clean output.
-2. **Filler penalty**: `−min(count × 0.5, 2.0) × 3.0` per generation.
+2. **Filler penalty (amplified)**: `−min(count × 0.5, 2.0) × 6.0` per generation. v45 doubles v36's 3.0 multiplier — the boost that pushed filler-free past 95 %.
 3. **Format bonus**: +0.2 each for capital-leading and terminal punctuation.
-4. **Substantive-deletion penalty**: strip filler words from input, compute multiplicity-aware (multiset) deletion ratio of substantive words. If `> 0.10` (threshold), apply `−10.0 × ramp` linearly scaling from 0 at 0.10 to 1.0 at 0.35 deletion. Clipped at `−10.0`.
+4. **Substantive-deletion penalty** (inherited from v36): strip filler words from input, compute multiset deletion ratio. If `> 0.10`, apply `−10.0 × ramp` linearly scaling from 0 at 0.10 to 1.0 at 0.35 deletion.
 5. **Preservation bonus**: `+1.5` when output/input word ratio is in `[0.70, 0.95]`.
+6. **Number-accuracy term (new in v40, fixed in v42)**: extract digit signatures from output and reference (after stripping commas/hyphens between digits). If they match, `+5.0`. If reference has digits but output disagrees, `−5.0`. **If reference has no digits but output adds them, `−5.0`** (v42's fix that solved v40's preserve-case over-conversion).
+
+### Number-data generation
+
+Programmatic generator covering 11 categories with **guaranteed ground truth** (digit form generated first, spoken form derived). Bedrock Claude Haiku 4.5 was the original augmentation target; replaced with a local Qwen3.6-27B-FP8 SGLang container (`enable_thinking: false` to disable reasoning mode). 3,229 programmatic seeds × 4 variants → 12,757 augmented examples after digit-signature acceptance filtering (98.4 % accept rate).
 
-Hyperparameters: `DEL_CAP=−10.0`, `DEL_THRESH=0.10`, `DEL_RAMP_END=0.35`, `PRES_BONUS=+1.5`, `PRES_BAND=(0.70, 0.95)`.
+### Why chained GRPO
 
-### Why full fine-tune
+v40 (single-pass GRPO from v39 with the basic combined reward) hit 92.4 % num + 93.8 % filler-free — strong but with a regression on `preserve` and `time` categories (model over-converted idioms). v42 fixed the over-conversion penalty (recovering preserve to 100 %), v43 amplified the number term to 5/-5 (regaining num to 95.3 %), v44 chained one more epoch (recovering `time` to 100 %), and v45 finally amplified the filler penalty (97.0 % filler-free + 95.9 % num — final).
 
-The campaign ran 14 LoRA variants (v25–v38, v50, v52) and one full fine-tune (v36) of the same substantive-deletion-aware reward. **All LoRA r=32 runs converged to ~7–8% sub-del-15-long** regardless of reward shape, oversampling ratio, threshold, cap, or warm-start checkpoint. The full FT run (v36) **broke through to 0.64%** — a 12× improvement. With 96 GB of VRAM available, full fine-tuning a 354M model is the right tool for shaping fine-grained word-level behavior; LoRA's low-rank constraint was the limiting factor.
+The chained approach proved more controllable than reaching the target in one shot: each pass corrected one specific failure mode without disturbing the others.
 
 ### Dataset
 
-**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup), plus a **1,352-sample preservation set** (long inputs specifically constructed to stress preservation under heavy filler removal). v36 GRPO trains on a 6,000-example mix: 3,000 flat from `train.jsonl` + 3,000 from the 1,352-sample preservation set (oversampled ~2.2× for a 50/50 balance).
+**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup), plus the **1,352-sample preservation set** and a **3,400-sample programmatic ITN set** (3,229 train + 171 stratified val) augmented to 12,757 by Qwen3.6-27B with digit-signature validation.
 
 ## All Variants
 
@@ -191,7 +191,8 @@ The campaign ran 14 LoRA variants (v25–v38, v50, v52) and one full fine-tune (
 
 - Optimized for **English** conversational/meeting-style speech
 - Domain-specific jargon (medical, legal) may not be corrected without additional fine-tuning
-- v36 is slightly more aggressive on short inputs than v23 (overall ROUGE-L 0.908 vs 0.936); for workloads dominated by short utterances where lexical fidelity matters more than filler removal, v23 may be a better fit
+- `digit_by_digit` (phone numbers, account numbers spoken as individual digits with hyphens) is the weakest ITN category at ~60 % — produces correct digits but sometimes formats them differently from the reference
+- Decimal conversion accuracy is 90 % (1/10 stratified val miss)
 - The substantive-deletion metric is reference-free (compares output to input only, after filler-stripping) so it cannot catch *semantic* hallucinations — only word-level losses
 - Not designed for formal written text — trained on spoken language patterns
 

config.json

@@ -18,7 +18,6 @@
   "conv_use_xavier_init": true,
   "dtype": "bfloat16",
   "eos_token_id": 7,
-  "full_attn_idxs": null,
   "hidden_size": 1024,
   "initializer_range": 0.02,
   "intermediate_size": 6656,
@@ -53,7 +52,7 @@
     "rope_type": "default"
   },
   "tie_word_embeddings": true,
-  "transformers_version": "5.6.2",
+  "transformers_version": "5.3.0",
   "use_cache": false,
   "use_pos_enc": true,
   "vocab_size": 65536,

generation_config.json

@@ -5,5 +5,5 @@
     7
   ],
   "pad_token_id": 0,
-  "transformers_version": "5.6.2"
+  "transformers_version": "5.3.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:06f28fd7204bc24d3e7f42cec46beeb18cd1db8e9041d394efa33b7d18f025ad
+oid sha256:b157b5499dc3ff418e449e669454fae67dd248152bfc13d008ff56abffec432f
 size 708984464