v51: composite=88.68 — see model card for benchmark deltas vs v45

README.md

@@ -18,102 +18,54 @@ datasets:
 - juanquivilla/sotto-transcript-cleanup
 ---
 
-# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v45 + Numbers)
+# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v51)
 
 [sottoasr.app](https://sottoasr.app) · [MLX 5-bit (recommended)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) · [MLX 4-bit (smaller)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) · [Training Dataset](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)
 
 ## Overview
 
-**Full-precision bf16** fine-tune of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base) for on-device speech-to-text transcript cleanup. This is the **training artifact** — for on-device deployment on Apple Silicon, use the [5-bit MLX variant](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) instead.
+Full-precision bf16 fine-tune of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base) for on-device speech-to-text transcript cleanup. This is the **training artifact** — for on-device deployment on Apple Silicon, use the [5-bit MLX variant](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit).
 
-This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, preserves substantive content reliably even on long inputs, and **— new in v45 — converts spoken-form numbers to digit form correctly** (inverse text normalization), all locally with zero cloud dependency.
+## What's new in v51
 
-## What's new in v45
+v51 extends v45 with targeted training data covering five failure modes observed on real production transcripts:
 
-v45 fixes the **inverse text normalization (ITN) failure mode** that v36 exhibited: when users dictated compound spoken numbers like "talk about server three sixty," v36 would either preserve the spoken form (looks unprofessional) or attempt the conversion incorrectly ("server 315" instead of "server 360"). v45 reliably produces the correct digit form across all common ITN categories — compound numbers, hundreds, four-digit years, times, decimals, percentages, currency, ordinals, dates — while continuing to preserve cardinals in idioms ("I'll be there in five" stays as written; "twenty four seven" → "24/7").
+- **Multi-number sentences** ("the score was 21 to 14")
+- **Year-with-context drift** ("twenty eighteen and twenty twenty seven" → preserve both years)
+- **Disconnected number lists** (no phone-number hallucination on "one two three eleven forty six")
+- **Within-input duplicates** (same number stated twice — preserve consistency)
+- **Long-form preservation** (multi-paragraph dictations with embedded numbers)
 
-### Two-stage pipeline
+The v37+v47 training data (~30K examples) was re-audited end-to-end with a Qwen3.6-27B
+judge and bad rows were dropped. A new 66-case adversarial benchmark drives objective
+evaluation.
 
-1. **SFT (v39)** — supervised fine-tuning of v36 on a 19K-row mix: 12,757 number-conversion examples (programmatically generated with ground-truth digit forms, then augmented 4× via Qwen3.6-27B for diversity) + 5K random flat samples + 1K preservation samples. 2 epochs, full FT, lr 2e-5.
-2. **GRPO (v40 → v42 → v43 → v44 → v45)** — five GRPO refinement passes that progressively tightened the reward shaping. The final v45 reward combines v36's substantive-deletion penalty + a number-accuracy term (digit-signature exact match, ±5.0) + an over-conversion penalty (penalize digits when reference has none) + an amplified filler penalty (6× multiplier). Each pass trained on a 6K-row mix (3K numbers + 2K flat + 1K preservation), batch=48, num_gen=8, 1000 steps, lr=5e-6 cosine.
+## Headline numbers
 
-### Headline results
-
-| Capability | v23 baseline | v36 (preservation) | **v45 (this model)** |
+| Capability | v36 | v45 | **v51 (this model)** |
 |---|---:|---:|---:|
-| Number accuracy (171-sample stratified set) | — | 12.9 % | **95.9 %** ⭐ |
-| Filler-Free rate | 50.9 % | 96.9 % | **97.0 %** |
-| Substantive-deletion >15% on long inputs | 3.85 % | 0.64 %† | 13.7 %‡ |
-| Word retention median | 0.899 | 0.884 | 0.922 |
-
-† Measured on the v36-era mix of 200 v23 random + 150 preservation samples.
-‡ Measured on a stricter eval — all 241 long inputs (>100 words) from `data_v23_paragraphs/val.jsonl`. On that same stricter eval v36 measures 13.3 %, so v45 is essentially tied with v36 on deletion behavior while gaining 83pp on number accuracy.
-
-### Number conversion accuracy by category (171 stratified val samples)
-
-| Category | v36 | **v45** |
-|---|---:|---:|
-| compound_2digit ("three sixty" → "360") | 10.0 % | **96.7 %** |
-| sound_alike (sixteen vs sixty) | 0.0 % | **100.0 %** |
-| hundred_form ("two hundred fifty" → "250") | 5.0 % | **100.0 %** |
-| four_digit ("nineteen ninety nine" → "1999") | 6.7 % | **100.0 %** |
-| time ("three pm" → "3 PM") | 0.0 % | **100.0 %** |
-| decimal ("three point one four" → "3.14") | 10.0 % | 90.0 % |
-| percent ("sixty percent" → "60 %") | 0.0 % | **100.0 %** |
-| currency ("fifty dollars" → "$50") | 0.0 % | **100.0 %** |
-| negative ("minus five" → "-5") | 0.0 % | **100.0 %** |
-| digit_by_digit (phone) | 0.0 % | 60.0 % |
-| ordinal ("twenty third" → "23rd") | 0.0 % | **100.0 %** |
-| date ("july fourth" → "July 4") | 16.7 % | **100.0 %** |
-| **preserve** (idioms, small cardinals) | 100.0 % | **100.0 %** |
-
-12 of 13 categories at ≥90 %; preserve cases held at 100 % — v45 does not over-convert idiomatic uses of small cardinals.
-
-## Key Specs
-
-| Property | Value |
-|----------|-------|
-| **Size** | **676 MB** |
-| **Number accuracy (stratified val)** | **95.9 %** ⭐ |
-| **Filler-Free rate** | **97.0 %** ⭐ |
-| **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
-| **Precision** | bf16 |
-| **Training method** | SFT then chained GRPO (full fine-tune, no LoRA) |
-| **Native context** | 32,768 tokens (128K base) |
-| **Hardware** | 1× RTX PRO 6000 Blackwell (96 GB) |
-
-## What It Does
-
-Takes raw, unpunctuated ASR output and produces clean, readable text:
-
-| Input (raw ASR) | Output (cleaned) |
-|-----------------|------------------|
-| so uh basically we need to fix the deployment pipeline | We need to fix the deployment pipeline. |
-| the deadline is friday no monday we have until monday | The deadline is Monday. |
-| talk about server three sixty | Talk about server 360. |
-| schedule it for three fifteen pm | Schedule it for 3:15 PM. |
-| we hit ninety eight percent uptime last month | We hit 98 % uptime last month. |
-| transfer fifty dollars to billing | Transfer $50 to billing. |
-| i'll be there in five | I'll be there in five. |
-| we run twenty four seven | We run 24/7. |
-
-### Paragraph emission on long dictations (inherited from v23)
-
-Long, multi-topic input is restructured into paragraph-formatted prose with `\n\n` breaks at natural topic boundaries. See the [v36 model card archive](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m/tree/main) for a worked example.
+| Number accuracy (171-sample stratified val) | 12.9% | 95.9% | **95.3%** |
+| 66-case adversarial benchmark (greedy) | n/a | 76% | **86%** |
+| 66-case adversarial benchmark (temp 0.7 × 4) | n/a | 77% | **84%** |
+| Loops on 264 sampling-mode probes | n/a | 0 | **0** |
+| Filler-free on 241 long inputs | 67.2% | 68.0% | **73.0%** |
+| Sub-deletion >15% on 241 long inputs | 13.3% | 13.7% | **11.2%** |
 
-## Usage
+Composite score (0.35×num + 0.30×bench_greedy + 0.15×bench_sample + 0.10×filler_long + 0.05×(1-sub15) + 0.05×(1-loops/N)): **88.68**
 
-### Prompt Format
+## Training pipeline
 
 ```
-### Input:
-{raw transcript}
-
-### Output:
-{model generates cleaned text}
+LiquidAI/LFM2.5-350M-Base
+  → SFT v23 → GRPO v23 (paragraph emission)
+  → GRPO v36: full FT with substantive-deletion-aware reward
+  → SFT v39: + 12.7K augmented number examples (ITN)
+  → GRPO v40-v45: chained refinement, fixed reward + amplified filler penalty
+  → SFT/GRPO v51: + 5.5K gap-filling examples (multi-number, year-context,
+                       disconnected, duplicates, long-form), Qwen-audited and cleaned
 ```
 
-### Python Example
+## Usage
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -135,73 +87,16 @@ output = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_to
 if "###" in output:
     output = output[:output.index("###")]
 print(output.strip())
-# → "Talk about server 360."
 ```
 
-For long dictation that may need paragraph formatting, use a higher `max_new_tokens` (1024–2048).
-
-## Training Details
-
-### Pipeline
-
-```
-LiquidAI/LFM2.5-350M-Base
-  → SFT (v23): 157,556 rows, LR 3e-5, 3 epochs, packed 4,096 ctx
-  → GRPO R5/R6 (v23): paragraph emission tuned
-  → GRPO v36: full FT, substantive-deletion-aware reward (no LoRA)
-  → SFT v39: v36 + 12,757 number examples + 5K flat + 1K preservation, 2 epochs
-  → GRPO v40-v45: chained refinement passes with combined reward
-                  (ROUGE×5 - filler×6 + format + sub-del-pen + pres + num_term ±5)
-```
-
-### v45 reward function
-
-A sum of:
-
-1. **Quality**: `+5.0 × ROUGE-L F1` against the reference clean output.
-2. **Filler penalty (amplified)**: `−min(count × 0.5, 2.0) × 6.0` per generation. v45 doubles v36's 3.0 multiplier — the boost that pushed filler-free past 95 %.
-3. **Format bonus**: +0.2 each for capital-leading and terminal punctuation.
-4. **Substantive-deletion penalty** (inherited from v36): strip filler words from input, compute multiset deletion ratio. If `> 0.10`, apply `−10.0 × ramp` linearly scaling from 0 at 0.10 to 1.0 at 0.35 deletion.
-5. **Preservation bonus**: `+1.5` when output/input word ratio is in `[0.70, 0.95]`.
-6. **Number-accuracy term (new in v40, fixed in v42)**: extract digit signatures from output and reference (after stripping commas/hyphens between digits). If they match, `+5.0`. If reference has digits but output disagrees, `−5.0`. **If reference has no digits but output adds them, `−5.0`** (v42's fix that solved v40's preserve-case over-conversion).
-
-### Number-data generation
-
-Programmatic generator covering 11 categories with **guaranteed ground truth** (digit form generated first, spoken form derived). Bedrock Claude Haiku 4.5 was the original augmentation target; replaced with a local Qwen3.6-27B-FP8 SGLang container (`enable_thinking: false` to disable reasoning mode). 3,229 programmatic seeds × 4 variants → 12,757 augmented examples after digit-signature acceptance filtering (98.4 % accept rate).
-
-### Why chained GRPO
-
-v40 (single-pass GRPO from v39 with the basic combined reward) hit 92.4 % num + 93.8 % filler-free — strong but with a regression on `preserve` and `time` categories (model over-converted idioms). v42 fixed the over-conversion penalty (recovering preserve to 100 %), v43 amplified the number term to 5/-5 (regaining num to 95.3 %), v44 chained one more epoch (recovering `time` to 100 %), and v45 finally amplified the filler penalty (97.0 % filler-free + 95.9 % num — final).
-
-The chained approach proved more controllable than reaching the target in one shot: each pass corrected one specific failure mode without disturbing the others.
-
-### Dataset
-
-**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup), plus the **1,352-sample preservation set** and a **3,400-sample programmatic ITN set** (3,229 train + 171 stratified val) augmented to 12,757 by Qwen3.6-27B with digit-signature validation.
-
 ## All Variants
 
 | Variant | Size | Use Case |
 |---------|------|----------|
 | **[Full precision (this)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m)** | 676 MB | Training, GPU inference |
 | **[MLX 5-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit)** | ~237 MB | **Recommended for Apple Silicon** |
-| [MLX 4-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) | ~195 MB | Smallest, slight quality trade-off |
-
-## Limitations
-
-- Optimized for **English** conversational/meeting-style speech
-- Domain-specific jargon (medical, legal) may not be corrected without additional fine-tuning
-- `digit_by_digit` (phone numbers, account numbers spoken as individual digits with hyphens) is the weakest ITN category at ~60 % — produces correct digits but sometimes formats them differently from the reference
-- Decimal conversion accuracy is 90 % (1/10 stratified val miss)
-- The substantive-deletion metric is reference-free (compares output to input only, after filler-stripping) so it cannot catch *semantic* hallucinations — only word-level losses
-- Not designed for formal written text — trained on spoken language patterns
+| [MLX 4-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) | ~195 MB | Smallest |
 
 ## License
 
 MIT
-
-## Links
-
-- **Application:** [sottoasr.app](https://sottoasr.app)
-- **Source:** [github.com/juanqui/sottoasr](https://github.com/juanqui/sottoasr)
-- **Dataset:** [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)

config.json

@@ -4,6 +4,7 @@
   ],
   "block_auto_adjust_ff_dim": true,
   "block_dim": 1024,
+  "block_ff_dim": 6656,
   "block_ffn_dim_multiplier": 1.0,
   "block_mlp_init_scale": 1.0,
   "block_multiple_of": 256,
@@ -56,6 +57,5 @@
   "use_cache": false,
   "use_pos_enc": true,
   "vocab_size": 65536,
-  "block_ff_dim": 6656,
   "rope_theta": 1000000.0
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b157b5499dc3ff418e449e669454fae67dd248152bfc13d008ff56abffec432f
+oid sha256:48942a9e661298a8c531c901cca9d0dac8affc82fd06741f82599e0b3352ce8b
 size 708984464