v36: full-FT GRPO with substantive-deletion-aware reward — filler-free 96.9%, sub-del-15-long 0.64%

README.md

@@ -10,12 +10,15 @@ tags:
 - asr-post-processing
 - LFM
 - LiquidAI
+- grpo
+- full-fine-tune
+- preservation
 pipeline_tag: text-generation
 datasets:
 - juanquivilla/sotto-transcript-cleanup
 ---
 
-# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v23 + Paragraphs)
+# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v36 + Preservation)
 
 [sottoasr.app](https://sottoasr.app) · [MLX 5-bit (recommended)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) · [MLX 4-bit (smaller)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) · [Training Dataset](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)
 
@@ -23,34 +26,39 @@ datasets:
 
 **Full-precision bf16** fine-tune of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base) for on-device speech-to-text transcript cleanup. This is the **training artifact** — for on-device deployment on Apple Silicon, use the [5-bit MLX variant](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) instead.
 
-This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, and **— new in v23 — restructures long dictations into paragraph-formatted prose**, all locally with zero cloud dependency.
+This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, and **— new in v36 — preserves substantive content reliably even on long inputs**, all locally with zero cloud dependency.
 
-## What's new in v23
+## What's new in v36
 
-v23 (this model) adds **paragraph emission** for long-form dictation. The previous v18/v22 production model produced output as a single run-on paragraph regardless of input length, which made multi-topic dictations hard to read. v23 was retrained on a dataset augmented with **4,012 new `paragraph_formatting` samples** generated via Bedrock Claude Haiku 4.5, teaching the model to insert `\n\n` paragraph breaks at natural topic / time-reference / discourse-marker boundaries.
+v36 fixes the **aggressive-edits failure mode** that earlier checkpoints occasionally exhibited: on long inputs the model would sometimes delete substantive content along with the fillers. v36 is a GRPO **full fine-tune** (all 354M params trainable, no LoRA) with a substantive-deletion-aware reward that penalizes any deletion of non-filler words beyond a 10% threshold. Result: high-substantive-deletion incidence on long inputs drops from **3.85% → 0.64%** while filler-free rate climbs from **50.9% → 96.9%**.
 
-| Capability | v22 (previous prod) | **v23 R6 (this model)** |
+| Capability | v23 baseline | **v36 (this model)** |
 |---|---|---|
-| Paragraph emission rate on long inputs | **0.0 %** | **91.5 %** |
-| ROUGE-L on paragraph-formatted inputs | 0.9521 | **0.9784** |
-| ROUGE-L on standard val set | 0.9539 | 0.9537 |
-| **Filler-Free rate on standard val set** | 90.3 % | **91.0 %** ⭐ |
+| Filler-Free rate | 50.9 % | **96.9 %** ⭐ |
+| Substantive-deletion >15% on long inputs | 3.85 % | **0.64 %** ⭐ |
+| ROUGE-L F1 on long inputs (>100 words) | 0.9242 | **0.9425** |
+| Set-based deletion >40% (legacy metric) | 11.4 % | **3.4 %** |
+| Word retention median | 0.899 | 0.884 |
+| Composite winner score | 131.7 | **189.2** |
 
-This is the **R6 variant** trained with GRPO at LR 5e-6 with **a 4-stage pipeline (SFT → GRPO → Stage-2 SFT recovery → final GRPO with paragraph rows excluded)** (the SFT base already learned paragraph emission, so GRPO can focus purely on the cleanup gradient). This produces stronger updates without the conflicted reward landscape that paragraph rows introduce., achieving **91.0 % filler-free vs v22's 90.3 %** — the first v23 variant to strictly beat v22 on this user-visible metric. The trade-off is a small drop in main val ROUGE-L (0.9499 vs v22's 0.9539, within natural seed variance) and slightly lower paragraph emission rate vs the R1 variant (89 % vs 91.5 %).
+The substantive-deletion metric is the one that maps most directly to the user-reported failure mode: it counts how often the model removes more than 15% of the *non-filler* words from a long input. v36 is roughly **6× rarer to do this than the v23 baseline**.
+
+v36 is the breakthrough run from a 28-run autonomous GRPO campaign. All 14 LoRA r=32 variants we tried converged to a ~7–8% sub-del-15-long floor regardless of reward shape, oversampling ratio, or warm-start checkpoint. The full fine-tune cleared the floor by a 12× margin — the conclusion: with 96 GB of VRAM available, full fine-tuning of a 354M model is the right tool for shaping fine-grained word-level behavior.
 
 ## Key Specs
 
 | Property | Value |
 |----------|-------|
 | **Size** | **676 MB** |
-| **ROUGE-L (val set, 1000 samples)** | **0.9537** |
-| **Exact Match** | **64.3 %** |
-| **Filler-Free** | **91.0 %** ⭐ (beats v22 by +0.7 pts) |
-| **Paragraph rate (long inputs)** | **91.5 %** |
-| **Latency** | **118 ms** average per transcript (RTX 4090) |
+| **ROUGE-L F1 (long inputs)** | **0.9425** |
+| **Filler-Free rate** | **96.9 %** ⭐ |
+| **Substantive-deletion >15% (long)** | **0.64 %** ⭐ |
 | **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
 | **Precision** | bf16 |
-| **Training context** | 4,096 tokens (packed); model supports 32,768 tokens natively, 128K base |
+| **Training method** | Full fine-tune GRPO (no LoRA) |
+| **Training context** | 2,048 prompt + 1,024 completion |
+| **Native context** | 32,768 tokens (128K base) |
+| **Hardware** | 1× RTX PRO 6000 Blackwell (96 GB), ~46 min for 1000 GRPO steps |
 
 ## What It Does
 
@@ -64,9 +72,9 @@ Takes raw, unpunctuated ASR output and produces clean, readable text:
 | okay so the thing is basically we're running out of disk space | We're running out of disk space. |
 | uh yes | Yes. |
 
-### NEW in v23: Paragraph emission on long dictations
+### Paragraph emission on long dictations (inherited from v23)
 
-Long, multi-topic input is now restructured into paragraph-formatted prose:
+Long, multi-topic input is restructured into paragraph-formatted prose with `\n\n` breaks at natural topic boundaries:
 
 **Input:**
 > okay so were having some real issues with the deployment pipeline and i want to walk through whats going wrong the main problem is that the redis cache is timing out during deploys we push a new version and then for about thirty seconds the connections hang and customers see errors so i think we need to add a graceful shutdown period before we kill the old pods now separately the elasticsearch cluster has been a pain we deployed a new svelte frontend last month and it caused index corruption we need to validate the schema before pushing anything live going forward i think the answer here is to add both of these checks to our standard deployment checklist
@@ -76,41 +84,24 @@ Long, multi-topic input is now restructured into paragraph-formatted prose:
 >
 > I think we need to add a graceful shutdown period before we kill the old pods. Now separately, the Elasticsearch cluster has been a pain. We deployed a new Svelte frontend last month and it caused index corruption. We need to validate the schema before pushing anything live. Going forward, I think the answer here is to add both of these checks to our standard deployment checklist.
 
-Notice the model:
-- Strips speech disfluencies ("okay so", "uh", "basically")
-- Capitalizes proper nouns (Redis, Elasticsearch, Svelte)
-- Adds correct punctuation
-- **Inserts a paragraph break at the topic shift** ("the elasticsearch cluster has been a pain")
-
 ## Benchmark Results
 
-### Main val set (1000 samples, cleaned val.jsonl from training data)
+Evaluated on a 350-sample held-out mix: 200 random samples from `data_v23_paragraphs/val.jsonl` plus 150 held-out preservation samples (the last 150 of `preservation_samples.jsonl`, never seen during training).
 
-| Metric | v23 R6 (this model) | v22 baseline |
-|---|---|---|
-| ROUGE-L | 0.9499 | 0.9539 |
-| Exact Match | 64.3 % | 64.8 % |
-| **Filler-Free** | **91.0 %** ⭐ | 90.3 % |
-| Paragraph rate | 0.0 % | 0.0 % |
-| Avg latency | 118 ms | 117 ms |
-
-### Paragraph val set (200 paragraph_formatting samples)
-
-| Metric | v23 R6 (this model) | v22 baseline | Δ |
-|---|---|---|---|
-| ROUGE-L | **0.9784** | 0.9521 | **+0.027** |
-| **Paragraph emission rate** | **91.5 %** | **0.0 %** | **+89 pts** |
-| Exact Match | 2.0 % | 0.0 % | +2.0 pts |
-| Avg latency | 1.45 s | 1.40 s | +50 ms |
-
-### vs Prompted Qwen 2B Baseline (from earlier benchmarks)
-
-| Metric | This model (354M) | Prompted Qwen 2B | Improvement |
-|--------|-------------------|-------------------|-------------|
-| ROUGE-L | **0.9537** | 0.891 | **+0.059** |
-| Exact Match | **64.3 %** | 37 % | **+26 pts** |
-| Inference | **118 ms** | 1.0 s | **8.5× faster** |
-| Parameters | 354M | 2B | **5.6× smaller** |
+### Headline numbers vs baselines
+
+| Metric | v23 baseline | v24 (Apr) | **v36 (this model)** |
+|--------|---:|---:|---:|
+| ROUGE-L F1 (overall) | 0.9357 | 0.9300 | 0.9076 |
+| ROUGE-L F1 (long inputs >100 words) | 0.9242 | 0.9144 | **0.9425** |
+| Filler-Free rate | 50.9 % | 74.9 % | **96.9 %** ⭐ |
+| Set-based deletion >40% | 11.4 % | 12.9 % | **3.4 %** |
+| **Substantive-deletion >15% on long inputs** | **3.85 %** | — | **0.64 %** ⭐ |
+| Substantive-deletion >25% on long inputs | 0.0 % | — | 0.0 % |
+| Set-based deletion median | 0.107 | 0.175 | 0.092 |
+| Word retention median | 0.899 | 0.818 | 0.884 |
+
+v36 is **slightly more aggressive on short inputs** (overall ROUGE-L drops from 0.936 → 0.908 — the model more readily strips fillers and tightens punctuation) but **better on long inputs** (ROUGE-L 0.924 → 0.943) where preservation matters most. For workloads dominated by long-form dictation (the design target), v36 is strictly better than v23.
 
 ## Usage
 
@@ -149,7 +140,7 @@ print(output.strip())
 # → "We need to fix the deployment pipeline."
 ```
 
-For long dictation that may need paragraph formatting, use a higher `max_new_tokens` (1024-2048).
+For long dictation that may need paragraph formatting, use a higher `max_new_tokens` (1024–2048).
 
 ## Training Details
 
@@ -157,42 +148,51 @@ For long dictation that may need paragraph formatting, use a higher `max_new_tok
 
 ```
 LiquidAI/LFM2.5-350M-Base
-  → SFT: 157,556 rows (v22 base + 4,012 paragraph_formatting),
-         LR 3e-5, β2=0.95, 3 epochs, batch 1×8,
-         cosine schedule, 50 warmup steps, weight_decay 0.01,
-         bf16+tf32, packed 4,096 context, seed 42
-         → eval_loss 1.016 (vs v22's 1.0306, -0.014)
-  → GRPO R6: LoRA r=32, alpha=16, all linear layers,
-             4-stage: SFT → GRPO → Stage-2 SFT recovery on v22-only data (LR 5e-6, 16 steps) → final GRPO LR 5e-6 with paragraph rows excluded, 5K samples × 4 generations,
-             reward = ROUGE-L × 5.0 - filler_count × 0.5 (capped 2.0) × 3.0 + format_bonus
-             → final main val ROUGE-L 0.9499 / Filler-Free 91.0 % / paragraph rate 89 %
+  → SFT (v23): 157,556 rows (v22 base + 4,012 paragraph_formatting),
+                LR 3e-5, β2=0.95, 3 epochs, bf16+tf32, packed 4,096 ctx
+  → GRPO R5/R6 (v23): LoRA r=32, paragraph emission tuned
+                      → main val ROUGE-L 0.9499, Filler-Free 91.0 %, paragraph rate 89 %
+  → GRPO v36 (this model): FULL fine-tune (no LoRA), all 354M params,
+                            substantive-deletion-aware reward, 6,000 examples
+                            (3,000 flat + 3,000 oversampled preservation, 50/50),
+                            batch=48, num_gen=8, 1000 steps, LR 5e-6, ~46 min
+                            → 350-sample eval: 96.9 % filler-free, 0.64 % sub-del-15-long
 ```
 
-### Dataset
+### v36 reward function
+
+The v36 reward is a sum of:
+
+1. **Quality**: `+5.0 × ROUGE-L F1` against the reference clean output.
+2. **Filler penalty**: `−min(count × 0.5, 2.0) × 3.0` per generation.
+3. **Format bonus**: +0.2 each for capital-leading and terminal punctuation.
+4. **Substantive-deletion penalty**: strip filler words from input, compute multiplicity-aware (multiset) deletion ratio of substantive words. If `> 0.10` (threshold), apply `−10.0 × ramp` linearly scaling from 0 at 0.10 to 1.0 at 0.35 deletion. Clipped at `−10.0`.
+5. **Preservation bonus**: `+1.5` when output/input word ratio is in `[0.70, 0.95]`.
 
-**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup):
+Hyperparameters: `DEL_CAP=−10.0`, `DEL_THRESH=0.10`, `DEL_RAMP_END=0.35`, `PRES_BONUS=+1.5`, `PRES_BAND=(0.70, 0.95)`.
 
-- 153,561 v22 base (prior versions: filler removal, crutch words, self-correction, false starts, grammar, dictation commands, list formatting, preserve-wording, mixed disfluencies, short utterances, long dictation)
-- **3,995 new paragraph_formatting samples** (held out 200 for `paragraph_val.jsonl`) — generated via AWS Bedrock Claude Haiku 4.5, instructed to produce 100–500 word raw input + 2–5 paragraph clean output, split at natural discourse boundaries
+### Why full fine-tune
 
-### Hardware
+The campaign ran 14 LoRA variants (v25–v38, v50, v52) and one full fine-tune (v36) of the same substantive-deletion-aware reward. **All LoRA r=32 runs converged to ~7–8% sub-del-15-long** regardless of reward shape, oversampling ratio, threshold, cap, or warm-start checkpoint. The full FT run (v36) **broke through to 0.64%** — a 12× improvement. With 96 GB of VRAM available, full fine-tuning a 354M model is the right tool for shaping fine-grained word-level behavior; LoRA's low-rank constraint was the limiting factor.
+
+### Dataset
 
-1× RTX 4090, ~42 minutes for SFT + ~30 minutes for GRPO = ~72 min total
+**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup), plus a **1,352-sample preservation set** (long inputs specifically constructed to stress preservation under heavy filler removal). v36 GRPO trains on a 6,000-example mix: 3,000 flat from `train.jsonl` + 3,000 from the 1,352-sample preservation set (oversampled ~2.2× for a 50/50 balance).
 
 ## All Variants
 
 | Variant | Size | Use Case |
 |---------|------|----------|
 | **[Full precision (this)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m)** | 676 MB | Training, GPU inference |
-| **[MLX 5-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit)** | 237 MB | **Recommended for Apple Silicon** |
-| [MLX 4-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) | 195 MB | Smallest, slight quality trade-off |
+| **[MLX 5-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit)** | ~237 MB | **Recommended for Apple Silicon** |
+| [MLX 4-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) | ~195 MB | Smallest, slight quality trade-off |
 
 ## Limitations
 
 - Optimized for **English** conversational/meeting-style speech
 - Domain-specific jargon (medical, legal) may not be corrected without additional fine-tuning
-- Paragraph emission is conditional on input structure — short single-topic inputs (typical) will not be paragraph-broken
-- Filler-free rate on long-form content is lower than on short inputs (long content has more legitimate uses of words like "so", "okay", "right", which the eval list flags)
+- v36 is slightly more aggressive on short inputs than v23 (overall ROUGE-L 0.908 vs 0.936); for workloads dominated by short utterances where lexical fidelity matters more than filler removal, v23 may be a better fit
+- The substantive-deletion metric is reference-free (compares output to input only, after filler-stripping) so it cannot catch *semantic* hallucinations — only word-level losses
 - Not designed for formal written text — trained on spoken language patterns
 
 ## License

config.json

@@ -4,7 +4,6 @@
   ],
   "block_auto_adjust_ff_dim": true,
   "block_dim": 1024,
-  "block_ff_dim": 6656,
   "block_ffn_dim_multiplier": 1.0,
   "block_mlp_init_scale": 1.0,
   "block_multiple_of": 256,
@@ -19,6 +18,7 @@
   "conv_use_xavier_init": true,
   "dtype": "bfloat16",
   "eos_token_id": 7,
+  "full_attn_idxs": null,
   "hidden_size": 1024,
   "initializer_range": 0.02,
   "intermediate_size": 6656,
@@ -52,10 +52,11 @@
     "rope_theta": 1000000.0,
     "rope_type": "default"
   },
-  "tie_embedding": true,
   "tie_word_embeddings": true,
-  "transformers_version": "5.3.0",
+  "transformers_version": "5.6.2",
   "use_cache": false,
   "use_pos_enc": true,
-  "vocab_size": 65536
-}
+  "vocab_size": 65536,
+  "block_ff_dim": 6656,
+  "rope_theta": 1000000.0
+}

generation_config.json

@@ -5,5 +5,5 @@
     7
   ],
   "pad_token_id": 0,
-  "transformers_version": "5.3.0"
+  "transformers_version": "5.6.2"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:272c8cd400d249fe900c6ab042f8dc0936f07b3dda8a1421bdd8b9f46a6ddaab
+oid sha256:06f28fd7204bc24d3e7f42cec46beeb18cd1db8e9041d394efa33b7d18f025ad
 size 708984464

tokenizer_config.json

@@ -6,6 +6,7 @@
   "extra_special_tokens": [],
   "is_local": true,
   "legacy": false,
+  "local_files_only": false,
   "model_input_names": [
     "input_ids",
     "attention_mask"