v23 R6 (4-stage SFT->GRPO->Stage2->GRPO): ROUGE-L 0.9537 (tied v22), Filler-Free 91.1% (beats v22 90.3%), paragraph rate 91.5% — definitive v23 model

README.md

@@ -29,24 +29,24 @@ This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.ap
 
 v23 (this model) adds **paragraph emission** for long-form dictation. The previous v18/v22 production model produced output as a single run-on paragraph regardless of input length, which made multi-topic dictations hard to read. v23 was retrained on a dataset augmented with **4,012 new `paragraph_formatting` samples** generated via Bedrock Claude Haiku 4.5, teaching the model to insert `\n\n` paragraph breaks at natural topic / time-reference / discourse-marker boundaries.
 
-| Capability | v22 (previous prod) | **v23 R5 (this model)** |
+| Capability | v22 (previous prod) | **v23 R6 (this model)** |
 |---|---|---|
-| Paragraph emission rate on long inputs | **0.0 %** | **89.5 %** |
-| ROUGE-L on paragraph-formatted inputs | 0.9521 | **0.9783** |
-| ROUGE-L on standard val set | 0.9539 | 0.9505 |
+| Paragraph emission rate on long inputs | **0.0 %** | **91.5 %** |
+| ROUGE-L on paragraph-formatted inputs | 0.9521 | **0.9784** |
+| ROUGE-L on standard val set | 0.9539 | 0.9537 |
 | **Filler-Free rate on standard val set** | 90.3 % | **91.0 %** ⭐ |
 
-This is the **R5 variant** trained with GRPO at LR 5e-6 with **paragraph rows excluded from the GRPO sample** (the SFT base already learned paragraph emission, so GRPO can focus purely on the cleanup gradient). This produces stronger updates without the conflicted reward landscape that paragraph rows introduce., achieving **91.0 % filler-free vs v22's 90.3 %** — the first v23 variant to strictly beat v22 on this user-visible metric. The trade-off is a small drop in main val ROUGE-L (0.9499 vs v22's 0.9539, within natural seed variance) and slightly lower paragraph emission rate vs the R1 variant (89 % vs 91.5 %).
+This is the **R6 variant** trained with GRPO at LR 5e-6 with **a 4-stage pipeline (SFT → GRPO → Stage-2 SFT recovery → final GRPO with paragraph rows excluded)** (the SFT base already learned paragraph emission, so GRPO can focus purely on the cleanup gradient). This produces stronger updates without the conflicted reward landscape that paragraph rows introduce., achieving **91.0 % filler-free vs v22's 90.3 %** — the first v23 variant to strictly beat v22 on this user-visible metric. The trade-off is a small drop in main val ROUGE-L (0.9499 vs v22's 0.9539, within natural seed variance) and slightly lower paragraph emission rate vs the R1 variant (89 % vs 91.5 %).
 
 ## Key Specs
 
 | Property | Value |
 |----------|-------|
 | **Size** | **676 MB** |
-| **ROUGE-L (val set, 1000 samples)** | **0.9505** |
-| **Exact Match** | **63.9 %** |
+| **ROUGE-L (val set, 1000 samples)** | **0.9537** |
+| **Exact Match** | **64.3 %** |
 | **Filler-Free** | **91.0 %** ⭐ (beats v22 by +0.7 pts) |
-| **Paragraph rate (long inputs)** | **89.5 %** |
+| **Paragraph rate (long inputs)** | **91.5 %** |
 | **Latency** | **118 ms** average per transcript (RTX 4090) |
 | **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
 | **Precision** | bf16 |
@@ -86,20 +86,20 @@ Notice the model:
 
 ### Main val set (1000 samples, cleaned val.jsonl from training data)
 
-| Metric | v23 R5 (this model) | v22 baseline |
+| Metric | v23 R6 (this model) | v22 baseline |
 |---|---|---|
 | ROUGE-L | 0.9499 | 0.9539 |
-| Exact Match | 63.9 % | 64.8 % |
+| Exact Match | 64.3 % | 64.8 % |
 | **Filler-Free** | **91.0 %** ⭐ | 90.3 % |
 | Paragraph rate | 0.0 % | 0.0 % |
 | Avg latency | 118 ms | 117 ms |
 
 ### Paragraph val set (200 paragraph_formatting samples)
 
-| Metric | v23 R5 (this model) | v22 baseline | Δ |
+| Metric | v23 R6 (this model) | v22 baseline | Δ |
 |---|---|---|---|
-| ROUGE-L | **0.9783** | 0.9521 | **+0.027** |
-| **Paragraph emission rate** | **89.5 %** | **0.0 %** | **+89 pts** |
+| ROUGE-L | **0.9784** | 0.9521 | **+0.027** |
+| **Paragraph emission rate** | **91.5 %** | **0.0 %** | **+89 pts** |
 | Exact Match | 2.0 % | 0.0 % | +2.0 pts |
 | Avg latency | 1.45 s | 1.40 s | +50 ms |
 
@@ -107,8 +107,8 @@ Notice the model:
 
 | Metric | This model (354M) | Prompted Qwen 2B | Improvement |
 |--------|-------------------|-------------------|-------------|
-| ROUGE-L | **0.9505** | 0.891 | **+0.059** |
-| Exact Match | **63.9 %** | 37 % | **+26 pts** |
+| ROUGE-L | **0.9537** | 0.891 | **+0.059** |
+| Exact Match | **64.3 %** | 37 % | **+26 pts** |
 | Inference | **118 ms** | 1.0 s | **8.5× faster** |
 | Parameters | 354M | 2B | **5.6× smaller** |
 
@@ -162,8 +162,8 @@ LiquidAI/LFM2.5-350M-Base
          cosine schedule, 50 warmup steps, weight_decay 0.01,
          bf16+tf32, packed 4,096 context, seed 42
          → eval_loss 1.016 (vs v22's 1.0306, -0.014)
-  → GRPO R5: LoRA r=32, alpha=16, all linear layers,
-             LR 5e-6 cosine, paragraph rows excluded from sample, 5K samples × 4 generations,
+  → GRPO R6: LoRA r=32, alpha=16, all linear layers,
+             4-stage: SFT → GRPO → Stage-2 SFT recovery on v22-only data (LR 5e-6, 16 steps) → final GRPO LR 5e-6 with paragraph rows excluded, 5K samples × 4 generations,
              reward = ROUGE-L × 5.0 - filler_count × 0.5 (capped 2.0) × 3.0 + format_bonus
              → final main val ROUGE-L 0.9499 / Filler-Free 91.0 % / paragraph rate 89 %
 ```

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fbd7682fe0c4dd75781450418401a61f62fc65da79f905a3be4d0f06d7613016
+oid sha256:272c8cd400d249fe900c6ab042f8dc0936f07b3dda8a1421bdd8b9f46a6ddaab
 size 708984464