---
license: mit
language:
- en
base_model: LiquidAI/LFM2.5-350M-Base
tags:
- speech-to-text
- transcript-cleanup
- text-correction
- asr-post-processing
- LFM
- LiquidAI
- grpo
- full-fine-tune
- preservation
pipeline_tag: text-generation
datasets:
- juanquivilla/sotto-transcript-cleanup
---

# SottoASR Transcript Cleanup — LFM2.5-350M (Full Precision, v36 + Preservation)

[sottoasr.app](https://sottoasr.app) · [MLX 5-bit (recommended)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) · [MLX 4-bit (smaller)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) · [Training Dataset](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)

## Overview

**Full-precision bf16** fine-tune of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base) for on-device speech-to-text transcript cleanup. This is the **training artifact** — for on-device deployment on Apple Silicon, use the [5-bit MLX variant](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit) instead.

This model powers on-device transcript cleanup in [SottoASR](https://sottoasr.app) — a local, privacy-first speech-to-text application for macOS. It removes filler words, corrects grammar, formats punctuation, handles false starts and self-corrections, restructures long dictations into paragraph-formatted prose, and **— new in v36 — preserves substantive content reliably even on long inputs**, all locally with zero cloud dependency.

## What's new in v36

v36 fixes the **aggressive-edits failure mode** that earlier checkpoints occasionally exhibited: on long inputs the model would sometimes delete substantive content along with the fillers. v36 is a GRPO **full fine-tune** (all 354M params trainable, no LoRA) with a substantive-deletion-aware reward that penalizes any deletion of non-filler words beyond a 10% threshold. Result: high-substantive-deletion incidence on long inputs drops from **3.85% → 0.64%** while filler-free rate climbs from **50.9% → 96.9%**.

| Capability | v23 baseline | **v36 (this model)** |
|---|---|---|
| Filler-Free rate | 50.9 % | **96.9 %** ⭐ |
| Substantive-deletion >15% on long inputs | 3.85 % | **0.64 %** ⭐ |
| ROUGE-L F1 on long inputs (>100 words) | 0.9242 | **0.9425** |
| Set-based deletion >40% (legacy metric) | 11.4 % | **3.4 %** |
| Word retention median | 0.899 | 0.884 |
| Composite winner score | 131.7 | **189.2** |

The substantive-deletion metric is the one that maps most directly to the user-reported failure mode: it counts how often the model removes more than 15% of the *non-filler* words from a long input. v36 is roughly **6× rarer to do this than the v23 baseline**.

v36 is the breakthrough run from a 28-run autonomous GRPO campaign. All 14 LoRA r=32 variants we tried converged to a ~7–8% sub-del-15-long floor regardless of reward shape, oversampling ratio, or warm-start checkpoint. The full fine-tune cleared the floor by a 12× margin — the conclusion: with 96 GB of VRAM available, full fine-tuning of a 354M model is the right tool for shaping fine-grained word-level behavior.

## Key Specs

| Property | Value |
|----------|-------|
| **Size** | **676 MB** |
| **ROUGE-L F1 (long inputs)** | **0.9425** |
| **Filler-Free rate** | **96.9 %** ⭐ |
| **Substantive-deletion >15% (long)** | **0.64 %** ⭐ |
| **Architecture** | Hybrid: 10 conv + 6 GQA attention (354M params) |
| **Precision** | bf16 |
| **Training method** | Full fine-tune GRPO (no LoRA) |
| **Training context** | 2,048 prompt + 1,024 completion |
| **Native context** | 32,768 tokens (128K base) |
| **Hardware** | 1× RTX PRO 6000 Blackwell (96 GB), ~46 min for 1000 GRPO steps |

## What It Does

Takes raw, unpunctuated ASR output and produces clean, readable text:

| Input (raw ASR) | Output (cleaned) |
|-----------------|------------------|
| so uh basically we need to fix the deployment pipeline | We need to fix the deployment pipeline. |
| the deadline is friday no monday we have until monday | The deadline is Monday. |
| what we what i wanted to say is the tests pass | What I wanted to say is the tests pass. |
| okay so the thing is basically we're running out of disk space | We're running out of disk space. |
| uh yes | Yes. |

### Paragraph emission on long dictations (inherited from v23)

Long, multi-topic input is restructured into paragraph-formatted prose with `\n\n` breaks at natural topic boundaries:

**Input:**
> okay so were having some real issues with the deployment pipeline and i want to walk through whats going wrong the main problem is that the redis cache is timing out during deploys we push a new version and then for about thirty seconds the connections hang and customers see errors so i think we need to add a graceful shutdown period before we kill the old pods now separately the elasticsearch cluster has been a pain we deployed a new svelte frontend last month and it caused index corruption we need to validate the schema before pushing anything live going forward i think the answer here is to add both of these checks to our standard deployment checklist

**Output:**
> We're having some real issues with the deployment pipeline, and I want to walk through what's going wrong. The main problem is that the Redis cache is timing out during deploys. We push a new version and then for about thirty seconds the connections hang, and customers see errors.
>
> I think we need to add a graceful shutdown period before we kill the old pods. Now separately, the Elasticsearch cluster has been a pain. We deployed a new Svelte frontend last month and it caused index corruption. We need to validate the schema before pushing anything live. Going forward, I think the answer here is to add both of these checks to our standard deployment checklist.

## Benchmark Results

Evaluated on a 350-sample held-out mix: 200 random samples from `data_v23_paragraphs/val.jsonl` plus 150 held-out preservation samples (the last 150 of `preservation_samples.jsonl`, never seen during training).

### Headline numbers vs baselines

| Metric | v23 baseline | v24 (Apr) | **v36 (this model)** |
|--------|---:|---:|---:|
| ROUGE-L F1 (overall) | 0.9357 | 0.9300 | 0.9076 |
| ROUGE-L F1 (long inputs >100 words) | 0.9242 | 0.9144 | **0.9425** |
| Filler-Free rate | 50.9 % | 74.9 % | **96.9 %** ⭐ |
| Set-based deletion >40% | 11.4 % | 12.9 % | **3.4 %** |
| **Substantive-deletion >15% on long inputs** | **3.85 %** | — | **0.64 %** ⭐ |
| Substantive-deletion >25% on long inputs | 0.0 % | — | 0.0 % |
| Set-based deletion median | 0.107 | 0.175 | 0.092 |
| Word retention median | 0.899 | 0.818 | 0.884 |

v36 is **slightly more aggressive on short inputs** (overall ROUGE-L drops from 0.936 → 0.908 — the model more readily strips fillers and tightens punctuation) but **better on long inputs** (ROUGE-L 0.924 → 0.943) where preservation matters most. For workloads dominated by long-form dictation (the design target), v36 is strictly better than v23.

## Usage

### Prompt Format

```
### Input:
{raw transcript}

### Output:
{model generates cleaned text}
```

### Python Example

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "juanquivilla/sotto-cleanup-lfm25-350m",
    dtype=torch.bfloat16, trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("juanquivilla/sotto-cleanup-lfm25-350m")

text = "so uh basically we need to fix the deployment pipeline"
prompt = f"### Input:\n{text}\n\n### Output:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
output = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
if "###" in output:
    output = output[:output.index("###")]
print(output.strip())
# → "We need to fix the deployment pipeline."
```

For long dictation that may need paragraph formatting, use a higher `max_new_tokens` (1024–2048).

## Training Details

### Pipeline

```
LiquidAI/LFM2.5-350M-Base
  → SFT (v23): 157,556 rows (v22 base + 4,012 paragraph_formatting),
                LR 3e-5, β2=0.95, 3 epochs, bf16+tf32, packed 4,096 ctx
  → GRPO R5/R6 (v23): LoRA r=32, paragraph emission tuned
                      → main val ROUGE-L 0.9499, Filler-Free 91.0 %, paragraph rate 89 %
  → GRPO v36 (this model): FULL fine-tune (no LoRA), all 354M params,
                            substantive-deletion-aware reward, 6,000 examples
                            (3,000 flat + 3,000 oversampled preservation, 50/50),
                            batch=48, num_gen=8, 1000 steps, LR 5e-6, ~46 min
                            → 350-sample eval: 96.9 % filler-free, 0.64 % sub-del-15-long
```

### v36 reward function

The v36 reward is a sum of:

1. **Quality**: `+5.0 × ROUGE-L F1` against the reference clean output.
2. **Filler penalty**: `−min(count × 0.5, 2.0) × 3.0` per generation.
3. **Format bonus**: +0.2 each for capital-leading and terminal punctuation.
4. **Substantive-deletion penalty**: strip filler words from input, compute multiplicity-aware (multiset) deletion ratio of substantive words. If `> 0.10` (threshold), apply `−10.0 × ramp` linearly scaling from 0 at 0.10 to 1.0 at 0.35 deletion. Clipped at `−10.0`.
5. **Preservation bonus**: `+1.5` when output/input word ratio is in `[0.70, 0.95]`.

Hyperparameters: `DEL_CAP=−10.0`, `DEL_THRESH=0.10`, `DEL_RAMP_END=0.35`, `PRES_BONUS=+1.5`, `PRES_BAND=(0.70, 0.95)`.

### Why full fine-tune

The campaign ran 14 LoRA variants (v25–v38, v50, v52) and one full fine-tune (v36) of the same substantive-deletion-aware reward. **All LoRA r=32 runs converged to ~7–8% sub-del-15-long** regardless of reward shape, oversampling ratio, threshold, cap, or warm-start checkpoint. The full FT run (v36) **broke through to 0.64%** — a 12× improvement. With 96 GB of VRAM available, full fine-tuning a 354M model is the right tool for shaping fine-grained word-level behavior; LoRA's low-rank constraint was the limiting factor.

### Dataset

**157,556 train / 7,121 val rows** in [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup), plus a **1,352-sample preservation set** (long inputs specifically constructed to stress preservation under heavy filler removal). v36 GRPO trains on a 6,000-example mix: 3,000 flat from `train.jsonl` + 3,000 from the 1,352-sample preservation set (oversampled ~2.2× for a 50/50 balance).

## All Variants

| Variant | Size | Use Case |
|---------|------|----------|
| **[Full precision (this)](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m)** | 676 MB | Training, GPU inference |
| **[MLX 5-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit)** | ~237 MB | **Recommended for Apple Silicon** |
| [MLX 4-bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) | ~195 MB | Smallest, slight quality trade-off |

## Limitations

- Optimized for **English** conversational/meeting-style speech
- Domain-specific jargon (medical, legal) may not be corrected without additional fine-tuning
- v36 is slightly more aggressive on short inputs than v23 (overall ROUGE-L 0.908 vs 0.936); for workloads dominated by short utterances where lexical fidelity matters more than filler removal, v23 may be a better fit
- The substantive-deletion metric is reference-free (compares output to input only, after filler-stripping) so it cannot catch *semantic* hallucinations — only word-level losses
- Not designed for formal written text — trained on spoken language patterns

## License

MIT

## Links

- **Application:** [sottoasr.app](https://sottoasr.app)
- **Source:** [github.com/juanqui/sottoasr](https://github.com/juanqui/sottoasr)
- **Dataset:** [juanquivilla/sotto-transcript-cleanup](https://huggingface.co/datasets/juanquivilla/sotto-transcript-cleanup)