v15: 5-bit MLX quant (237MB, ROUGE-L ~0.955)

README.md

@@ -1,7 +1,73 @@
 ---
-language: en
-pipeline_tag: text-generation
+license: mit
+language:
+- en
+base_model: juanquivilla/sotto-cleanup-lfm25-350m
 tags:
+- speech-to-text
+- transcript-cleanup
 - mlx
-library_name: mlx
+- quantized
+- apple-silicon
+pipeline_tag: text-generation
 ---
+
+# SottoASR Transcript Cleanup — MLX 5-bit (Recommended)
+
+5-bit quantized [MLX](https://github.com/ml-explore/mlx) version of [sotto-cleanup-lfm25-350m](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) for on-device inference on Apple Silicon Macs.
+
+## Key Specs
+
+| Property | Value |
+|----------|-------|
+| **Base model** | juanquivilla/sotto-cleanup-lfm25-350m (v15) |
+| **Quantization** | 5-bit affine, group_size=64 |
+| **Size on disk** | ~237 MB |
+| **Bits per weight** | 5.502 |
+| **ROUGE-L** | ~0.955 (vs 0.960 bf16) |
+| **Inference** | ~50ms on M1 Pro |
+
+## Quantization Recipe
+
+```bash
+mlx_lm.convert \
+  --hf-path juanquivilla/sotto-cleanup-lfm25-350m \
+  --mlx-path sotto-cleanup-lfm25-350m-mlx-5bit \
+  -q --q-bits 5 --q-group-size 64 \
+  --trust-remote-code
+```
+
+## Usage
+
+```python
+from mlx_lm import load, generate
+from mlx_lm.sample_utils import make_sampler
+
+model, tokenizer = load("juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit")
+sampler = make_sampler(temp=0.0)
+
+text = "so uh basically we need to fix the deployment"
+prompt = f"### Input:\n{text}\n\n### Output:\n"
+
+output = generate(model, tokenizer, prompt=prompt, max_tokens=256, sampler=sampler, verbose=False)
+print(output.split("###")[0].strip())
+# → "We need to fix the deployment."
+```
+
+## Benchmark (bf16 parent model)
+
+| Metric | Score |
+|--------|-------|
+| ROUGE-L | 0.960 |
+| Exact Match | 69.6% |
+| Zero-Filler Rate | 88.1% |
+
+See [parent model card](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) for full benchmark breakdown and training details.
+
+## Other Variants
+
+| Variant | Size | Repo |
+|---------|------|------|
+| bf16 (full precision) | 676 MB | [sotto-cleanup-lfm25-350m](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) |
+| **5-bit MLX (this)** | **237 MB** | — |
+| 4-bit MLX | 195 MB | [sotto-cleanup-lfm25-350m-mlx-4bit](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-4bit) |

chat_template.jinja

@@ -0,0 +1,7 @@
+{{- bos_token -}}{%- set system_prompt = "" -%}{%- set ns = namespace(system_prompt="") -%}{%- if messages[0]["role"] == "system" -%} {%- set ns.system_prompt = messages[0]["content"] -%} {%- set messages = messages[1:] -%}{%- endif -%}{%- if tools -%} {%- set ns.system_prompt = ns.system_prompt + ("
+" if ns.system_prompt else "") + "List of tools: <|tool_list_start|>[" -%} {%- for tool in tools -%} {%- if tool is not string -%} {%- set tool = tool | tojson -%} {%- endif -%} {%- set ns.system_prompt = ns.system_prompt + tool -%} {%- if not loop.last -%} {%- set ns.system_prompt = ns.system_prompt + ", " -%} {%- endif -%} {%- endfor -%} {%- set ns.system_prompt = ns.system_prompt + "]<|tool_list_end|>" -%}{%- endif -%}{%- if ns.system_prompt -%} {{- "<|im_start|>system
+" + ns.system_prompt + "<|im_end|>
+" -}}{%- endif -%}{%- for message in messages -%} {{- "<|im_start|>" + message["role"] + "
+" -}} {%- set content = message["content"] -%} {%- if content is not string -%} {%- set content = content | tojson -%} {%- endif -%} {%- if message["role"] == "tool" -%} {%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%} {%- endif -%} {{- content + "<|im_end|>
+" -}}{%- endfor -%}{%- if add_generation_prompt -%} {{- "<|im_start|>assistant
+" -}}{%- endif -%}

config.json

@@ -18,7 +18,9 @@
     "conv_dim": 1024,
     "conv_use_xavier_init": true,
     "dtype": "bfloat16",
-    "eos_token_id": 7,
+    "eos_token_id": [
+        7
+    ],
     "hidden_size": 1024,
     "initializer_range": 0.02,
     "intermediate_size": 6656,

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.3.0"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c3d11ae83793305087d53be408fa1bfe375c99bceb25fbbe18037ec390bcaa8c
+oid sha256:c4e1f08a02244130bc54148e0eb7a4764bf5511d458c9803edfbd33489befec7
 size 243830226

tokenizer_config.json

@@ -15,6 +15,7 @@
   "sp_model_kwargs": {},
   "spaces_between_special_tokens": false,
   "tokenizer_class": "TokenizersBackend",
+  "tool_parser_type": "pythonic",
   "use_default_system_prompt": false,
   "use_fast": true
 }