Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,93 @@
+---
+license: apache-2.0
+base_model: Nanbeige/Nanbeige4.1-3B
+tags:
+  - litert
+  - litert-lm
+  - litertlm
+  - on-device
+  - edge
+  - reasoning
+pipeline_tag: text-generation
+library_name: litert-lm
+---
+
+# Nanbeige4.1-3B — LiteRT-LM (blockwise int4)
+
+[Nanbeige/Nanbeige4.1-3B](https://huggingface.co/Nanbeige/Nanbeige4.1-3B) converted to the
+**LiteRT-LM** (`.litertlm`) format for on-device inference with Google's
+[LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime (the engine behind the
+official `litert-community/*` models).
+
+Nanbeige4.1-3B is a fresh (Dec 2025) **phone-size reasoning** model on a plain dense
+Llama architecture (Apache-2.0), reported to be competitive with much larger models. It
+works the problem inside a `<think>…</think>` block before giving the final answer.
+
+| | |
+|---|---|
+| **File** | `model.litertlm` (~2.2 GB; embedding externalized so every section is <2 GiB → loads on iOS) |
+| **Quantization** | int4 weights — **blockwise (block 32) + OCTAV** optimal-clipping, symmetric; embedding INT8 |
+| **Compute** | integer |
+| **Context (KV cache)** | 4096 |
+| **Base model** | Nanbeige/Nanbeige4.1-3B (Apache-2.0) |
+| **Decode speed** | ~89 tok/s (Mac M4 Max, Metal GPU, greedy) |
+
+## Usage
+
+```bash
+# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
+litert_lm_main \
+  --model_path model.litertlm \
+  --backend gpu \
+  --input_prompt "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts total?"
+```
+
+The `.litertlm` bundle carries the tokenizer and the prompt template (ChatML —
+`<|im_start|>role\n … <|im_end|>`), so no separate tokenizer files are needed. This is a
+**reasoning** model: it emits a `<think>…</think>` chain then the final answer (best
+evaluated with a generous token budget), and stops cleanly at `<|im_end|>`.
+
+## Run on Android
+
+Install a recent [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery)
+(1.0.16+ can import `.litertlm` directly from Hugging Face), download `model.litertlm`
+(or import this repo in-app), pick the **GPU** backend (CPU also works), and chat. Give it
+a high max-tokens — it's a reasoning model with long chains of thought.
+
+## Quality — GSM8K
+
+Measured on GSM8K (n=50, greedy, 0-shot chain-of-thought, **max-tokens 2048** — a
+reasoning model needs the budget to finish; scoring it at 512 tokens falsely penalises it):
+
+| Configuration | GSM8K |
+|---|---|
+| **This model — LiteRT int4 (block32 + OCTAV)** | **84.0%** |
+
+84% is a strong on-device GSM8K for a 3B, non-degenerate; the model also passes the local
+quality gate **8/8** with a clean stop at `<|im_end|>`. Blockwise-32 + OCTAV optimal-clipping
+(data-free) preserves the accuracy versus a naive min-max int4.
+
+## Conversion
+
+Converted with [`litert-torch`](https://github.com/google-ai-edge/litert) using a
+**blockwise int4** recipe (INT4 weights, block size 32, symmetric, OCTAV optimal-clipping)
+with the embedding at INT8, KV cache 4096, and a ChatML prompt template. Nanbeige4.1 is a
+standard dense `LlamaForCausalLM`, so it rides the existing converter and runtime with no
+custom graph code.
+
+**`externalize_embedder=True` (required for iPhone).** The large 166k-token vocab makes the
+weights a >2 GiB single TFLite section, which exceeds the ~2 GiB single-section `mmap` limit
+on iOS. Externalizing the embedding drops the main section under 2 GiB so the model loads on
+**iPhone (Metal GPU)** as well as Android/desktop. Same weights, so GSM8K is unchanged.
+
+**Added-tokens tokenizer fix.** Nanbeige's 10 special tokens (`<|im_start|>`, `<|im_end|>`,
+`<think>`, `</think>`, `<tool_call>`, …) live at vocab ids 166100–166109, above the base
+SentencePiece vocab (166100). The base SP conversion drops them, so the reasoning model would
+generate `<think>` (id 166103) and the runtime would crash with *"Token id out of range."* The
+converted tokenizer here appends those added tokens as USER_DEFINED SentencePiece pieces at
+their exact ids (padded to the model vocab), so `<think>` and friends decode correctly.
+
+## License
+
+Apache-2.0, inherited from the base model
+[Nanbeige/Nanbeige4.1-3B](https://huggingface.co/Nanbeige/Nanbeige4.1-3B).