---
license: apache-2.0
library_name: gguf
base_model: Nanbeige/Nanbeige4.2-3B
pipeline_tag: text-generation
tags:
- gguf
- quantized
- llama.cpp
- nanbeige4.2
- nanbeige
- open-source
- webgpu
---

# Nanbeige4.2-3B-GGUF

Quantized GGUF versions of [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) for efficient local inference via llama.cpp, LM Studio, and Ollama.

Nanbeige4.2-3B is a loop transformer (num_loops=2) trained on Chinese/English data. 3B parameters, 256K context, optimized for general-purpose chat and instruction following.

## Quantization overview

This repo ships **best quality per compression band** — no Q2, no I-quants, no XL variants. Just the cleanest K-quant in each size band plus the lossless baselines.

| File | Size | Bits/weight | Use case |
|------|------|-------------|----------|
| `Nanbeige4.2-3B-F16.gguf` | ~8.2 GB | 16 | Full precision, lossless |
| `Nanbeige4.2-3B-BF16.gguf` | ~8.2 GB | 16 (bfloat16) | Faster loading, equivalent quality |
| `Nanbeige4.2-3B-Q8_0.gguf` | ~4.3 GB | 8 | Near-lossless |
| `Nanbeige4.2-3B-Q6_K.gguf` | ~3.4 GB | 6 | Excellent quality |
| `Nanbeige4.2-3B-Q5_K_M.gguf` | ~2.9 GB | ~5.5 | High quality |
| `Nanbeige4.2-3B-Q4_K_M.gguf` | ~2.5 GB | ~4.5 | Recommended default |
| `Nanbeige4.2-3B-Q3_K_L.gguf` | ~2.3 GB | ~3.5 | Tight memory, lowest viable quality |

All K-quants use an importance matrix (imatrix) calibrated against Project Gutenberg text for better quality at low bit-widths.

## Running

### llama.cpp (CLI)

```bash
llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -c 4096 --color -i   --temp 0.1 --top-k 50 --repeat-penalty 1.1
```

### llama.cpp (one-liner via HF)

```bash
llama-cli -hf nicolasembleton/Nanbeige4.2-3B-GGUF:Q4_K_M -c 4096 --color -i
```

### Python (llama-cpp-python)

```python
from llama_cpp import Llama

llm = Llama(
    model_path="Nanbeige4.2-3B-Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    n_gpu_layers=99,  # offload all layers to GPU if available
)
print(llm("Hello, how are you?", max_tokens=256)["choices"][0]["text"])
```

### Ollama

Create a `Modelfile`:

```
FROM ./Nanbeige4.2-3B-Q4_K_M.gguf
```

Then:

```bash
ollama create nanbeige4.2-3b -f Modelfile
ollama run nanbeige4.2-3b
```

## In-browser (ONNX Runtime Web + Transformers.js)

For browser-based inference, use ONNX Runtime Web directly — no web-llm needed.

### Option 1: Our ONNX export (cross-browser, including Apple Safari)

[`nicolasembleton/Nanbeige4.2-3B-ONNX`](https://huggingface.co/nicolasembleton/Nanbeige4.2-3B-ONNX) — BF16 weights with a working `num_loops=2` unroll. Works in Chrome, Firefox, Edge (WebGPU fast path) and Apple Safari 17+ macOS (WASM fallback).

```js
import * as ort from "onnxruntime-web";

const session = await ort.InferenceSession.create(
  "https://huggingface.co/nicolasembleton/Nanbeige4.2-3B-ONNX/resolve/main/model.onnx",
  { executionProviders: ["webgpu", "wasm"] },  // Safari 17 macOS falls back to WASM
);

const inputIds = BigInt64Array.from(/* your token ids */);
const feeds = {
  input_ids: new ort.Tensor("int64", inputIds, [1, inputIds.length]),
  attention_mask: new ort.Tensor("int64", new BigInt64Array(inputIds.length).fill(1n), [1, inputIds.length]),
  position_ids: new ort.Tensor("int64", [...Array(inputIds.length).keys()].map(BigInt), [1, inputIds.length]),
};
const { logits } = await session.run(feeds);
```

### Option 2: Community WebGPU-only export (Chrome, Firefox, Edge)

[`Michionlion/Nanbeige4.2-3B-ONNX-WebGPU`](https://huggingface.co/Michionlion/Nanbeige4.2-3B-ONNX-WebGPU) — Q4F16 WebGPU-optimized, smaller (~3 GB) but no WASM fallback.

```js
import { pipeline } from "@huggingface/transformers";

const generator = await pipeline(
  "text-generation",
  "Michionlion/Nanbeige4.2-3B-ONNX-WebGPU",
  {
    device: "webgpu",
    dtype: "q4f16",
    model_file_name: "model_webgpu_mlp",
    use_external_data_format: 2,
  },
);
const output = await generator("Hello, how are you?", { max_new_tokens: 256 });
```

> **Architecture note:** Nanbeige's `num_loops=2` (two passes per physical layer) was previously only available in community ONNX exports via custom kernels (`MatMulNBitsMlp`). Our export unrolls the loop at the Python level — 44 sequential layer calls with shared 22 weights — producing a standard ONNX graph that runs in stock ONNX Runtime Web.

> **Note:** This **GGUF** repo is for native/server-side inference (llama.cpp, Ollama, LM Studio). The ONNX repo above covers browser inference with cross-browser support including Apple Safari WASM fallback.

## Architecture

`NanbeigeForCausalLM` — loop transformer with 22 layers, num_loops=2 (two passes per layer). 48 heads, 8 KV heads, 3072 hidden, 166144 vocab, 256K context.

Built with llama.cpp `b10276` (Aug 2026) — the first release to include Nanbeige architecture support.

## Files

- `*.gguf` — quantized model files
- `README.md` — this file

## License

Inherited: Apache 2.0 license (see [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B)).

## Citation

```bibtex
@misc{nanbeige42-3b-gguf,
  title = {{Nanbeige4.2-3B-GGUF}},
  author = {{Nanbeige, quantizations by nicolasembleton}},
  year = {{2026}},
  howpublished = {{Hugging Face}},
  note = {{GGUF quantizations of Nanbeige4.2-3B. For browser inference use nicolasembleton/Nanbeige4.2-3B-ONNX (cross-browser including Apple Safari) or Michionlion/Nanbeige4.2-3B-ONNX-WebGPU (WebGPU-only).}},
}}
```