---
base_model: Nanbeige/Nanbeige4.2-3B
base_model_relation: quantized
quantized_by: iamimmanuelraj
pipeline_tag: text-generation
library_name: gguf
license: other
license_name: nanbeige-license
language:
- en
- zh
tags:
- gguf
- llama.cpp
- nanbeige
- looped-transformer
- quantized
- imatrix
- text-generation
- conversational
---

# Nanbeige4.2-3B — GGUF

GGUF quantizations of [**Nanbeige/Nanbeige4.2-3B**](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) — a 3B "Looped Transformer" (22 layers, `num_loops=2`, hidden 3072, untied 166k vocab, RoPE θ 70M).

**33 quants** from full-precision `F16`/`BF16` down to 1-bit and ternary, including **imatrix-guided IQ** quants for best quality-per-byte at the low end.

Nanbeige4.2-3B support is **merged into mainline [llama.cpp](https://github.com/ggml-org/llama.cpp)** — just build the latest:

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build -j
```

---

## 🚀 Run

**llama.cpp (chat):**
```bash
./build/bin/llama-cli -hf iamimmanuelraj/Nanbeige4.2-3B-GGUF:Q4_K_M -cnv
```
or with a local file:
```bash
./build/bin/llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -cnv
```

**llama-server (OpenAI-compatible API):**
```bash
./build/bin/llama-server -m Nanbeige4.2-3B-Q4_K_M.gguf -c 8192
```

Download a single quant:
```bash
pip install -U "huggingface_hub[cli]"
hf download iamimmanuelraj/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf --local-dir .
```

---

## 📦 Quants

⭐ = recommended.  IQ quants are **imatrix-guided** (better quality at the same size).

### Full precision
| Quant | Size | Notes |
|---|---|---|
| F16  | 8.34 GB | Full 16-bit reference |
| BF16 | 8.34 GB | Full 16-bit (bfloat) |

### High quality
| Quant | Size | Notes |
|---|---|---|
| ⭐ Q8_0   | 4.43 GB | Near-lossless, best quality |
| Q6_K   | 3.42 GB | Excellent, hard to distinguish from F16 |
| ⭐ Q5_K_M | 2.99 GB | Very high quality, great default for quality |
| Q5_K_S | 2.94 GB | Very high quality, slightly smaller |
| Q5_0   | 2.94 GB | Legacy 5-bit |
| Q5_1   | 3.17 GB | Legacy 5-bit, larger |

### Balanced (recommended range)
| Quant | Size | Notes |
|---|---|---|
| ⭐ Q4_K_M | 2.57 GB | **Best size/quality balance — start here** |
| Q4_K_S | 2.50 GB | Balanced, slightly smaller |
| Q4_0   | 2.48 GB | Legacy 4-bit |
| Q4_1   | 2.71 GB | Legacy 4-bit, larger |
| ⭐ IQ4_XS | 2.38 GB | Imatrix 4-bit, beats Q4 at smaller size |
| IQ4_NL | 2.49 GB | Imatrix 4-bit non-linear |

### Small
| Quant | Size | Notes |
|---|---|---|
| Q3_K_L | 2.31 GB | 3-bit, larger/better |
| Q3_K_M | 2.17 GB | 3-bit balanced |
| Q3_K_S | 2.00 GB | 3-bit smaller |
| IQ3_M  | 2.08 GB | Imatrix 3-bit, good |
| IQ3_S  | 2.00 GB | Imatrix 3-bit |
| IQ3_XS | 1.93 GB | Imatrix 3-bit smaller |
| IQ3_XXS| 1.78 GB | Imatrix 3-bit, very small |

### Very small
| Quant | Size | Notes |
|---|---|---|
| Q2_K   | 1.76 GB | 2-bit, usable |
| Q2_K_S | 1.68 GB | 2-bit smaller |
| Q2_0   | 1.60 GB | Legacy 2-bit |
| IQ2_M  | 1.66 GB | Imatrix 2-bit, best small quality |
| IQ2_S  | 1.59 GB | Imatrix 2-bit |
| IQ2_XS | 1.46 GB | Imatrix 2-bit smaller |
| IQ2_XXS| 1.37 GB | Imatrix 2-bit, very small |

### Extreme / experimental
| Quant | Size | Notes |
|---|---|---|
| IQ1_M  | 1.27 GB | Imatrix 1-bit, experimental |
| IQ1_S  | 1.21 GB | Imatrix 1-bit, most degraded |
| Q1_0   | 0.94 GB | 1-bit, smallest, heavy quality loss |
| TQ1_0  | 1.37 GB | Ternary, experimental |
| TQ2_0  | 1.52 GB | Ternary, experimental |

---

## Which should I pick?

- **Most people → `Q4_K_M`** (2.57 GB): best balance of size, speed, quality.
- **Want max quality → `Q8_0`** or `Q6_K`.
- **Tight on RAM/VRAM → `IQ4_XS`** (2.38 GB) or `Q3_K_M` (2.17 GB).
- **Tiny devices → `IQ2_M`** (1.66 GB); below that quality drops sharply.
- **≤ 1.3 GB (`IQ1`/`Q1`/`TQ`)**: experimental — expect noticeable degradation.

Prefer **IQ** over the same-size legacy quant when available — imatrix calibration gives better quality per byte.

---

## Notes

- IQ quants built with an importance matrix (imatrix) for better low-bit fidelity.
- `Q8_0` KV cache recommended for this architecture.
- License follows the [base model's license](https://huggingface.co/Nanbeige/Nanbeige4.2-3B).

*Quantized with [llama.cpp](https://github.com/ggml-org/llama.cpp).*