README.md

---
language:
- en
license: apache-2.0
base_model: inclusionAI/Ling-3.0-tiny
tags:
- gguf
- llama.cpp
- quantized
- moe
- bailingmoev3
- hybrid-model
- local-llm
- text-generation
pipeline_tag: text-generation
---

# Ling-3.0-tiny-GGUF

GGUF quantizations of [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny), converted for use with compatible `llama.cpp`-based runtimes.

This repository includes a complete selection of standard K-quants and importance-matrix (IQ) quantizations, so you can choose the best balance of model size, speed, and output quality for your hardware.

> **Runtime compatibility:** Ling-3.0-tiny uses the BailingMoeV3 / hybrid architecture. Use a runtime with explicit support for this architecture. Generic or older `llama.cpp` builds may not load these files correctly.

## Available files

| Quantization | Best for |
|---|---|
| `F16` | Highest-fidelity baseline; re-quantization and high-memory systems |
| `Q8_0` | Near-F16 quality with substantially lower memory use |
| `Q6_K` | High-quality local inference |
| `Q5_K_M` | Strong quality-to-size balance |
| `Q5_K_S` | Slightly smaller alternative to Q5_K_M |
| `Q5_0` | Legacy-style 5-bit option |
| `Q4_K_M` | Recommended default for most users |
| `Q4_K_S` | Smaller Q4 K-quant alternative |
| `Q4_0` | Compact legacy-style 4-bit option |
| `IQ4_NL` | High-quality importance-matrix 4-bit option |
| `IQ4_XS` | Compact importance-matrix 4-bit option |
| `Q3_K_L` | Higher-quality 3-bit K-quant |
| `Q3_K_M` | Balanced 3-bit K-quant |
| `Q3_K_S` | Smaller 3-bit K-quant |
| `IQ3_M` | Strong quality-per-GB option for constrained systems |
| `IQ3_S` | Smaller 3-bit IQ option |
| `IQ3_XS` | Very compact IQ 3-bit option |
| `IQ3_XXS` | Extremely compact IQ 3-bit option |
| `Q2_K` | Low-memory K-quant option |
| `IQ2_M` | Compact IQ quant with better quality potential than very-low-bit options |
| `IQ2_S` | Low-memory IQ option |
| `IQ2_XS` | Very small IQ option |
| `IQ2_XXS` | Extremely small IQ option |
| `IQ1_M` | Experimental ultra-low-memory option |
| `IQ1_S` | Smallest experimental option |

## Recommended downloads

| Your priority | Recommended file |
|---|---|
| Best quality | `Ling-3.0-tiny-F16.gguf` |
| Near-original quality | `Ling-3.0-tiny-Q8_0.gguf` |
| High quality with lower memory use | `Ling-3.0-tiny-Q6_K.gguf` |
| Best general-purpose choice | `Ling-3.0-tiny-Q4_K_M.gguf` |
| Small but capable | `Ling-3.0-tiny-IQ3_M.gguf` |
| Tight VRAM / RAM budget | `Ling-3.0-tiny-IQ2_M.gguf` |
| Experimental minimum size | `Ling-3.0-tiny-IQ1_S.gguf` |

For most users, start with **Q4_K_M**. If you have more RAM or VRAM, try **Q5_K_M**, **Q6_K**, or **Q8_0**. IQ quants can offer attractive quality-to-size trade-offs, but results and compatibility may vary by runtime and hardware.

## Usage

Download one `.gguf` file, then run it with a compatible build of `llama.cpp`.

```bash
llama-cli \
  -m Ling-3.0-tiny-Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  -p "Write a concise explanation of retrieval-augmented generation."
```

`-ngl 99` attempts to offload all supported layers to the GPU. Remove it or set `-ngl 0` for CPU-only inference.

## Important notes

- These files are quantized derivatives of the original model; output quality changes depending on the chosen quantization.
- Very low-bit quants, especially IQ1 and IQ2 variants, are intended for memory-constrained or experimental use and may noticeably reduce output quality.
- Use the original model’s license, terms, and usage requirements.
- Validate the selected quantization on your own workload before production use.

## Conversion details

- Base model: [`inclusionAI/Ling-3.0-tiny`](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
- Format: GGUF
- Conversion/runtime branch: BailingMoeV3-compatible `llama.cpp` fork
- Standard K-quants: generated from the F16 GGUF
- IQ quants: generated using an importance matrix calibrated on a text corpus

## Credits

- Original model by [inclusionAI](https://huggingface.co/inclusionAI)
- GGUF conversion and quantization by [NANI-Nithin](https://huggingface.co/NANI-Nithin)
- GGUF tooling by the [llama.cpp](https://github.com/ggml-org/llama.cpp) community

## Disclaimer

This is a community GGUF conversion and is not an official release by inclusionAI. Please report conversion, loading, or compatibility issues in this repository’s Discussions section.

## Reproducibility

This repository was generated with a BailingMoeV3-enabled llama.cpp fork.
The exact source checkout checkpoint is recorded below:

```json
{
  "stage": "01_checkout_bailing_llama",
  "status": "complete",
  "timestamp_utc": "2026-08-11T10:36:13.114546+00:00",
  "model": "inclusionAI/Ling-3.0-tiny",
  "llama_repo": "https://github.com/aetherbird/llama.cpp.git",
  "llama_branch": "bailingmoe3-support",
  "repo_dir": "/mnt/ling/src/llama.cpp",
  "commit": "3a0124fa8c20356ed5e6bf0c0ebae1566d6f49c1"
}
```

## Files

- `F16`: Conversion baseline.
- `Q4_K_M`: General local-inference default.
- `Q5_K_M`, `Q6_K`, `Q8_0`: Higher-fidelity variants.
- `IQ*`: Importance-matrix variants, generated only when supported by the pinned quantizer.

Use a Ling/BailingMoeV3-compatible runtime to load these files.