README.md

metadata

language:
  - en
license: apache-2.0
base_model: inclusionAI/Ling-3.0-tiny
tags:
  - gguf
  - llama.cpp
  - quantized
  - moe
  - bailingmoev3
  - hybrid-model
  - local-llm
  - text-generation
pipeline_tag: text-generation

Ling-3.0-tiny-GGUF

GGUF quantizations of inclusionAI/Ling-3.0-tiny, converted for use with compatible llama.cpp-based runtimes.

This repository includes a complete selection of standard K-quants and importance-matrix (IQ) quantizations, so you can choose the best balance of model size, speed, and output quality for your hardware.

Runtime compatibility: Ling-3.0-tiny uses the BailingMoeV3 / hybrid architecture. Use a runtime with explicit support for this architecture. Generic or older llama.cpp builds may not load these files correctly.

Available files

Quantization	Best for
F16	Highest-fidelity baseline; re-quantization and high-memory systems
Q8_0	Near-F16 quality with substantially lower memory use
Q6_K	High-quality local inference
Q5_K_M	Strong quality-to-size balance
Q5_K_S	Slightly smaller alternative to Q5_K_M
Q5_0	Legacy-style 5-bit option
Q4_K_M	Recommended default for most users
Q4_K_S	Smaller Q4 K-quant alternative
Q4_0	Compact legacy-style 4-bit option
IQ4_NL	High-quality importance-matrix 4-bit option
IQ4_XS	Compact importance-matrix 4-bit option
Q3_K_L	Higher-quality 3-bit K-quant
Q3_K_M	Balanced 3-bit K-quant
Q3_K_S	Smaller 3-bit K-quant
IQ3_M	Strong quality-per-GB option for constrained systems
IQ3_S	Smaller 3-bit IQ option
IQ3_XS	Very compact IQ 3-bit option
IQ3_XXS	Extremely compact IQ 3-bit option
Q2_K	Low-memory K-quant option
IQ2_M	Compact IQ quant with better quality potential than very-low-bit options
IQ2_S	Low-memory IQ option
IQ2_XS	Very small IQ option
IQ2_XXS	Extremely small IQ option
IQ1_M	Experimental ultra-low-memory option
IQ1_S	Smallest experimental option

Recommended downloads

Your priority	Recommended file
Best quality	Ling-3.0-tiny-F16.gguf
Near-original quality	Ling-3.0-tiny-Q8_0.gguf
High quality with lower memory use	Ling-3.0-tiny-Q6_K.gguf
Best general-purpose choice	Ling-3.0-tiny-Q4_K_M.gguf
Small but capable	Ling-3.0-tiny-IQ3_M.gguf
Tight VRAM / RAM budget	Ling-3.0-tiny-IQ2_M.gguf
Experimental minimum size	Ling-3.0-tiny-IQ1_S.gguf

For most users, start with Q4_K_M. If you have more RAM or VRAM, try Q5_K_M, Q6_K, or Q8_0. IQ quants can offer attractive quality-to-size trade-offs, but results and compatibility may vary by runtime and hardware.

Usage

Download one .gguf file, then run it with a compatible build of llama.cpp.

llama-cli \
  -m Ling-3.0-tiny-Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  -p "Write a concise explanation of retrieval-augmented generation."

-ngl 99 attempts to offload all supported layers to the GPU. Remove it or set -ngl 0 for CPU-only inference.

Important notes

• These files are quantized derivatives of the original model; output quality changes depending on the chosen quantization.
• Very low-bit quants, especially IQ1 and IQ2 variants, are intended for memory-constrained or experimental use and may noticeably reduce output quality.
• Use the original model’s license, terms, and usage requirements.
• Validate the selected quantization on your own workload before production use.

Conversion details

• Base model: inclusionAI/Ling-3.0-tiny
• Format: GGUF
• Conversion/runtime branch: BailingMoeV3-compatible llama.cpp fork
• Standard K-quants: generated from the F16 GGUF
• IQ quants: generated using an importance matrix calibrated on a text corpus

Credits

• Original model by inclusionAI
• GGUF conversion and quantization by NANI-Nithin
• GGUF tooling by the llama.cpp community

Disclaimer

This is a community GGUF conversion and is not an official release by inclusionAI. Please report conversion, loading, or compatibility issues in this repository’s Discussions section.

Reproducibility

This repository was generated with a BailingMoeV3-enabled llama.cpp fork. The exact source checkout checkpoint is recorded below:

{
  "stage": "01_checkout_bailing_llama",
  "status": "complete",
  "timestamp_utc": "2026-08-11T10:36:13.114546+00:00",
  "model": "inclusionAI/Ling-3.0-tiny",
  "llama_repo": "https://github.com/aetherbird/llama.cpp.git",
  "llama_branch": "bailingmoe3-support",
  "repo_dir": "/mnt/ling/src/llama.cpp",
  "commit": "3a0124fa8c20356ed5e6bf0c0ebae1566d6f49c1"
}

Files

• F16: Conversion baseline.
• Q4_K_M: General local-inference default.
• Q5_K_M, Q6_K, Q8_0: Higher-fidelity variants.
• IQ*: Importance-matrix variants, generated only when supported by the pinned quantizer.

Use a Ling/BailingMoeV3-compatible runtime to load these files.