--- license: apache-2.0 base_model: - BSC-LT/ALIA-40b-instruct-2601 base_model_relation: quantized library_name: llama.cpp pipeline_tag: text-generation tags: - gguf - quantized - llama-cpp - imatrix - llama - spanish - multilingual - european - sovereign-ai - kld-measured - conversational - endpoints_compatible language: - es - ca - gl - eu - en --- ![banner](banner.png)
KIKOCIS // EU-SOVEREIGN LLM // IMATRIX GGUF + KLD
   ╔═══════════════════╗
   ║   A L I A · 40B   ║  ES·CA·GL·EU
   ╚═══════════════════╝
   ┌───┐ ┌───┐ ┌───┐ ┌───┐
   │IQ2│ │Q3 │ │Q4 │ │Q8 │
   └───┘ └───┘ └───┘ └───┘
    ●─────●─────●─────●
       KLD vs Q8-ref
ALIA-40b · GGUF
llama · 46B · 160K ctx · imatrix (es) · KLD-measured
FORMAT
GGUF (imatrix)
SIZES
~13.5 – 40 GB
ARCH
Llama · 46B · 48L
CONTEXT
163840
IMATRIX
es corpus
VALIDATION
KLD vs Q8
LANGUAGES
ES·CA·GL·EU·EN
LICENSE
Apache-2.0
# ALIA-40b-instruct-2601 — GGUF (imatrix + KLD) > imatrix-quantized GGUFs of **ALIA-40b**, the Barcelona Supercomputing Center's **~46B sovereign LLM for Spain / the EU** (Spanish + Catalan, Galician, Basque + European languages), with a **160K** native context. Runs from **~13.5 GB** (IQ2_M) to **~40 GB** (Q8_0). Calibrated (imatrix) **on Spanish text**, with per-quant **KLD** fidelity vs the near-lossless Q8_0 reference. Credit: this is **BSC-LT's** model — [`BSC-LT/ALIA-40b-instruct-2601`](https://huggingface.co/BSC-LT/ALIA-40b-instruct-2601); ours is the quant ladder + metrics. ## ✅ Recommended files | Use case | File | Notes | |---|---|---| | Safe default | `ALIA-40b-instruct-2601-Q4_K_M.gguf` | The one most people want — good quality, ~23 GB (48 GB RAM). | | Strong quality/size | `ALIA-40b-instruct-2601-Q5_K_M.gguf` | Closer to the reference, ~27 GB. | | Runs on 32 GB | `ALIA-40b-instruct-2601-Q3_K_M.gguf` | ~19 GB — fits a 32 GB machine. | | Smallest (tight RAM) | `ALIA-40b-instruct-2601-IQ2_M.gguf` | ~13.5 GB i-quant — runs a 46B on 24–32 GB, at a real quality cost. | | Reference / max fidelity | `ALIA-40b-instruct-2601-Q8_0.gguf` | ~40 GB, near-lossless (the KLD reference). | ## 📦 Files (the ladder) | Quant | Bits | File size | RAM (approx) | Notes | |---|---|---:|---:|---| | IQ2_M | ~2.7 | ~13.5 GB | 24–32 GB | Smallest — aggressive i-quant. | | Q3_K_M | 3 | ~18.7 GB | 32 GB | Fits a 32 GB machine. | | Q4_K_M | 4 | ~22.9 GB | 48 GB | Safe default. | | Q5_K_M | 5 | ~26.8 GB | 48–64 GB | Strong quality/size. | | Q8_0 | 8 | ~40 GB | 64 GB+ | Reference, near-lossless. | ## 📊 Metrics — fidelity vs the Q8_0 reference **KLD** (Kullback–Leibler divergence, nats) measures how far each quant's output distribution drifts from the reference — lower = closer. **Top-1 match** = how often the quant's top token agrees with the reference. Measured with `llama-perplexity --kl-divergence` over a **Spanish** corpus at ctx 512. > **Why the reference is Q8_0, not F16?** ALIA-40b's F16 is ~81 GB and does not fit this machine's GPU (Metal). Q8_0 is near-lossless (its own KLD vs F16 would be ~0.005), so it's a faithful stand-in reference for measuring how much the smaller quants drift. KLD values here are therefore **relative to Q8_0** (Q8_0 = 0 by definition). | Model | Size GB | KLD mean | KLD p95 | KLD max | Top-1 match | |---|---:|---:|---:|---:|---:| | **Q8_0 (reference)** | 40.0 | 0.0000 | 0.0000 | 0.0000 | 100.0% | | Q5_K_M | 26.78 | 0.0098 | 0.0408 | 4.893 | 96.61% | | Q4_K_M | 22.90 | 0.0337 | 0.1505 | 4.105 | 93.94% | | Q3_K_M | 18.67 | 0.1114 | 0.5419 | 7.904 | 88.89% | | IQ2_M | 13.54 | 0.3799 | 1.9178 | 12.259 | 77.82% | Full per-quant reports in [`reports/`](reports); machine-readable summary in [`metrics/quant-summary.csv`](metrics/quant-summary.csv); SHA-256 of every file in [`reports/artifact-sha256sums.txt`](reports/artifact-sha256sums.txt). ### 📈 Charts ![kld vs size](metrics/chart-kld-vs-size.png) ![top-1 match](metrics/chart-top1.png) ## 🧮 Will it fit? (RAM cheat-sheet) A 46B is memory-hungry; add KV-cache on top (it grows with context — 160K is a lot). | you have | quant | context | |---|---|---| | 24 GB | IQ2_M | ~8–16K | | 32 GB | Q3_K_M / IQ2_M | ~16–32K | | 48 GB | Q4_K_M / Q5_K_M | ~32–64K | | 64 GB+ | Q8_0 | large (up to 160K with room) | ## 🚀 How to run it ```bash # ollama ollama run hf.co/KikoCis/ALIA-40b-instruct-2601-GGUF:Q4_K_M # llama.cpp llama-server -m ALIA-40b-instruct-2601-Q4_K_M.gguf -c 32768 --jinja -ngl 99 ``` **Recommended sampling**: temperature ~0.7, top_p ~0.9. Chat/instruct model (uses its built-in template) — great for **Spanish and the co-official languages** (Catalan, Galician, Basque) + European languages. ## ⚠️ Good to know - **Strengths**: a genuinely **sovereign, EU-built** 46B — strong Spanish + co-official + European multilinguality, 160K context, permissive licence. - **Limits**: it's a **46B** — even the smallest quant needs ~24 GB RAM; IQ2_M trades real quality for size. Not a specialised coding model. - KLD is measured **vs Q8_0** (F16 doesn't fit this GPU) — see the note above. ## 📊 Evaluation methodology - **What**: quantization **fidelity** vs the Q8_0 reference — `llama-perplexity --kl-divergence` (KLD mean/p95/max, ΔPPL, top-1 agreement). - **Corpus**: Spanish text, ctx 512, same corpus used for imatrix calibration. - **Reference**: Q8_0 (near-lossless stand-in for F16, which is too large for this GPU). - **Date**: 2026-07. *Caveat: relative fidelity ranking across quants of this model.* ## 🔁 Provenance & reproducibility - **Scripts**: [`scripts/`](scripts) — exact convert → Q8 → imatrix → quant → KLD commands. - **imatrix**: [`alia-40b-es.imatrix`](alia-40b-es.imatrix) — importance matrix (Spanish calibration). - **Checksums**: [`reports/artifact-sha256sums.txt`](reports/artifact-sha256sums.txt). - **Source**: [`BSC-LT/ALIA-40b-instruct-2601`](https://huggingface.co/BSC-LT/ALIA-40b-instruct-2601) — weights **unmodified** (faithful quantization). ## 🗒️ Changelog - 2026-07 v1: initial imatrix GGUF ladder (IQ2_M → Q8_0) + KLD metrics, Spanish-calibrated. ## 📚 Credit & license Model, weights, training: **© Barcelona Supercomputing Center — the ALIA / langtech-bsc project** ([model](https://huggingface.co/BSC-LT/ALIA-40b-instruct-2601) · [ALIA-kit](https://langtech-bsc.gitbook.io/alia-kit)). Quant ladder + imatrix (es) + KLD metrics: KikoCis. **Apache-2.0** (same as upstream). No weights modified.