Instructions to use KikoCis/AMALIA-9B-0626-SFT-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Use Docker

docker model run hf.co/KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KikoCis/AMALIA-9B-0626-SFT-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KikoCis/AMALIA-9B-0626-SFT-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Ollama
How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with Ollama:
```
ollama run hf.co/KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
```

Unsloth Studio

How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KikoCis/AMALIA-9B-0626-SFT-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KikoCis/AMALIA-9B-0626-SFT-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for KikoCis/AMALIA-9B-0626-SFT-GGUF to start chatting

Docker Model Runner
How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with Docker Model Runner:
```
docker model run hf.co/KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M
```

Lemonade

How to use KikoCis/AMALIA-9B-0626-SFT-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull KikoCis/AMALIA-9B-0626-SFT-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.AMALIA-9B-0626-SFT-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

AMALIA-9B-0626-SFT-GGUF / README.md

KikoCis

card + metrics + charts + imatrix + Modelfiles

85b16f1 verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

10.7 kB

	---
	license: apache-2.0
	base_model:
	- amalia-llm/AMALIA-9B-0626-SFT
	base_model_relation: quantized
	library_name: llama.cpp
	pipeline_tag: text-generation
	tags:
	- gguf
	- quantized
	- llama-cpp
	- imatrix
	- llama
	- portuguese
	- european-portuguese
	- kld-measured
	- conversational
	- endpoints_compatible
	language:
	- pt
	---

	![banner](banner.png)

	<div style="border:2px solid currentColor; font-family:ui-monospace,'SF Mono','Cascadia Mono',Consolas,monospace;">
	<div style="border-bottom:1px solid currentColor; padding:6px 12px; font-size:11px; letter-spacing:3px; text-transform:uppercase; opacity:0.7; text-align:center;">KIKOCIS // EUROPEAN-PORTUGUESE LLM // IMATRIX GGUF + KLD</div>
	<div style="padding:14px; display:flex; flex-wrap:wrap; align-items:center; justify-content:center; gap:18px;">
	<pre style="margin:0; font-size:9px; line-height:1.15;">
	╔═══════════════╗
	║ A M A L I A ║ pt-PT
	╚═══════════════╝
	┌──┐ ┌──┐ ┌──┐ ┌──┐
	│Q3│ │Q4│ │Q5│ │Q8│ ← ladder
	└──┘ └──┘ └──┘ └──┘
	●────●────●────●
	KLD vs F16
	</pre>
	<div style="text-align:center;">
	<div style="font-size:23px; font-weight:800; letter-spacing:1px;">AMALIA-9B · GGUF</div>
	<div style="font-size:12.5px; letter-spacing:1px; opacity:0.8; margin-top:5px;"><span style="white-space:nowrap;">llama · 9B</span> · <span style="white-space:nowrap;">32K ctx</span> · <span style="white-space:nowrap;">imatrix (pt)</span> · <span style="white-space:nowrap;">KLD-measured</span></div>
	</div>
	</div>
	<table style="display:table; table-layout:fixed; width:100%; margin:0; border-collapse:collapse; font-size:12px;">
	<tr>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">FORMAT</div><div style="font-weight:700;">GGUF (imatrix)</div></td>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">SIZES</div><div style="font-weight:700;">3.1 – 9.1 GB</div></td>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">ARCH</div><div style="font-weight:700;">Llama · 42L</div></td>
	<td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">CONTEXT</div><div style="font-weight:700;">32768</div></td>
	</tr>
	<tr>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">IMATRIX</div><div style="font-weight:700;">pt-PT corpus</div></td>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">VALIDATION</div><div style="font-weight:700;">KLD vs F16</div></td>
	<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">LANGUAGE</div><div style="font-weight:700;">Português (EU)</div></td>
	<td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">LICENSE</div><div style="font-weight:700;">Apache-2.0</div></td>
	</tr>
	</table>
	</div>

	# AMALIA-9B-0626-SFT — GGUF (imatrix + KLD)
	> imatrix-quantized GGUFs of AMALIA-9B, an open European-Portuguese instruction model — one of the first open LLMs developed in Portugal. Runs from ~3.1 GB (IQ2_M) to ~9.1 GB (Q8_0). Calibrated (imatrix) on Portuguese text, and every quant is measured against the full-precision model with KLD so you can pick a size with the fidelity cost in front of you. Credit: this is the AMALIA-LLM project's model — [`amalia-llm/AMALIA-9B-0626-SFT`](https://huggingface.co/amalia-llm/AMALIA-9B-0626-SFT); ours is the quant ladder + metrics.

	## ✅ Recommended files
	\| Use case \| File \| Notes \|
	\|---\|---\|---\|
	\| ⭐ Best value \| `amalia-9b-Q5_K_M.gguf` \| The pick — lower KLD than Q6_K at 1 GB less (0.051 vs 0.078). ~6.1 GB. \|
	\| Safe default \| `amalia-9b-Q4_K_M.gguf` \| Standard K-quant, ~5.2 GB; noticeably less faithful than Q5 (KLD 0.22 vs 0.05). \|
	\| Max fidelity \| `amalia-9b-Q8_0.gguf` \| Lowest KLD (0.006) — near-lossless. ~9.1 GB. \|
	\| Tightest RAM \| `amalia-9b-Q3_K_M.gguf` \| ~4.2 GB — runs on 8 GB machines, still coherent. \|
	\| Smallest (aggressive) \| `amalia-9b-IQ2_M.gguf` \| ~3.1 GB — fits very tight RAM, at a real quality cost (top-1 61%). \|

	## 📦 Files (the full ladder)
	\| Quant \| Bits \| File size \| Notes \|
	\|---\|---\|---:\|---\|
	\| IQ2_M \| ~2.7 \| ~3.1 GB \| Smallest — aggressive i-quant, real quality trade-off. \|
	\| Q3_K_M \| 3 \| ~4.2 GB \| Smallest comfortable quant. \|
	\| IQ4_XS \| ~4.3 \| ~4.7 GB \| Compact i-quant (imatrix). \|
	\| Q4_K_M \| 4 \| ~5.2 GB \| Standard K-quant. \|
	\| Q5_K_M \| 5 \| ~6.1 GB \| ⭐ Best quality/size. \|
	\| Q6_K \| 6 \| ~7.0 GB \| Near-lossless. \|
	\| Q8_0 \| 8 \| ~9.1 GB \| Highest precision. \|
	<!-- K-quants (Q_K) = solid general quants. I-quants (IQ) = smaller, need an imatrix (shipped). Higher number = more bits = closer to the original + bigger. -->

	## 📊 Metrics — objective quality vs the F16 reference
	KLD (Kullback–Leibler divergence, nats) measures how far each quant's output distribution drifts from the full-precision model — the gold-standard fidelity metric (lower = closer). Top-1 match = how often the quant's top token agrees with F16. Measured with `llama-perplexity --kl-divergence` over a Portuguese corpus at ctx 2048, vs the F16 GGUF.

	\| Model \| Size GB \| PPL \| ΔPPL vs F16 \| KLD mean \| KLD p95 \| Top-1 match \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| F16 (reference) \| 17.1 \| 20.04 \| 0.000 \| 0.0000 \| 0.0000 \| 100.0% \|
	\| Q8_0 \| 9.06 \| 20.20 \| +0.177 \| 0.0057 \| 0.0198 \| 96.38% \|
	\| Q6_K \| 7.00 \| 19.93 \| -0.091 \| 0.0779 \| 0.1740 \| 89.06% \|
	\| Q5_K_M \| 6.08 \| 19.80 \| -0.227 \| 0.0511 \| 0.1747 \| 88.89% \|
	\| Q4_K_M \| 5.20 \| 20.23 \| +0.203 \| 0.2230 \| 0.7429 \| 81.39% \|
	\| IQ4_XS \| 4.70 \| 20.78 \| +0.757 \| 0.2173 \| 0.7600 \| 80.55% \|
	\| Q3_K_M \| 4.25 \| 22.24 \| +2.215 \| 0.3258 \| 1.1486 \| 73.58% \|
	\| IQ2_M \| 3.11 \| 21.99 \| +1.968 \| 0.8211 \| 2.9052 \| 60.69% \|

	<sub>Full per-quant reports in [`reports/`](reports); machine-readable summary in [`metrics/quant-summary.csv`](metrics/quant-summary.csv); SHA-256 of every file in [`reports/artifact-sha256sums.txt`](reports/artifact-sha256sums.txt). Note: this is an instruction-tuned model measured on raw text, so the absolute PPL is higher than a base model's — what matters here is the relative KLD/top-1 across quants.</sub>

	> Why does Q5_K_M beat Q6_K on KLD here? It's real and reproducible for this model: Q5_K_M's imatrix-weighted bit allocation is very effective, so it lands closer to F16 on mean-KLD than Q6_K (which stays marginally ahead on top-1). Both are near-lossless — so Q5_K_M is the better value, and Q8_0 is the true fidelity ceiling.

	> A note on i-quants: we tested the i-quants too. For this model the K-quants win at low bit — e.g. IQ3_M came out worse than Q3_K_M on KLD, so it isn't shipped. IQ2_M is included as the smallest option for very tight RAM, but with an honest quality cost (top-1 61%); if you can spare the space, Q3_K_M is a big step up.

	### 📈 Charts
	![kld vs size](metrics/chart-kld-vs-size.png)
	![top-1 match](metrics/chart-top1.png)

	## 🧮 Will it fit? (RAM/VRAM cheat-sheet)
	Memory ≈ weights + KV-cache (KV grows with context). Rough guide:
	\| you have \| comfortable quant \| context \|
	\|---\|---\|---\|
	\| 6 GB \| IQ2_M / Q3_K_M \| ~8–16K \|
	\| 8 GB \| Q3_K_M / Q4_K_M \| ~16–32K \|
	\| 12 GB \| Q5_K_M / Q6_K \| ~32K \|
	\| 16 GB+ \| Q6_K / Q8_0 \| ~32K (native) \|

	## 🚀 How to run it
	```bash
	# ollama (pulls straight from this repo)
	ollama run hf.co/KikoCis/AMALIA-9B-0626-SFT-GGUF:Q5_K_M

	# llama.cpp
	llama-server -m amalia-9b-Q5_K_M.gguf -c 32768 --jinja
	llama-cli -m amalia-9b-Q5_K_M.gguf -p "Explica em português o que é a saudade." -ngl 99
	```
	Recommended sampling: temperature ~0.7, top_p ~0.9. It's a chat/instruct model (uses its built-in chat template) — prompt it in European Portuguese for best results.

	## ⚠️ Good to know
	- Strengths: European-Portuguese fluency and instruction-following; one of the first open LLMs built in Portugal.
	- Limits: specialised for Portuguese — not a coding/agentic model, and not tuned for English. Low-bit quants (IQ2/Q3) trade real fidelity for size (see KLD).
	- Absolute perplexity looks high only because it's measured on raw text with an instruct model; the KLD table is the fair quant-quality signal.

	## 📊 Evaluation methodology (how the numbers were measured)
	- What: quantization fidelity vs the F16 GGUF — not a task benchmark. `llama-perplexity --kl-divergence` (KLD mean/p95/max, ΔPPL, top-1 agreement).
	- Corpus: European-Portuguese text (Wikipedia + diverse), ctx 2048, same corpus family used for the imatrix calibration.
	- Reference: the model's own F16 GGUF (KLD = 0 by definition).
	- Date: 2026-07. Caveat: relative fidelity ranking across quants of this model; not comparable across different models/corpora.

	## 🔁 Provenance & reproducibility
	- Scripts: [`scripts/`](scripts) — exact convert → imatrix → quant → KLD commands.
	- imatrix: [`amalia-9b-pt.imatrix`](amalia-9b-pt.imatrix) — importance matrix (Portuguese calibration) used for the IQ/K quants.
	- Checksums: [`reports/artifact-sha256sums.txt`](reports/artifact-sha256sums.txt) — `shasum -a 256 -c` to verify.
	- Source: [`amalia-llm/AMALIA-9B-0626-SFT`](https://huggingface.co/amalia-llm/AMALIA-9B-0626-SFT) — weights unmodified (faithful quantization).

	## 🗒️ Changelog
	- 2026-07 v1: imatrix GGUF ladder (IQ2_M → Q8_0) + KLD/PPL metrics, Portuguese-calibrated.

	## 📚 Credit & license
	Model, weights, training data: © the AMALIA-LLM project — [model](https://huggingface.co/amalia-llm/AMALIA-9B-0626-SFT) · [website](https://amaliallm.pt/) · [paper (PROPOR 2026)](https://aclanthology.org/2026.propor-1.38/). Quant ladder + imatrix (pt) + KLD/PPL metrics: KikoCis. Apache-2.0 (same as upstream). No weights modified.