AmarettoLabs/amaretto-embed-148m-GGUF

amaretto-embed-148m — GGUF

GGUF builds of AmarettoLabs/amaretto-embed-148m for llama.cpp — a 148M-parameter EmbeddingGemma specialised for 8 Latin-script languages + code, for running text embeddings on edge / CPU / Jetson hardware.

For the full model description, benchmarks, and the vocabulary-slicing + distillation method, see the source model card and the code repository.

Not using llama.cpp? There are also ONNX builds for onnxruntime (fp32 exact, or 297 MB weight-only int8 at ≈0.999), and the source model itself for PyTorch / sentence-transformers.

Files

Quantized with llama.cpp. Fidelity is measured against the original PyTorch model — not against the f16 GGUF, so a subtly wrong conversion couldn't hide behind quants that merely agree with each other.

file	size	notes
amaretto-embed-148m-f16.gguf	293 MB	exact match to the source model at every length
amaretto-embed-148m-Q8_0.gguf	157 MB	near-lossless
amaretto-embed-148m-Q6_K.gguf	127 MB	recommended default
amaretto-embed-148m-Q5_K_M.gguf	113 MB	smallest; best for short/medium text

Fidelity is length-dependent, so it is reported per sequence length (cosine vs PyTorch, over a fixed prompt set spanning nearly the full 2048-token context):

tokens	f16	Q8_0	Q6_K	Q5_K_M
15	1.0000	0.9998	0.9990	0.9969
135	1.0000	0.9996	0.9982	0.9950
978	1.0000	0.9993	0.9970	0.9915
1822	1.0000	0.9991	0.9960	0.9902

The f16 build reproduces the PyTorch model exactly at every length, which verifies the conversion itself; every quant is then measured against that verified anchor.

Choosing a quant. Fidelity drops as sequences get longer, and faster the more aggressive the quantization. For long documents (≳1000 tokens) prefer Q6_K or higher; Q5_K_M is best suited to short and medium text. Q8_0 is near-lossless across the whole range.

All builds produce 768-dimensional embeddings (Matryoshka: truncate to 512/256/128 and re-normalize). The prompt set, reference vectors, and the CI checks that enforce these numbers live in the code repo.

Usage (llama.cpp)

This is an embedding model with mean pooling and task prefixes — include the prefix that matches your use (e.g. task: search result | query: for queries, title: none | text: for documents), exactly as with EmbeddingGemma.

# one embedding
llama-embedding -m amaretto-embed-148m-Q6_K.gguf \
  -p "task: search result | query: how do I sort a list in python?" \
  --pooling mean --embd-normalize 2

# many at once (one text per line in prompts.txt), JSON output
llama-embedding -m amaretto-embed-148m-Q6_K.gguf -f prompts.txt \
  --pooling mean --embd-normalize 2 --embd-output-format json

Also works through the llama-server embeddings endpoint (--embeddings --pooling mean).

Intended use & limitations

A general-purpose text-embedding model — retrieval, semantic search, RAG, clustering, classification, STS — for English, Spanish, Portuguese, French, German, Italian, Dutch, Polish, and source code.

Not multilingual. The vocabulary for non-target languages was removed; text in other scripts (Chinese, Japanese, Korean, Arabic, Cyrillic, Greek, Indic, Thai, …) degrades severely. Use the original google/embeddinggemma-300m for broad multilingual coverage.

Benchmarks (as % of EmbeddingGemma): ≥99.3% on retrieval and STS, ~98.7% on code retrieval, ~97.9% on classification, ~96% on long-document retrieval. Full tables on the source model card.

License — Gemma Terms of Use

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

These GGUF files are a format conversion of amaretto-embed-148m, itself a Gemma Model Derivative of google/embeddinggemma-300m. Your use, reproduction, and distribution are governed by the Gemma Terms of Use, a copy of which is distributed here in the LICENSE file. By using these files you accept those terms.

• Use restrictions (§3.2): you may not use these models in violation of the Gemma Prohibited Use Policy.
• If you redistribute these or a derivative, the Gemma Terms (§3.1) require you to pass on the use restrictions, include the LICENSE, mark modified files, and ship a NOTICE with the required Gemma notice.
• Trademarks (§4.2): not affiliated with, endorsed by, or sponsored by Google. "Gemma" / "EmbeddingGemma" are used descriptively to identify the upstream model.

Built by AmarettoLabs. Converted with llama.cpp.