zenmagnets/Nemotron-3-Embed-1B-Q4_K_M-GGUF

Nemotron-3-Embed-1B Q4_K_M GGUF

An independently converted and quantized GGUF of NVIDIA's nvidia/Nemotron-3-Embed-1B-BF16, prepared for local embedding inference in LM Studio and llama.cpp-compatible runtimes.

This repository is not an official NVIDIA release and is not affiliated with or endorsed by NVIDIA.

File

File	Quantization	Size	SHA-256
nemotron-3-embed-1b-q4_k_m.gguf	Q4_K_M	749,352,096 bytes (714.6 MiB)	9a74166f51dbc280073748fa199bea49283bd21f7f9280f2dec2b4d975ddfd1d

The model produces 2,048-dimensional, L2-normalized embeddings. Its GGUF metadata declares a 262,144-token maximum context. The release was functionally tested at a 4,096-token context; very large contexts were not validated and require substantially more memory.

Use with LM Studio

Download the GGUF from Files and versions, then drag it into LM Studio or place it in LM Studio's models directory. LM Studio should classify it as an embedding model.

Load it with a 4,096-token context and full GPU offload, start the local server, and call the OpenAI-compatible embeddings endpoint:

curl http://127.0.0.1:1234/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nemotron-3-embed-1b-q4",
    "input": [
      "query: What is retrieval-augmented generation?",
      "passage: Retrieval-augmented generation adds retrieved documents to a model prompt."
    ]
  }'

The exact model identifier can differ if LM Studio assigns another load name; check http://127.0.0.1:1234/v1/models when needed.

Retrieval format

Use the prefixes specified by NVIDIA:

• Queries: query: <text>
• Documents: passage: <text>

Embeddings are normalized, so cosine similarity and dot product give equivalent rankings (within normal floating-point tolerance).

Conversion provenance

• Upstream repository: nvidia/Nemotron-3-Embed-1B-BF16
• Exact upstream revision: 0677b2025cbc37daf92d7b9c7a225de8cfbc5b0b
• Conversion and quantization: llama.cpp build b10015, commit 12127defda4f41b7679cb2477a4b0d65ee6a0c8f
• Conversion path: upstream BF16 Safetensors → F16 GGUF → Q4_K_M GGUF
• Importance matrix: none
• Output GGUF version: 3
• Quantization version: 2

The conversion was mechanical. The model was not fine-tuned or otherwise trained. See QUANTIZATION.md for the reproducibility record and embedded metadata summary.

Validation

Validated on an Apple M3 Max with LM Studio 0.4.19 Build 2 and its llama.cpp runtime 2.25.2, using full GPU offload and a 4,096-token context.

Check	Result
Output dimensions	2,048
L2 norm (three test vectors)	1.00000002 / 1.00000001 / 0.99999997
Relevant query/passage cosine	0.654999
Unrelated query/passage cosine	0.044103
Median latency, one input	30.46 ms
Median batch time, 32 inputs	974.98 ms
Median batch throughput	32.82 texts/s

Latency includes local HTTP and JSON serialization. These numbers are a runtime smoke test, not a retrieval-quality evaluation. Hardware, text length, context settings, and runtime versions affect performance.

Limitations

• Q4_K_M is lossy and can change retrieval scores or rankings relative to the upstream BF16 model.
• No MTEB/MMTEB accuracy evaluation was run on this quantization; consult NVIDIA's upstream model card for BF16 benchmark results.
• This model is intended for embeddings, retrieval, semantic search, and reranking pipelines. It is not a chat or text-generation model.
• Users remain responsible for evaluating model behavior, data handling, and suitability for their application.

License and attribution

The model materials are distributed under the OpenMDW License Agreement 1.1. The upstream model incorporates Ministral-3-3B-Instruct-2512 materials released under Apache 2.0.

This repository retains NVIDIA's LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md. By using or distributing the model, you agree to the applicable license terms. NVIDIA retains ownership of its names and trademarks; their use here is solely to identify the upstream model.

Upstream resources

• NVIDIA upstream model card
• NVIDIA Nemotron-3-Embed blog post
• Compact Language Models via Pruning and Knowledge Distillation
• MMTEB: Massive Multilingual Text Embedding Benchmark