---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- gguf
- context-1
- chroma
- moe
- qwen
- llama.cpp
- quantized
base_model: chromadb/context-1
---

# Context-1 GGUF Quantizations

GGUF quantized versions of [chromadb/context-1](https://huggingface.co/chromadb/context-1), converted for inference with [llama.cpp](https://github.com/ggml-org/llama.cpp), [LM Studio](https://lmstudio.ai/), and other GGUF-compatible engines.

## About Context-1

**Context-1** is a 20.9B parameter Mixture-of-Experts (MoE) causal language model developed by [Chroma](https://www.trychroma.com/). It uses the `GptOssForCausalLM` architecture with 32 experts and 4 active per token, providing strong performance with efficient inference.

| Detail | Value |
|--------|-------|
| **Architecture** | GptOssForCausalLM (MoE) |
| **Total Parameters** | ~20.9B |
| **Active Parameters** | ~3B per token (4 of 32 experts) |
| **Hidden Size** | 2880 |
| **License** | Apache-2.0 |

## Quantization Details

Quantized from the F16 safetensors in the original repository using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-quantize` tool, running on NVIDIA H100 via [Modal](https://modal.com/).

| File | Format | Size |
|------|--------|------|
| `context-1-Q4_K_M.gguf` | Q4_K_M | 15.81 GB |

More quantization levels (Q3, Q5, Q8, etc.) may be added in the future.

## Usage

### llama.cpp

```bash
# Download
huggingface-cli download nicoism/context-1-GGUF context-1-Q4_K_M.gguf --local-dir .

# Run
./llama-cli -m context-1-Q4_K_M.gguf -p "Your prompt here" -ngl 99
```

### LM Studio

Search for `nicoism/context-1-GGUF` in LM Studio's model browser and download the desired quantization.

### Python (llama-cpp-python)

```python
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="nicoism/context-1-GGUF",
    filename="context-1-Q4_K_M.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)
```

## Chat Template

This model uses a custom chat template based on the OpenAI/Oss architecture with support for multi-channel output (analysis, commentary, final), tool calling, and built-in browser/python tools. The template is embedded in the GGUF files.

Format overview:

```
<|start|>system<|message|>...<|end|>
<|start|>developer<|message|>...<|end|>
<|start|>user<|message|>...<|end|>
<|start|>assistant<|channel|>final<|message|>...<|end|>
```

For the full template, see [`chat_template.jinja`](https://huggingface.co/chromadb/context-1/blob/main/chat_template.jinja) in the original repository.

## Limitations

- GGUF quantization introduces minor quality degradation compared to the original F16 weights. Q4_K_M provides a good balance of quality and size.
- This model inherits any biases and limitations from the base model.
- Requires a GPU or sufficient RAM for inference (16 GB+ for Q4_K_M).

## License

Apache-2.0 — same as the [original model](https://huggingface.co/chromadb/context-1).

## Acknowledgements

- [Chroma](https://www.trychroma.com/) for the original model
- [llama.cpp](https://github.com/ggml-org/llama.cpp) for the quantization tooling
- [Modal](https://modal.com/) for the GPU compute infrastructure