Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,105 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- gguf
+- context-1
+- chroma
+- moe
+- qwen
+- llama.cpp
+- quantized
+base_model: chromadb/context-1
+---
+
+# Context-1 GGUF Quantizations
+
+GGUF quantized versions of [chromadb/context-1](https://huggingface.co/chromadb/context-1), converted for inference with [llama.cpp](https://github.com/ggml-org/llama.cpp), [LM Studio](https://lmstudio.ai/), and other GGUF-compatible engines.
+
+## About Context-1
+
+**Context-1** is a 20.9B parameter Mixture-of-Experts (MoE) causal language model developed by [Chroma](https://www.trychroma.com/). It uses the `GptOssForCausalLM` architecture with 32 experts and 4 active per token, providing strong performance with efficient inference.
+
+| Detail | Value |
+|--------|-------|
+| **Architecture** | GptOssForCausalLM (MoE) |
+| **Total Parameters** | ~20.9B |
+| **Active Parameters** | ~3B per token (4 of 32 experts) |
+| **Hidden Size** | 2880 |
+| **License** | Apache-2.0 |
+
+## Quantization Details
+
+Quantized from the F16 safetensors in the original repository using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-quantize` tool, running on NVIDIA H100 via [Modal](https://modal.com/).
+
+| File | Format | Size |
+|------|--------|------|
+| `context-1-Q4_K_M.gguf` | Q4_K_M | 15.81 GB |
+
+More quantization levels (Q3, Q5, Q8, etc.) may be added in the future.
+
+## Usage
+
+### llama.cpp
+
+```bash
+# Download
+huggingface-cli download nicoism/context-1-GGUF context-1-Q4_K_M.gguf --local-dir .
+
+# Run
+./llama-cli -m context-1-Q4_K_M.gguf -p "Your prompt here" -ngl 99
+```
+
+### LM Studio
+
+Search for `nicoism/context-1-GGUF` in LM Studio's model browser and download the desired quantization.
+
+### Python (llama-cpp-python)
+
+```python
+from llama_cpp import Llama
+
+llm = Llama.from_pretrained(
+    repo_id="nicoism/context-1-GGUF",
+    filename="context-1-Q4_K_M.gguf",
+    n_gpu_layers=-1,
+)
+
+response = llm.create_chat_completion(
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+print(response)
+```
+
+## Chat Template
+
+This model uses a custom chat template based on the OpenAI/Oss architecture with support for multi-channel output (analysis, commentary, final), tool calling, and built-in browser/python tools. The template is embedded in the GGUF files.
+
+Format overview:
+
+```
+<|start|>system<|message|>...<|end|>
+<|start|>developer<|message|>...<|end|>
+<|start|>user<|message|>...<|end|>
+<|start|>assistant<|channel|>final<|message|>...<|end|>
+```
+
+For the full template, see [`chat_template.jinja`](https://huggingface.co/chromadb/context-1/blob/main/chat_template.jinja) in the original repository.
+
+## Limitations
+
+- GGUF quantization introduces minor quality degradation compared to the original F16 weights. Q4_K_M provides a good balance of quality and size.
+- This model inherits any biases and limitations from the base model.
+- Requires a GPU or sufficient RAM for inference (16 GB+ for Q4_K_M).
+
+## License
+
+Apache-2.0 — same as the [original model](https://huggingface.co/chromadb/context-1).
+
+## Acknowledgements
+
+- [Chroma](https://www.trychroma.com/) for the original model
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) for the quantization tooling
+- [Modal](https://modal.com/) for the GPU compute infrastructure