Update README with FP8 variant and formats table
Browse files
README.md
CHANGED
|
@@ -29,7 +29,20 @@ This is the successor to [Carnice-MoE-35B-A3B](https://huggingface.co/samuelcard
|
|
| 29 |
|
| 30 |
Training methodology adapted from **[kai-os/Carnice-9b](https://huggingface.co/kai-os/Carnice-9b)** — same two-stage approach and datasets, applied to the larger MoE architecture. Key inspiration: training on actual Hermes Agent execution traces for native agentic behavior.
|
| 31 |
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
## Model Details
|
| 35 |
|
|
|
|
| 29 |
|
| 30 |
Training methodology adapted from **[kai-os/Carnice-9b](https://huggingface.co/kai-os/Carnice-9b)** — same two-stage approach and datasets, applied to the larger MoE architecture. Key inspiration: training on actual Hermes Agent execution traces for native agentic behavior.
|
| 31 |
|
| 32 |
+
## Available Formats
|
| 33 |
+
|
| 34 |
+
| Format | Size | Location | Use Case |
|
| 35 |
+
|---|---|---|---|
|
| 36 |
+
| **BF16 SafeTensors** | 67 GB | Root | Full precision, Transformers / vLLM |
|
| 37 |
+
| **FP8 Dynamic** | 34 GB | `fp8/` | vLLM optimized, ~2x faster inference |
|
| 38 |
+
| **GGUF** | 19-65 GB | [GGUF repo](https://huggingface.co/samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B-GGUF) | llama.cpp, Ollama, LM Studio |
|
| 39 |
+
|
| 40 |
+
### FP8 Usage (vLLM)
|
| 41 |
+
|
| 42 |
+
```bash
|
| 43 |
+
# Clone the repo and point vLLM to the fp8/ subfolder
|
| 44 |
+
vllm serve samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B --quantization fp8 --dtype auto
|
| 45 |
+
```
|
| 46 |
|
| 47 |
## Model Details
|
| 48 |
|