# Qwen3.8-27B-ABLITERATED-GGUF deployment

This kit serves the public, ungated GGUF release with a current `llama-server` build. The default is `Q4_K_M`, 16,384 tokens of context, full GPU offload, and text-only operation.

## Requirements

- A recent build of [llama.cpp](https://github.com/ggml-org/llama.cpp) with `llama-server`
- Python and the Hugging Face CLI: `pip install -U huggingface_hub`
- Enough combined RAM/VRAM for the selected quant, context state, compute buffers, and optional vision projector

The architecture advertises 262,144 tokens, but context allocation is a deployment choice. Start at 16K, measure memory at the desired concurrency, and scale deliberately.

## One-command launch

```bash
chmod +x deploy/serve.sh
./deploy/serve.sh
```

The server listens on port 8080 and exposes the OpenAI-compatible API. The public repository requires no Hugging Face token.

## Select a quant

```bash
QUANT=Q5_K_M ./deploy/serve.sh
```

Accepted standard ladder values are:

```text
Q2_K Q3_K_S Q3_K_M Q4_K_S Q4_K_M Q5_K_S Q5_K_M Q6_K Q8_0
```

`Q4_K_M` is the recommended starting point. No IQ/IK or importance-matrix quants are part of this release.

## Enable vision and video inputs

```bash
ENABLE_VISION=1 ./deploy/serve.sh
```

This downloads and loads the compact `Q8_0` projector. Select the full projector with:

```bash
ENABLE_VISION=1 MMPROJ_TYPE=F16 ./deploy/serve.sh
```

## CPU-only and hybrid inference

CPU-only:

```bash
GPU_LAYERS=0 ./deploy/serve.sh
```

For hybrid inference, choose a positive `GPU_LAYERS` value that fits the available VRAM. Model weights not offloaded to the GPU remain in system RAM.

## Context and network settings

```bash
CTX_SIZE=32768 HOST=127.0.0.1 PORT=8080 ./deploy/serve.sh
```

Bind to a private interface or place the endpoint behind authenticated infrastructure for shared use. The model and its embedded prompt are not an access-control boundary.

## Direct llama-server launch

```bash
llama-server \
  -m Qwen3.8-27B-ABLITERATED-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-ABLITERATED-Q8_0.gguf \
  -ngl 999 -fa on --jinja \
  --host 0.0.0.0 --port 8080 -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 20
```

Keep `--jinja` enabled: the GGUF contains the release's default Blackfrost system template. A client-supplied system message is additive to that embedded default.

## API smoke test

```bash
curl http://127.0.0.1:8080/health

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-ABLITERATED",
    "messages": [{"role": "user", "content": "Reply with exactly READY and nothing else."}],
    "temperature": 0,
    "max_tokens": 64
  }'
```

Reasoning-capable responses may include `reasoning_content` separately from final `content`. Allow enough output tokens for workloads that use the reasoning channel.

## Operational checklist

- Pin a tested llama.cpp build in production.
- Record quant, context, sampler, template, and projector in benchmark reports.
- Authenticate shared endpoints and log tool execution independently.
- Sandbox code and file tools and grant them least-privilege credentials.
- Test structured output, tools, multimodal inputs, and long context separately before relying on them.