Instructions to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="magiccodingman/Qwen3.8-27B-heretic-ara-fp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-fp8")
model = AutoModelForMultimodalLM.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-fp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3.8-27B-heretic-ara-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-fp8

SGLang

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-fp8
```

Qwen3.8-27B-heretic-ara-fp8 / FP8_VALIDATION.md

magiccodingman

Upload folder using huggingface_hub

ae4c4a3 verified 4 days ago

preview code

Raw

History Blame Contribute Delete

2.59 kB

Qwen3.8-27B Heretic ARA FP8 validation

Result

PASS

Source: heretic-org/Qwen3.8-27B-heretic-ara
FP8 derivative: magiccodingman/Qwen3.8-27B-heretic-ara-fp8
Format: native Qwen fine-grained FP8 (float8_e4m3fn) weights
Weight scaling: 128 x 128 blocks with FP32 weight_scale_inv tensors
Activation scheme in model metadata: dynamic FP8
Quantized matrix tensors: 407
Tensor payload: 30,869,678,304 bytes (30.87 GB decimal / 28.75 GiB)

Structural and numerical checks

Source tensor entries: 1,199
Candidate tensor entries: 1,606 (407 added scale tensors)
Non-quantized tensors preserved byte-for-byte: 792
Aggregate weight RMSE after dequantization: 0.00034144
Aggregate relative RMSE: 0.0264554
FP8 tensor type, scale type, positive scale values, block geometry, index closure, and safetensors headers: all passed

Forward-pass KLD check

Exact forward KL was measured as D_KL(P_BF16 || P_FP8) in FP32 across a diverse 12-prompt corpus. The two runs used the same tokenizer, BF16 compute, sequence settings, and device map.

Scored next-token positions: 638
Mean KL: 0.00315279
Median KL: 0.00185845
P95 KL: 0.0104304
P99 KL: 0.0222929
Maximum single-position KL: 0.0977299
BF16 reference NLL: 2.78304195
FP8 candidate NLL: 2.78553748
NLL delta: +0.00249553
Top-1 token agreement: 96.5517%
Acceptance gate: mean KL < 0.01
Result: PASS

The corpus covered code, mathematics, science, JSON/tool syntax, speculative-decoding concepts, Spanish, Arabic, proof writing, inference engineering, SQL, and FP8 technical reasoning.

Important hardware limitation

RTX 3090 (SM 8.6) cannot execute this model's native W8A8 FP8 path. Transformers therefore dequantized the stored FP8 weights to BF16 for these forward comparisons. The KLD result directly validates the stored weight quantization and shows no material degradation from the conversion. It does not benchmark native dynamic-FP8 activation kernels or FP8 KV cache behavior.

On the target Radeon Pro R9700 system, perform a short native-runtime smoke test using the intended serving engine and explicitly enable its FP8 KV-cache option. KV-cache precision is normally a runtime setting rather than baked into the weight checkpoint.

Provenance

The checkpoint layout and exclusion list were matched to the official Qwen/Qwen3.8-27B-FP8 native format. Conversion used Python 3.12.3, PyTorch 2.13.0+cu130, Transformers 5.14.1, and Safetensors 0.8.0. The source directory was not modified.