Instructions to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k")
model = AutoModelForCausalLM.from_pretrained("jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k

SGLang

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k with Docker Model Runner:
```
docker model run hf.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k
```

Excellent Job! Question about the process.

by nevaback - opened 1 day ago

Discussion

nevaback

1 day ago

Nice job! I was able to run this stably on a 4 node Jetson Thor cluster using TRITON_MLA_SPARSE attention. Not a very conventional setup, but can get 6-8 t/sec decode.

What was the process to quantize this. It would be very cool to get this same thing done for mindlab-research/Macaron-V1-Coding-Venti which is based on GLM-5.2, but seems post training made it quite a bit stronger on coding and terminal calling. Have you heard of that model?

jarrelscy

Owner about 24 hours ago

Hi! Its very straightforward - essentially follows two recipes, the https://github.com/vahe1994/AQLM repo here to first generate a completely 2-bit version of the entire model, then uses standard REAP techniques to identify hot / cold experts, and instead of removing the cold experts, replaces them with their 2-bit versions from AQLM.

jarrelscy

Owner about 24 hours ago

I can do this one next. mindlab-research/Macaron-V1-Coding-Venti

nevaback

about 8 hours ago

What sort of hardware is required to run that process?

Oh, also, maybe mindlab-research/Macaron-V1-Venti has the same coding ability and better overall ability, so maybe it is better than doing the coding specific one. Also, I asked here https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4/discussions/4 about how hard it would be to put the vision in border on top of Macaron-V1-Venti. It seems promising.

jarrelscy

Owner about 1 hour ago

For the AQLM process, codebook initialization using k-means is cheap and can be done on CPU but won't lead to very good outputs. To improve the AQLM weights you need to finetune the codebook on traces that you care about. I chose coding, prose, some medical texts. Generating these traces needs you to run the full unquantized model so this is the floor in terms of hardware (ideally b200x8). I naively grafted the vision encoder from baseten onto the quantized model and it worked so you could try it with Macaron.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment