Instructions to use QuantTrio/GLM-4.7-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/GLM-4.7-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantTrio/GLM-4.7-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuantTrio/GLM-4.7-AWQ")
model = AutoModelForCausalLM.from_pretrained("QuantTrio/GLM-4.7-AWQ", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/GLM-4.7-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/GLM-4.7-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-4.7-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantTrio/GLM-4.7-AWQ

SGLang

How to use QuantTrio/GLM-4.7-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/GLM-4.7-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-4.7-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/GLM-4.7-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-4.7-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuantTrio/GLM-4.7-AWQ with Docker Model Runner:
```
docker model run hf.co/QuantTrio/GLM-4.7-AWQ
```

s-yanev commited on May 15

Commit

f46b7c8

verified ·

1 Parent(s): cc0298a

Revert remplate

Browse files

Reverting back the changes of the jinja template from this PR -https://huggingface.co/QuantTrio/GLM-4.7-AWQ/discussions/7
Sadly the model is breaking when structured output is requested and thinking is enabled. Making the model to only produce thinking block.
If structured output is needed, best will be to put it in the system prompt - this is also what GLM suggest as well.
Sorry for the confusion.

Files changed (1) hide show

chat_template.jinja +2 -27

chat_template.jinja CHANGED Viewed

@@ -1,17 +1,14 @@
 [gMASK]<sop>
-{%- if tools and not (structured_output is defined and structured_output.enabled) -%}
 <|system|>
 # Tools
 You may call one or more functions to assist with the user query.
 You are provided with function signatures within <tools></tools> XML tags:
 <tools>
 {% for tool in tools %}
 {{ tool | tojson(ensure_ascii=False) }}
 {% endfor %}
 </tools>
 For each function call, output the function name and arguments within the following XML format:
 <tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
 {%- macro visible_text(content) -%}
@@ -78,29 +75,7 @@ For each function call, output the function name and arguments within the follow
 <tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
 {% endif -%}
 {%- elif m.role == 'system' -%}
-<|system|>
-{%- if structured_output is defined and structured_output.enabled and loop.index0 == 0 -%}
-# Structured Output
-You are a structured output engine.
-Output ONLY valid JSON. Follow the JSON schema exactly.
-Do not output:
-- markdown
-- prose
-- explanations
-- XML
-- comments
-- chain-of-thought
-JSON Schema:
-{{ structured_output.schema | tojson(indent=2) }}
-{{ visible_text(m.content) }}
-{%- else -%}
-{{ visible_text(m.content) }}
-{%- endif -%}
 {%- endif -%}
 {%- endfor -%}
 {%- if add_generation_prompt -%}

 [gMASK]<sop>
+{%- if tools -%}
 <|system|>
 # Tools
 You may call one or more functions to assist with the user query.
 You are provided with function signatures within <tools></tools> XML tags:
 <tools>
 {% for tool in tools %}
 {{ tool | tojson(ensure_ascii=False) }}
 {% endfor %}
 </tools>
 For each function call, output the function name and arguments within the following XML format:
 <tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
 {%- macro visible_text(content) -%}
 <tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
 {% endif -%}
 {%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
 {%- endif -%}
 {%- endfor -%}
 {%- if add_generation_prompt -%}