cudabenchmarktest commited on
Commit
d8da57a
·
verified ·
1 Parent(s): 04a6d44

Fix Modelfile: add RENDERER/PARSER qwen3.5 for tool calling; relative GGUF path; remove hardcoded TEMPLATE override

Browse files
Files changed (1) hide show
  1. Modelfile +37 -10
Modelfile CHANGED
@@ -1,12 +1,39 @@
1
- FROM /home/roko/Documents/Projects/Adjacent/distill_qwen/outputs/gguf/qwen3.5-9b-qwen3.6-reasoning-distilled/qwen3.5-9b-qwen3.6-reasoning-distilled.q4km.gguf
2
- TEMPLATE """{{- if .System }}<|im_start|>system
3
- {{ .System }}<|im_end|>
4
- {{ end }}{{- range .Messages }}<|im_start|>{{ .Role }}
5
- {{ .Content }}<|im_end|>
6
- {{ end }}<|im_start|>assistant
7
- """
8
- PARAMETER stop "<|im_end|>"
9
- PARAMETER stop "<|im_start|>"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  PARAMETER temperature 0.6
11
  PARAMETER top_p 0.95
12
- SYSTEM """You are a reasoning model distilled from Qwen3.6-plus. Think step-by-step inside <think></think> tags, then provide a clear final answer."""
 
1
+ # Modelfile for Ollama — qwen3.5-9b-qwen3.6-reasoning-distilled
2
+ #
3
+ # CRITICAL: `RENDERER qwen3.5` and `PARSER qwen3.5` are REQUIRED for tool
4
+ # calling / function calling to work. Without them Ollama falls back to
5
+ # generic text handling, `ollama show` reports "does not support tools",
6
+ # and /api/chat ignores the `tools` field.
7
+ #
8
+ # The embedded chat template in the GGUF already supports tool use. These
9
+ # two directives wire Ollama's native Go renderer + parser for Qwen3.5 so
10
+ # the model's <tool_call> output gets translated to structured
11
+ # message.tool_calls in the API response.
12
+ #
13
+ # Default here uses the Q4_K_M (~5.6 GB) quant. To use Q8_0 (~9.5 GB)
14
+ # replace the FROM line with:
15
+ # FROM ./qwen3.5-9b-qwen3.6-reasoning-distilled.q80.gguf
16
+ # To use F16 (~17.9 GB):
17
+ # FROM ./qwen3.5-9b-qwen3.6-reasoning-distilled.f16.gguf
18
+ #
19
+ # To create the model locally after downloading the GGUF + this Modelfile:
20
+ # ollama create qwen3.5-9b-qwen3.6-distilled:q4km -f Modelfile
21
+ # ollama show qwen3.5-9b-qwen3.6-distilled:q4km # should list: tools, thinking
22
+ # ollama run qwen3.5-9b-qwen3.6-distilled:q4km
23
+ #
24
+ # Or pull directly from this HF repo (specify the quant as the tag):
25
+ # ollama pull hf.co/cudabenchmarktest/qwen3.5-9b-qwen3.6-reasoning-distilled-GGUF:Q4_K_M
26
+ # ollama pull hf.co/cudabenchmarktest/qwen3.5-9b-qwen3.6-reasoning-distilled-GGUF:Q8_0
27
+ # ollama pull hf.co/cudabenchmarktest/qwen3.5-9b-qwen3.6-reasoning-distilled-GGUF:F16
28
+ # NOTE: Direct `hf.co/...` pulls do not apply this Modelfile — Ollama will
29
+ # auto-derive one from GGUF metadata. If the auto-derived model loses tool
30
+ # support, download this Modelfile + the GGUF and `ollama create` manually.
31
+
32
+ FROM ./qwen3.5-9b-qwen3.6-reasoning-distilled.q4km.gguf
33
+
34
+ RENDERER qwen3.5
35
+ PARSER qwen3.5
36
+
37
  PARAMETER temperature 0.6
38
  PARAMETER top_p 0.95
39
+ PARAMETER stop "<|im_end|>"