New GGUFs required? "chat : add new template for DeepSeek V4 Flash 0731 (#26398)"
https://github.com/ggml-org/llama.cpp/pull/26398#issuecomment-5170353135:
@ggerganov it will have to be re-embedded into existing GGUFs (gguf-new-metadata). New GGUFs should pick it up automatically.
Note that this is not a critical template fix, main things it enables is supporting reasoning levels and adding the prompt DSv4 was trained to emit structured output.
But the later bugfix caught by @coder543 will actually benefit existing GGUFs.
Should we assume that Unsloth's quants do not need this chat template updates (with GGUFs re-upload)? I've read the Unsloth's DeepSeek 4 Flash 0731 guide and it said that it has some template fixes integrated, and this was before above pull request was merged. So, no update needed, all GGUFs still good as they are?
You can change chat template just by passing an argument to llama-server specifying the path to the chat template. No need for new GGUFs.
The chat template Unsloth uses is mostly correct already, but I recall seeing one minor issue when I was auditing it against that PR. I would have to dig into my notes to try to find out what the issue was, but if the template is working for you, it’s probably fine? If you want to switch templates, you can do that easily.
The problem is that an override to the new template file doesn't work out-the-box on my Windows 11 machine. It has some escape character quirks specific to Windows (I suppose). I could probably fix it by replacing affecting characters manually, but I won't be confident in that I wouldn't break something important in the process 😅. I would be happy to just have a proper solution that is either integrated or overrides cleanly.
Here's my error messeage when I try to override template file, btw:
0.01.217.978 E common_chat_templates_init: error: lexer: unknown escape character \u
...iting a '<' + dsml_token + 'tool_calls\u003e" block like the following:\n\n\u...
How did you download this file? This doesn't look like a Windows issue, this looks like you somehow ended up JSON-encoding the jinja template? \u003e is ">".
Just download the file again?
curl.exe -L --fail -o deepseek-ai-DeepSeek-V4.jinja "https://raw.githubusercontent.com/ggml-org/llama.cpp/refs/heads/master/models/templates/deepseek-ai-DeepSeek-V4.jinja"
Yay! Works now. 👍 You were right, somehow my downloaded file was completely butchered in the process. Lesson learned, thank you very much for your help! 🍀