mradermacher

/

sarvam-30b-uncensored-i1-GGUF

like 1

For anyone wondering why reasoning does not work in llama.cpp with these ggufs, here's a fix I made for the template and params:

default template inside gguf was not working - does not produce thinking. But we can use external one.
what was needed is detection of thinking from template - it looks for reasoning_content within template
when llama detects that, it writes
srv init: init: chat template, thinking = 1
in the console
and it also needed correct reasoning-format inside llama to wrap the thinking block
and it needed a phrase for it to jumpstart thinking within template

so the full fix:
command line parameters for llama.cpp (only these 2):
--chat-template-file sarvamNewTemplate.txt --reasoning-format deepseek-legacy

and then the new template:

{{- '[@BOS@]\n' -}}
{%- for m in messages -%}
    {%- if m.role == 'system' -%}
        {{- '<|start_of_turn|><|system|>\n' + m.content + '<|end_of_turn|>\n' -}}
    {%- elif m.role == 'user' -%}
        {{- '<|start_of_turn|><|user|>\n' + m.content + '<|end_of_turn|>\n' -}}
    {%- elif m.role == 'assistant' -%}
        {{- '<|start_of_turn|><|assistant|>\n' -}}
        {%- if m.reasoning_content -%}
            {{- '<think>\n' + m.reasoning_content + '\n</think>\n' -}}
        {%- endif -%}
        {{- m.content + '<|end_of_turn|>\n' -}}
    {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    {{- '<|start_of_turn|><|assistant|>\n<think>\nLet me think through this carefully.\n' -}}
{%- endif -%}

now it always reasons and outputs reasoning text correctly within reasoning block in frontend
generated text is also printed correctly, outside the thinking block