Does it even work?
I tried it on 2 different setups 2x5060 and a 3x3060 -- with identical parameters on llamap.cpp -- temp 0.2 / ctx =256K, both with Q4_K_M and q8_0 kv
I started both models fresh and gave identical prompt to both
Hello - write me a 30 line poem that ends in cat
5060 produced 30 line poem in 9.9s using 264 tokens.
3060 produced 38 line poem in 20 sec using 454 tokens, when I asked it to fix the count it took 4K tokens to finish the output with plenty of thinking tokens.
With /reset 3060 consistently uses 650+ tokens and produces 30 lines.
With cold boot, 3060 results in totally unpredicatable - never once it came close to correct answer, it's not even behaving as a base model.
I am very new at this -- but this doesn't seem remotely correct or reliable -- am I missing something?
I didn't do a good job reporting -- I looked at it again and it's garbage all over ..
5060 didn't produce 30, but 32 lines.
I asekd it to make it exactly 30 lines, and all hell broke lose -- it produced , 36, 48, even 56 lines -- but not 30!
Then it stopped at 20, 12, 12, 12, 16, 16, 20 -- and didn't even finish the last line as asked -- Not once did it do 30 lines in at least 20 attempts!
This is all on 2x5060.
3060 setup produced wrong asnwers but corrected -- but 5060 is actually worse of the two.
For comparison I ran qwen3.6-27B Q6 on 3060 setup and it also produced wrong answer once, it hallucinated when it tried to correct it, but recovered on it's own and produced the correct answer.
Needless to say this took north of 4K tokens -
Frankly I think this is too simplistic a test for the model to fail this badly -- I am sure these guys are smarter than I, but I will not use this model for anything.
@sa13ma not so long ago MTP wasn't working as expected with parallelization, maybe it's been fixed since then but did you try with --parallel 1 just to see? And what about without kv quantization? Was it with thinking ON?
just did 6 attempts, 3/3 OK - 30 lines, ends with "cat" - you exact same prompt Hello - write me a 30 line poem that ends in cat
first 3 attempts with temp 0.75, 3 next ones with temp 0.2 (that said, 4th attempt with 0.2 ended up in a loop, which I don't think 0.75 would have resulted in)
cmd: |
/home/user/llama.cpp/build/bin/llama-server
--model /mnt/storage/GGUFs/Qwen3.6-27B/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf
-lv 4
--webui-mcp-proxy
--tools all
--no-warmup
--ctx-size 100000
--no-context-shift
--n-gpu-layers 66
--temp 0.75
--top-p 0.95
--top-k 20
--min-p 0
--presence-penalty 0.2
--repeat-penalty 1
--jinja
--host 0.0.0.0
--port ${PORT}
--flash-attn on
# --mlock
--ctx-checkpoints 4
--parallel 1
--spec-type draft-mtp
--spec-draft-n-max 2
# --backend-sampling
My other settings were identical to yours except --spec-draft-n-max 3
As main usee case in local inference is coding ( @ temp. 0.1), so for the intended use case it failed 3/3 !
Base Qwen3.6-27B-Q6_K produces clean output ---though it does use 1575 tokens (on 3060), on 5060 it does a bit better in tokens ( idnw)
To get this model to generate clean output is near impossible -- in one case I was able to do it, it took more than 4K topens and multiple iterations of reprompting with changed phrase etc.
I don't know much about the math behind this code, but these mushrooming newly "trained" variants are not tested at all, writeup shows no quantified output execept token count.
I don't think this method of training is yielding a sueprior results.
Why do you assume --temp 0.1 is good for coding?
But wait you were talking about the 30 lines poem test, which I ran too and don't get the same outcome. Why are we talking about coding now? "main use case in local inference is coding ( @ temp. 0.1), so for the intended use case it failed 3/3" wdym?
Mac M3, GGUF Q8_0, 1st run, temp 0.1
@owao You ran 2 tests -- Temp @0.75 worked, and at @0.2 it failed. -- I had identical results.
@oplatek Recommended setting for coding in most models (including qwen 3.6) is 0.1.
If the model deteriorates @ 0.2 -- it's likely to be much worse @ 0.1.
I was pointing out that most people, using it at @0.1 are likely to get awful results and with what I have seen.
I would not use it even @1.0 -- it has no logical use for me, getting wrong or poorer answers faster is not useful.
And there is no posted data on performance at any temp setting vis-a-vis base model except number of tokens-- if I have missed it, please help me see it.
@sa13ma we can't really tell it failed @0.2, 3/4 succeeded and 4th attempt was just a loop (I also observed loop with Qwen3.6-27b from time to time, that's why I use presence penalty 0.2 even if it didn't do the trick here), but I think it bet it was unusual and unlucky. But maybe I missed your point im not sure. But what I can tell is that over enough runs, your results shouldn't be hardware dependent! Or something is wrong. Maybe you could try running with temp 0 to see if at least greedy decoding gives you the same exact output between your different hardware configs?
and what about unquantized kv cache? Cause even if there have been improvement on accuracy with quantization there, last time I saw ggreganov do some benchmarks it was still noticeable, even in q8_0
last time I saw ggreganov do some benchmarks it was still noticeable, even in q8_0
Maybe not actually with the new rotation logic! https://github.com/ggml-org/llama.cpp/pull/21038#issuecomment-4150413357 cool to see!
@owao Well -- I am not sure q8_0 SHOULD matter -- if base model is not affected why should the derivative be so far?
I will put the matter in another way -- heuristically.
Prompt has only 2 operative commands -- #1 count the lines, #2 end with cat.
Base model can follow them and deliver an answer.
The thinking cap strips out it's brain so badly that it is unable to follow both most of the time, (it either produces a different line count or a different ending).
Sometimes it can't even follow either of the constraints.
I am not expert .. this is a laymans way of looking at it.
I can't explain the HW difference, I expected speed difference, but even with base model there is also a huge difference (2-3x) in tokens used, thinking is very different.
I see a lot more thinking tokens on 3060s compared to 5060s(both setups are Q6_K and q8_0 for kv) - but no one says how many tokens the model will use produce for an outcome, yes time is measured but quality of output is the only critera for evaluation.
I have read some of the numbers and experiments done with quants .. but not knowing much, one thing is obvious to me that lessons learned on one model maynot apply to another at all.
These experiments (of removing thinking or guard rails etc. ) are being done more as experiments and not necessarily are meant for "production use".
I'm confused, why do you say
The thinking cap strips out it's brain so badly that it is unable to follow both most of the time
when I shared that at least in my test it succeeded 6 out of 7? I don't get it
even with base model there is also a huge difference (2-3x) in tokens used, thinking is very different
this is the point of this model?
I see a lot more thinking tokens on 3060s compared to 5060s
You really shouldn't if testing the same model, unless your HW is faulty, or unstable OC?
I'm confused, why do you say
>>>> The thinking cap strips out it's brain so badly that it is unable to follow both most of the time
that's just a way of explaining why, I am not an expert in this field -- I tried to explain the loss in in simple terms..
when I shared that at least in my test it succeeded 6 out of 7? I don't get it
>> even with base model there is also a huge difference (2-3x) in tokens used, thinking is very different
That is the point of this model
but it doesn't do without losing something -- the "promise" as I understood it, that it's an improvement without loss.
If there is a loss then it should be stated so one can evaluate it's use.
why 6/7 ? No one runs a taks with a 15% probability that they are dealing with a "wrong" answer -- 99% you don't know the answer to every prompt and you trust the model will be doing it right.
(I had misread your original comment -- i thought you said 0.2 all failed -- I ran it about 10 times on both machines and one machine failed 100% one did work after repromts)
BTW I ran the same test on intel iarc a770 with lllama3-8b and it also did 25 lines repeatedly but was correct on 4th attempt!
Same test on iarc with Ornith1-9b -- it did it right the first time in 254 tokens!
I see a lot more thinking tokens on 3060s compared to 5060s
You really shouldn't testing the same model, unless your HW is faulty, or unstable OC?
????? I shoudln't be ? I don't get that -- no I don't overclock -- and if my hardware is faulty it will surely not fail in this fanastic manner
3060 and 5060 hardware is different, math is different , I have 3x3060s in one machine and 2x5060s .. the way code will split load across boards is different
I can't even fathom in ways it will affect the output but I was surprised by the way thinking tokens were spat out by 3060s vs 5060 .. 3060 did split out more thinking tokens and produced better output
5060 didn't think much :-( and when it did, it god hopelessly lost in loops
Anyways I am done with this experiment now -- there is nothing I can learn from this digression except not to chase the experiments!
I realized I confused slot parallelization with tensor parallelization and --parallel is for... parallelization slots, setting the number of "concurrent requests" not how many gpu to split across. So it was pointless suggesting to set it to 1...