Reasoning does not work
I am using lama-server.exe from "llama-b10087" release.
No matter if use or not "--reasoning on" (latest command in the newer releases) or "--default-chat-template-kwargs '{"enable_thinking": true}'", there is no reasoning step.
In fact, the reasoning is "erratic". If I question "Explain black holes and the plausibility of "white holes".", there will be a reasoning step. But if I try to send a file to debug, it does not think. Or even if a test with a "who are you", it also does not think. Like if the model itself was deciding when to think. If that is the case, is a conceptual error, because a file to debug must be allways in thinking mode.
I don't think mainline llama.cpp has full support for this models family arch yet.
Unsloth states this at the top of their release of this models model card:
"""
...
Laguna support is not in a tagged llama.cpp release yet, so build llama.cpp from ggml-org/llama.cpp#25165:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
gh pr checkout 25165
build with CUDA (drop -DGGML_CUDA=ON for a CPU-only build)
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-cli llama-server
cd ..
...
"""
I noticed that your using an .exe from the b10087 release, which means your likely working with windows.
If your wanting to work with the model now, you will have to compile the source code from the pull request 25165.
Compiling may or may not be available for you depending on your compiling experience with windows.
If you have never compiled any C type code before, it might be best to wait for the main releases of llama.cpp to have support for the models family before trying again. Compiling, expecially with windows can be very difficult. But if you know what your are doing, I'd say go for it!
llama.cpp team will likely have full support with their regular releases eventually.
It does seem that poolside and llama.cpp dev team are working for support in main releases of llama.cpp as seen in the github pull that unsloth mentioned at the top of this models readme/model card. https://github.com/ggml-org/llama.cpp/pull/25165
I hope that this helps. If not, then hang in there, it's coming eventually.
The PR has been merged 20h ago.
So you can also use a very recent main build.
Ok, So it looks like I was needing to catch up with the info available.
I see that the https://github.com/ggml-org/llama.cpp/releases/tag/b10087 shows that family model support has been applied to the main release. The same one you are using. I was not aware of this at the time of my last comment.
You might have something interesting here. If you're wanting to help, then be prepared to share additional details. And if you don't feel like it, that's totally fine too.
I'm using the poolside fork, their Q4_K_M thinks like crazy. Same llama.cpp with Unsloth Q6_K, it mostly doesn't think but sometimes does.
In fact there are people complaining for the exact same issue at the official model page : https://huggingface.co/poolside/Laguna-S-2.1-GGUF/discussions/2
There are some guys stating that it works fine in VLLM, but I did not tested yet. Possibly a problem with the llama.cpp release.
I'm using the poolside fork, their Q4_K_M thinks like crazy. Same llama.cpp with Unsloth Q6_K, it mostly doesn't think but sometimes does.
That's interesting...
Just wanted to add here, my setup was missing config, and I saw this post, https://www.reddit.com/r/LocalLLaMA/comments/1v3wyre/comment/oz9ypep/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
they suggested --rope-scaling yarn --rope-scale 32 --yarn-orig-ctx 8192
After adding that it thinks perfectly.
Just an idea I'm toying with. I'm wondering if this could be possibly relate to the model being very sensitive about needing to see it's own previous thinking blocks. (see: https://huggingface.co/unsloth/Laguna-S-2.1-GGUF#controlling-reasoning from the unsloth model card.)
Here is what unsloth had to say about it:
"... The model will generally reason before calling tools and between tool calls, and may stop reasoning in follow-up steps if prior thinking blocks are dropped. ..."
Maybe double checking that preserve thinking is indeed being done right is in order. Or at least being a bit more careful and suss about it.
Side note: Interesting details about the rope and yarn scaling. Thanks for Sharing H-J-D. There was a lot of other interesting details in the reddit link that was nice to know as well. Such as the potential being seen so far from this model.
Overall, it feels there is a lot of teething problems related to this model everywhere I look. The initial potential others are claiming are indeed interesting. Once things balance out, it sounds like this could be a very decent coding model for a smaller size. Which I like hearing about. Hope that pans out as such.
I've downloaded latest laguna (two quants - q4_k_xl and q5_k_xl) today via Unsloth Studio (with auto-updated llama). It works, but it doesn't think (when run via Unsloth Studio), despite all UI options set.
Poolside's nvfp4-mlx (via omlx) thinks, but it has other issues.
I'm stupid, it works