HauhauCS

/

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive

like 3.52k

Startup code：
C:\Users\TK\Desktop\vllm\llama-b8851-bin-win-cuda-12.4-x64>llama-server.exe -m "C:\Users\TK\Desktop\vllm\models\Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-IQ3_M.gguf" -c 65536 --flash-attn on -ctk iq4_nl -ctv iq4_nl -ngl 40 --cpu-moe --cpu-mask 0xFFFFFFFF --batch-size 7400 --ubatch-size 3700 --cont-batching --threads 24 --api-key 123456 -rea off --jinja

proof at my log:
prompt eval time = 442.60 ms / 15 tokens ( 29.51 ms per token, 33.89 tokens per second)
eval time = 4581.70 ms / 146 tokens ( 31.38 ms per token, 31.87 tokens per second)
total time = 5024.29 ms / 161 tokens

Ok

keep up and keep sharing

(C:\Users\TK\Desktop\vllm\llama-b8851-bin-win-cuda-12.4-x64>llama-server.exe -m "C:\Users\TK\Desktop\vllm\models\Qwen3.6-35B-A3B-APEX-I-Compact.gguf" -c 16384 --flash-attn on -ctk q8_0 -ctv q8_0 -ngl 41 --cpu-moe --cpu-mask 0xFFFFFFFF --batch-size 9600 --ubatch-size 4800 --threads 24 --api-key 123456 -rea off --jinja --cache-ram 8192 --parallel 1 --kv-unified --no-mmap --no-context-shift)

40.46T/S

Can a 4080 8GB with 16GB RAM be deployed?

Can a 4080 8GB with 16GB RAM be deployed?

Q3_K_M or Q2,don't think about Q4

Looking how you are smothering it, I am guessing you can easily get 50 to 60 t/s out of that server or better.

Try this instead:

(C:\Users\TK\Desktop\vllm\llama-b8851-bin-win-cuda-12.4-x64>llama-server.exe -m "C:\Users\TK\Desktop\vllm\models\Qwen3.6-35B-A3B-APEX-I-Compact.gguf" -c 16384 --flash-attn on -ctk q4_0 -ctv q4_0 --jinja --no-mmap)

Couple things here...
You are letting llama.cpp manage your ram which it can do it massively better than you can. --no-nmap will do that.
--ngl probably cuts your speed in half the moment you start screwing with that instead of letting llama.cpp manage it with --no-nmap.
q4_0 on KV so you cut down your VRAM KV usage by half freeing up more for the model. You can reliably get 50,000 tokens out of q4 before you start to have errors. You are only running 16384 so you are far from that.
llama.cpp should automatically detect your proc and assign appropriate threads. Note hyperthreading slows LLM's down. You only want to use physical cores.

Note the above was tested on ROCm. CUDA may vary. Hope it helps ya.

Edit: You will get even more speed if you upgrade to Linux.
Another Edit: If you start pulling off the speed I think you can, you can probably get to at least an IQ4_XS for better smarts. Q3's are pretty dumb.

Looking how you are smothering it, I am guessing you can easily get 50 to 60 t/s out of that server or better.

Try this instead:

(C:\Users\TK\Desktop\vllm\llama-b8851-bin-win-cuda-12.4-x64>llama-server.exe -m "C:\Users\TK\Desktop\vllm\models\Qwen3.6-35B-A3B-APEX-I-Compact.gguf" -c 16384 --flash-attn on -ctk q4_0 -ctv q4_0 --jinja --no-mmap)

Couple things here...
You are letting llama.cpp manage your ram which it can do it massively better than you can. --no-nmap will do that.
--ngl probably cuts your speed in half the moment you start screwing with that instead of letting llama.cpp manage it with --no-nmap.
q4_0 on KV so you cut down your VRAM KV usage by half freeing up more for the model. You can reliably get 50,000 tokens out of q4 before you start to have errors. You are only running 16384 so you are far from that.
llama.cpp should automatically detect your proc and assign appropriate threads. Note hyperthreading slows LLM's down. You only want to use physical cores.

Note the above was tested on ROCm. CUDA may vary. Hope it helps ya.

Edit: You will get even more speed if you upgrade to Linux.
Another Edit: If you start pulling off the speed I think you can, you can probably get to at least an IQ4_XS for better smarts. Q3's are pretty dumb.

thanks sir, --no-mmap is really better than single ngl