AlexanderKyng

/

Qwen3.6-27B-Fusion-711-Uncensored-MTP-48gb

like 2

so far no reboots , working as it should !! not as fast as q5 version but quality of response is definitely better than q5!!
22.34.879.775 I slot print_timing: id 2 | task 9605 | prompt eval time = 1598.08 ms / 1005 tokens ( 1.59 ms per token, 628.88 tokens per second)
22.34.879.780 I slot print_timing: id 2 | task 9605 | eval time = 16850.10 ms / 385 tokens ( 43.77 ms per token, 22.85 tokens per second)
22.34.879.781 I slot print_timing: id 2 | task 9605 | total time = 18448.18 ms / 1390 tokens

even this speed is quite good for my work !! Thank you very much for this awesome work!!
UPDATE -- this is without mtp enabled -- as if i enable mtp --its get -- out of memory error !! still for dense model with 64k context -- this is a good margin
--split-mode tensor
--tensor-split 1,1
--host 0.0.0.0
--port 1106
--jinja --reasoning-preserve --flash-attn on --parallel 4

i am using this command to run

Hi, thank you again for your feedback, it is of an extreme value to me.

I tried my command, as written on the model card, with --parallel 4 and it OOMed as it did for you. The trick was to lower the batch and ubatch sizes (-u 1024 -ub 1024), did you try that ? It worked for me :

   --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 4 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 1024 -ub 1024 \
    --spec-type draft-mtp \
    --spec-draft-device CUDA1 \
    --alias Qwen3.6-27b \
    --jinja

ok i will try this

this command too goes out of memory -- any way -- this model too caused restart and the mixed q5 also caused restart today -- but i found that whenever the propmt processing goes beyond 64k -- the pc restarts on this model -- may be some gpu spike happen at that time -- now i will download smaller qaunt of qwen3.6 27b and try to load it will same conditions -- if it does not cause reboot after 64k context then i will keep using that

That is quite troubling...
On my end, batch and ubatch at 1024 do not go OOM, maybe because I also use an NVLink.
For the restart error, did you ever had such bugs with other LLMs ? Do you use Linux ? And do you have a sufficiently powerful PSU ? These are just a few things that could help pinpointing where the problem lies. I never had a single problem with these quants even at 260K in context... I trully do not understand what can trigger such error..

more update -- qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp this one with q5_k_m loaded and ran successfully in lmstudio +grok ran the model