unsloth

/

DeepSeek-V4-Flash-GGUF

like 153

Follow

Unsloth AI 27.5k

Cool!!! Thanks for the model!

BTW, I noticed the UD-Q8_K_XL is from yesterday. Is it fixed too?

Yes UD-Q8_K_XL works, but you will need to use the PR: https://github.com/ggml-org/llama.cpp/pull/25402

Tried to use it with Unsloth Studio (fresh installation, everything's fine - no errors).

IQ3_XXS
hardware: Threadripper 3960X, 128GB DDR4, 2x RTX 3090 (x16, x16, no nvlink).

1. No matter the context window size, it fills up 24+24GB of VRAM almost entirely: 22 to 23GB per each GPU. This stays true even at no context whatsoever (like, 4K).
2. Generation speed is initially low (7 - 8 t/s and it falls even lower to 4 - 5 t/s when I increase the context window to 256K - these numbers were seen at 35K / 100K and 35K / 256K, so, a pretty small chat history).
3. System RAM usage grows with each chat turn. Every time the model answers, I see +1GB to +2GB more of system RAM getting occupied by this thing. What's going to happen when it reaches 128/128GB of RAM?
4. Generally speaking, I find it weird that the total memory usage (VRAM+RAM) gets so high. GPUs are both doing compute - showing spiky graphs of activity during inference. Let's say at some point the model occupies 155 to 160GB of the total memory pool (same 35K / 100K chat), while GGUF file is only ~100GB. Isn't this weird, considering that DeepSeek V4 promised us a 1M context window at lower memory requirements?

Now let's compare it to LMStudio with other quants (by tarruda - IQ3_XXS or Q3_K):
-- Pre-fill speed is just as atrocious. It's slow in both cases, thanks to my DDR4, I'd assume.
-- Generation speed, on the other hand, is where things get interesting. I get 10 t/s when I disable one of my RTX 3090 in LMStudio.
-- It fails to saturate VRAM so tightly for both GPUs (one gets full while the other stays at 10GB); multi-GPU rig just generally seems useless for MoE models, given the increase in t/s with 1 GPU.

Feeling deeply confused. Perhaps I'm doing something wrong.

Tried to use it with Unsloth Studio (fresh installation, everything's fine - no errors).

IQ3_XXS
hardware: Threadripper 3960X, 128GB DDR4, 2x RTX 3090 (x16, x16, no nvlink).

1. No matter the context window size, it fills up 24+24GB of VRAM almost entirely: 22 to 23GB per each GPU. This stays true even at no context whatsoever (like, 4K).
2. Generation speed is initially low (7 - 8 t/s and it falls even lower to 4 - 5 t/s when I increase the context window to 256K - these numbers were seen at 35K / 100K and 35K / 256K, so, a pretty small chat history).
3. System RAM usage grows with each chat turn. Every time the model answers, I see +1GB to +2GB more of system RAM getting occupied by this thing. What's going to happen when it reaches 128/128GB of RAM?
4. Generally speaking, I find it weird that the total memory usage (VRAM+RAM) gets so high. GPUs are both doing compute - showing spiky graphs of activity during inference. Let's say at some point the model occupies 155 to 160GB of the total memory pool (same 35K / 100K chat), while GGUF file is only ~100GB. Isn't this weird, considering that DeepSeek V4 promised us a 1M context window at lower memory requirements?

Now let's compare it to LMStudio with other quants (by tarruda - IQ3_XXS or Q3_K):
-- Pre-fill speed is just as atrocious. It's slow in both cases, thanks to my DDR4, I'd assume.
-- Generation speed, on the other hand, is where things get interesting. I get 10 t/s when I disable one of my RTX 3090 in LMStudio.
-- It fails to saturate VRAM so tightly for both GPUs (one gets full while the other stays at 10GB); multi-GPU rig just generally seems useless for MoE models, given the increase in t/s with 1 GPU.

Feeling deeply confused. Perhaps I'm doing something wrong.

How can you use the quant in other apps without using the llama.cpp PR? Did you manually install it? Unfortunately the new PR we did causes context to use much more VRAM like 1GB at minimum unfortunately but it's the only way to get proper accuracy. And the more context you use, the even more VRAM/RAM you will use

@shimmyshimmer

Oh, perhaps I didn't make it clear - I was comparing Unsloth's IQ3_XXS running in Unsloth Studio to this https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF running in LMStudio. Basically, no, I did not attempt to shove Unsloth's IQ3_XSS into LMStudio.

My personal experience is kind of diminished by this weird sensation of meddling with something too janky for my level of knowledge.
Anyway, I hope other folks will provide a more sensible feedback.

@shimmyshimmer

Oh, perhaps I didn't make it clear - I was comparing Unsloth's IQ3_XXS running in Unsloth Studio to this https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF running in LMStudio. Basically, no, I did not attempt to shove Unsloth's IQ3_XSS into LMStudio.

My personal experience is kind of diminished by this weird sensation of meddling with something too janky for my level of knowledge.
Anyway, I hope other folks will provide a more sensible feedback.

Thank you would you happen to have logs for Unsloth Studio? But yes, in general what I meant is regardless of which DeepSeek GGUF version you use, you need to update it to the latest version of llama.cpp from like 1 day ago to get proper outputs otherwise you'll get bad results. Using the latest version will also cause you to use much more VRAM regardless of which GGUF you use. You should try updating your llama.cpp version to the latest then compare speeds.

Hi, thanks for the GGUF release. Are there any plans to release an MTP / draft GGUF model for DeepSeek-V4-Flash speculative decoding in llama.cpp?

good

Hey hey @danielhanchen ,

I am a bit curious and need advice. (My setup is 8 RTX3090s)
Before official llama.cpp received the fixes I ran your Q8 GGUF with following llama.cpp fork https://github.com/fairydreaming/llama.cpp.git

With that fork, I am able to let it run Q8 GGUF at 350k context with -b 2048 -ub256

Now I tried to run the same Q8 GGUF with the latest llama.cpp official release I get OOM with the same startup command. I have to lower the context to 100k to get it running.

This is my startup command I use with llama fork is as follows.

./build/bin/llama-server
-m /mnt/extra/models/deepseek4flash/DeepSeek-V4-Flash-UD-Q8_K_XL-00001-of-00005.gguf
--host 0.0.0.0
--port 8788
--alias MainLLM
-ngl 99
-fa on
--no-mmap
--jinja
--ctx-checkpoints 0
-c 250000
-b 2048
-ub 256
--cache-type-k q8_0
--cache-type-v q8_0
--parallel 1

does it properly working with cpu+4gpu setups i have 40gb vram but its consuming everything at 60k context size pp sucks at 27t/sec since we cant use the b and ub

This version can run, but the KV context occupies an especially large amount of video memory—about 30 GB for a 32 k context. Why?

This version can run, but the KV context occupies an especially large amount of video memory—about 30 GB for a 32 k context. Why?

I have the same problem. Even using kv q8_0, I fill a 3090 with 32k ctx. And, when I try to split it into two 3090s, it only double the memory usage (I mean, it fills both the gpus).

This version can run, but the KV context occupies an especially large amount of video memory—about 30 GB for a 32 k context. Why?

I have the same problem. Even using kv q8_0, I fill a 3090 with 32k ctx. And, when I try to split it into two 3090s, it only double the memory usage (I mean, it fills both the gpus).

OOhhh.... I just saw there's a PR to address that: https://github.com/ggml-org/llama.cpp/pull/24231

His fork looks interesting as well. He is promising that he fixed DS4F with llama.cpp. More context with better speed.
https://x.com/superalesha/status/2074569147279724715?s=20

are you saying these quants are better than antirez?

I can run ds4 just fine, but with these I am having issues. Even the UD-IQ2_XSS with only 60,000 context can't load in llama.cpp on Strix Halo, I have 124GB allocated to the iGPU, but when it tries to allocate context it always runs out of memory:

podman run --rm -it --name llama-cockpit-server --device /dev/dri --device /dev/kfd --group-add keep-groups --security-opt seccomp=unconfined --security-opt label=disable --userns=keep-id -v /home/kyuz0/models:/models:ro -p 127.0.0.1:8080:8080 docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-7.2.4 llama-server -m /models/DeepSeek-V4-Flash-GGUF/UD-IQ2_XXS/DeepSeek-V4-Flash-UD-IQ2_XXS-00001-of-00003.gguf -c 60000 -ngl 999 --host 0.0.0.0 --port 8080 --no-mmap -fa 1 --jinja --reasoning on --chat-template-kwargs '{"reasoning_effort":"max"}'

Press Ctrl+C to stop the server and return to the UI.

0.00.018.860 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.020.494 I srv    load_model: loading model '/models/DeepSeek-V4-Flash-GGUF/UD-IQ2_XXS/DeepSeek-V4-Flash-UD-IQ2_XXS-00001-of-00003.gguf'
0.00.401.680 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort
0.17.877.183 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 49050.10 MiB on device 0: cudaMalloc failed: out of memory
0.17.877.193 E ggml_gallocr_reserve_n_impl: failed to allocate ROCm0 buffer of size 51432753408
0.17.877.196 E graph_reserve: failed to allocate compute buffers
0.17.885.652 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.17.885.663 E cmn  common_init_: failed to create context with model '/models/DeepSeek-V4-Flash-GGUF/UD-IQ2_XXS/DeepSeek-V4-Flash-UD-IQ2_XXS-00001-of-00003.gguf'
0.17.885.669 E cmn  common_init_: failed to create context with model '/models/DeepSeek-V4-Flash-GGUF/UD-IQ2_XXS/DeepSeek-V4-Flash-UD-IQ2_XXS-00001-of-00003.gguf'

Even when I am able to run it, I seem to get stuff in Chinese and sometimes it just hangs, I have a feeling at least on Strix Halo the llama.cpp implementation is not stable.

This model is too slow and takes up a lot of VRAM. Could you convert this Tencent model to GGUF: https://huggingface.co/tencent/Hy3 This model's capability is better than DeepSeek-V4-Flash.

I get like 40-80 tokens/sec (really not usable) for prompt processing for a 192gb 4000MT 24GB 5090 Notebook.
for generation like 8 toc/sec (ok)

settings:

@echo off

"E:\llama_ai\llama-b9977-bin-win-cuda-13.3-x64\llama-server.exe" ^
-hf "unsloth/DeepSeek-V4-Flash-GGUF:UD-Q4_K_XL" ^
--alias "DeepSeek-V4-Flash-GGUF:UD-Q4_K_XL" ^
--gpu-layers 999 ^
-ot ".[0-9]+.ffn_(gate|up|down)_exps.=CPU" ^
--no-mmproj ^
--flash-attn on ^
--no-mmap ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
--ctx-size 50536 ^
--ubatch-size 2048 ^
--batch-size 2048 ^
--reasoning on ^
--threads 16 ^
--parallel 1 ^
--host 0.0.0.0 ^
--port 11434 ^
--presence_penalty 0.0 ^
--temp 1.00 ^
--top-p 1.00 ^
--min-p 0.00 ^
--top-k 20 ^
--jinja

pause

pause

this is disappointing.

1.33.671.244 W resolve_fused_ops: Lightning Indexer not supported, set to disabled
Something to be concerned about?

Context size -> is really bad scaling concerning RAM (not anything the technical paper is on about). Vram also suddely spikes up at like>60k (strange gb steps during loading up: 7, 15, 21, >23.6gb (40...) gb)
When bigger 23gb vram , model stalls indefinitely...