kodelow

/

Hy3-NVFP4-W4A16

like 12

Follow

Kodelow 8

Hi!
It'll definitely fit but not with full context.
Most likely Hy3 use full attention and default KV cache precision is FP16 so I expect it to be memory heavy.

It's possible though with some heavy offloading or KV eviction strategy like TriAttention or similar

BR,
/kodelow

Would be great if somebody could post his results for 2x 6k Pros

Would be great if somebody could post his results for 2x 6k Pros

Same, I am running Deepseek v4 Flash with 1M context on them currently, once someone else confirms if it's worth it, I might switch...

Deepseek V4 Flash is much more efficient in terms of KV cache memory footprint.

So if you’re considering Hy3 vs DS v4 on 2x 6k Pros I’d say it’s not worth it - memory budget is too tight.

BR,
/kodelow

I'm going to try with 4x RTX PRO 6000. If the model codes like a champ, I'll spin it up on 2x. If it's trash, I'll reclaim the disk space.

i promised to give it a try and report back, so here it is. first, I don't want to crap all over the model with negative only comments. when i see people doing this, it makes them seem totally ungrateful for this amazing gift called opensource. i am fully grateful for the gift and the decision by tencent to allow all of us to have access to these model weights. thank you tencent!

having said that, the model is trash and almost certainly benchmaxxed. when i see the model is beating other models i use on every benchmark, i expect that to translate into better outputs. but in NONE of my testing did i see better performance than my two main daily drivers, deepseek-v4-flash and minimax-m3. two models that do worse than hy3 on nearly every benchmark.

for my testing i am in progress on 5 or so projects with small-medium size codebases, already opened in my coding agent (pi). i switched the model to hy3 on all projects and continued taking tickets (issues in the code repo), implementing new code or doing fixes, as specified in the ticket detail. both deepseek v4 and minimax-m3 work pretty well in these use cases. minimax-m3 being a bit better. hy3 was decidedly WORSE in all codebases than what I've seen from deepseek-v4 flash and minimax-m3.

regarding the OPs original question about 2x PRO RTX6000, that's a great big NO. I started at context size 131072 and kept halving it, attempting to get the model to load. the shards load but then i get an oom. my final attempt was context size of 16k, which oom'd, even with gpu mem set to 0.95.

i didn't see any memory oom with 4x RTX 600 PRO Blackwell 96GB.

My final thoughts after a few hours of use:

pros:

• fast! beats minimax-m3 nvfp4 on the same hardware, especially parallel requests

cons:

• makes lots of stupid mistakes, like when i use a skill it looks for the skill scripts in current directory instead the scripts directory inside the skill itself.
• way too many occurences of spewing chinese characters for no reason at all, EVEN IN CODE SOURCE FILES!
• way too many failed tool calls (tried both jinja file from this hf repo and without, using vllm nightly, built fresh 7/7/2026)
• even with 2 draft tokens, fairly low acceptance rate. i fear 3 would be a disaster
• too many occurences of the model just "stops" generating, no error, nothing to trace, it just decides it's done and i have to prompt it to keep going

final verdict

i'll be reclaiming the disk space and going back to minimax-m3, waiting for a day we really do have a model that can get close to frontier performance in a ~300B parameter model. i really hope model makers begin to focus on actual performance in real world coding instead of the current trend of benchmaxxing.

Thanks a lot for such a great review Aaron!

I assume some of those could be quantization artifacts but not all of them

I’ll add a disclaimer to the model card and will try to attribute those issues to KV cache, and template.

BR,
/kodelow

I guess it would be helpful to see how I'm running the model, in case I've done something wrong to cause the issues I'm seeing, so here is the compose. again, the vllm nightly was built today. I'm kicking myself for having wasted half a day on this.

cat Hy3-NVFP4-W4A16/docker-compose.yaml 
services:
    hy3:
      image: registry.local/vllm:2026-07-07-nightly
      restart: unless-stopped
      container_name: hy3
      shm_size: 32g
      ipc: host
      cap_add:
        - SYS_PTRACE
        - IPC_LOCK
      ulimits:
        memlock: -1
        stack: 67108864
      ports:
        - "9070:8000"
      volumes:
        - /mnt/data/models:/mnt/data/models
      environment:
        - OMP_NUM_THREADS=16
        - NCCL_P2P_DISABLE=1
        - NCCL_PROTO=LL
        - NCCL_ALGO=Ring
        - NCCL_MIN_NCHANNELS=8
        - NCCL_NTHREADS=512
        - NCCL_IB_DISABLE=1
        - VLLM_WORKER_MULTIPROC_METHOD=spawn
      deploy:
        resources:
          reservations:
            devices:
              - driver: nvidia
                device_ids: ["0","1", "2", "3"]
                capabilities: [gpu]
      command: >
        --model /mnt/data/models/kodelow/Hy3-NVFP4-W4A16
        --served-model-name hy3
        --port 8000
        --tensor-parallel-size 4
        --max-model-len 196608
        --speculative-config.method mtp
        --speculative-config.num_speculative_tokens 2
        --tool-call-parser hy_v3
        --reasoning-parser hy_v3
        --enable-auto-tool-choice 
        --kv-cache-dtype fp8
        --gpu-memory-utilization 0.95

Thanks for sharing config!

Well I thought it was Minimax M3 level as well according to anecdotal reports and first evals after quantization - apparently it’s not.

At this point I can only figure out if this KV cache, broken template, poorly quantized weights or base model quirks.

BR,
/kodelow

I'm looking to answer that exact question. Giving the FP8 a try. That'll tell me if something went way wrong with the NVFP4 conversion or something like that.

Appreciate all the work you put into this one. Sorry it didn't work for me.

It might be broken template or template parser: check this out - https://github.com/sgl-project/sglang/pull/29920

This PR states similar behavior to what you just reported
I also went to FP8 model page and discussion section pointing out to template issues on both sglang and vllm