Finally Qwen3.5-122B-A10B-NVFP4 working on Thor!
Thank you, it works on my Thor too! And thank you for uploading the resharded model.
I actually ran it in another container and got it up to 256k context window, and there's still room. I basically need a single request at a time and prefer higher capability so context window over concurrent requests!
The commands I used are here:
https://github.com/pastoriomarco/thor_llm/tree/main/models/qwen3.5-122b-a10b-nvfp4-resharded
Hey, sorry for the late reply, glad it worked for you!
Since then I've been able to get 2–2.5× decode speed improvements on Thor using DFlash block-diffusion speculative decoding on select models that have draft heads available. The docker image builds from a specific vllm pull request supporting dflash diffusion block speculative decoding.
The new image and benchmarks are here:
- Docker image + DFlash recipes: https://huggingface.co/patrickbdevaney/qwen-3.6-35b-a3b-dflash-jetson-agx-thor
- Benchmark results and serve scripts: https://github.com/patrickbdevaney/dflash-vllm-resources
Measured results on Thor:
- Qwen3.6-35B-A3B-NVFP4: 100–139 tok/s with DFlash (vs ~40 tok/s autoregressive)
- Qwen3.6-27B-NVFP4: ~50 tok/s with DFlash (vs ~17 tok/s autoregressive — 248% of the bandwidth ceiling)
- Qwen3.5-122B-A10B-NVFP4: 40–52 tok/s with DFlash at k=10, exceeding its theoretical autoregressive ceiling on some tasks
Given your preference for single request + maximum context, the 122B with DFlash should be a solid upgrade over the base config. The serve scripts in the repo handle the 122B-specific gotchas (Cutlass MoE backend, drop_caches pre-flight, etc.).