Can see, can touch but can’t use
Thank Qwen for your open models.
But 180B is too large for most people.
Hope that you guys will release smaller models, dense and Moe.
Thank so much.
But 180B is too large for most people.
Quantization is meant for that, but we will probably need to wait a few more days until the inference engines work well with it. A6B should run really fast though.
But 180B is too large for most people.
Quantization is meant for that, but we will probably need to wait a few more days until the inference engines work well with it. A6B should run really fast though.
Even at q4 you need more than 64 GB RAM. So OP is still right. Most systems have 16- 32 GB RAM and 4-8 GB VRAM. I hope we will see a 35B too on this new architecture, which can be run on consumer hardware.
50GB is max i can go. waiting for quants :)
comment sent twice sorry.
Even at q4 you need more than 64 GB RAM. So OP is still right. Most systems have 16- 32 GB RAM and 4-8 GB VRAM. I hope we will see a 35B too on this new architecture, which can be run on consumer hardware.
Theres always openrouter, runpod or other providers if hardware is too expensive, this model should be really cheap to run in the cloud. But yeah, i hope the models continue to become even more efficient in the future or the hardware prices cheaper.
51B n-gram embedding lives in CPU RAM, not VRAM. Check PLE offloading.
51B n-gram embedding lives in CPU RAM, not VRAM. Check PLE offloading.
Doesn't change a thing. 120B parameters still have to load in RAM, and not everyone has over 64 GB of RAM.