Instructions to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ")
model = AutoModelForMultimodalLM.from_pretrained("ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ

SGLang

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with Docker Model Runner:
```
docker model run hf.co/ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ
```

This result is so exciting — has anyone done a full benchmark test on it?

by modderbug - opened 1 day ago

Discussion

modderbug

1 day ago

I'd like to know what the actual hands-on experience is like when integrating a code agent.

modderbug

about 23 hours ago

(EngineCore pid=57211) INFO 08-27 15:18:36 [weight_utils.py:897] Auto-prefetch is disabled because the filesystem (9P) is not a recognized network FS (NFS/Lustre) and the checkpoint size (11.02 GiB) exceeds 90% of available RAM (12.11 GiB).
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:20<00:41, 20.56s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [01:22<00:44, 44.98s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [02:26<00:00, 53.82s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [02:26<00:00, 48.99s/it]
(EngineCore pid=57211)
(EngineCore pid=57211) INFO 08-27 15:21:03 [default_loader.py:430] Loading weights took 147.47 seconds
(EngineCore pid=57211) INFO 08-27 15:21:04 [gpu_model_runner.py:5405] Model loading took 10.4 GiB memory and 148.801831 seconds
(EngineCore pid=57211) INFO 08-27 15:21:04 [interface.py:911] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=57211) INFO 08-27 15:21:04 [interface.py:935] Padding mamba page size by 0.13% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=57211) INFO 08-27 15:21:18 [backends.py:1094] Using cache directory: /home/wuxiaowei/.cache/vllm/torch_compile_cache/ebeed8401b/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=57211) INFO 08-27 15:21:18 [backends.py:1155] Dynamo bytecode transform time: 12.37 s
(EngineCore pid=57211) [rank0]:W0827 15:21:19.096000 57211 site-packages/torch/_inductor/utils.py:1953] Not enough SMs to use max_autotune_gemm mode
(EngineCore pid=57211) INFO 08-27 15:21:51 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 33.26 s
(EngineCore pid=57211) INFO 08-27 15:21:57 [backends.py:920] collected artifacts: 65 entries, 21 artifacts, 37731543 bytes total
(EngineCore pid=57211) INFO 08-27 15:21:57 [decorators.py:708] saved AOT compiled function to /home/wuxiaowei/.cache/vllm/torch_compile_cache/torch_aot_compile/790c848470ff514e07f3c1414211a7322a44a3ee6ee07ed6d185e5bfcc7f316b/rank_0_0/model
(EngineCore pid=57211) INFO 08-27 15:21:57 [monitor.py:53] torch.compile took 51.96 s in total
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] EngineCore failed to start.
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] Traceback (most recent call last):
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2893, in _run_ninja_build
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     subprocess.run(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/subprocess.py", line 571, in run
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     raise CalledProcessError(retcode, process.args,
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] The above exception was the direct cause of the following exception:
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] Traceback (most recent call last):
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     super().__init__(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 143, in __init__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 293, in _initialize_kv_caches
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 92, in collective_rpc
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 504, in determine_available_memory
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     self.model_runner.profile_run()
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6493, in profile_run
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6156, in _dummy_run
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     outputs = self.model(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]               ^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.runnable(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self._call_impl(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return forward_call(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 559, in forward
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     hidden_states = self.language_model.model(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 670, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     output = self.aot_compiled_fn(self, *args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/aot_compile.py", line 244, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 646, in forward
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     def forward(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/caching.py", line 217, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.optimized_call(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "<string>", line 277, in execution_fn
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.runnable(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/piecewise_backend.py", line 380, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return range_entry.runnable(*args)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_inductor/standalone_compile.py", line 321, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.inner_fn(*args)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/aot_compile_types.py", line 221, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.compiled_fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1277, in forward
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return compiled_fn(full_args)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1141, in runtime_wrapper
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     result = _codegen_runtime_wrapper(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(runtime_wrapper_orchestration)", line 8, in _runtime_wrapper
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2548, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.compiled_fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 763, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self.current_callable(inputs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/.cache/vllm/torch_compile_cache/torch_aot_compile/790c848470ff514e07f3c1414211a7322a44a3ee6ee07ed6d185e5bfcc7f316b/inductor_cache/qy/cqydeviygsfek3mfzmgsfkm4ptu4uvd6aarrqswrmvkftlzbo47w.py", line 750, in call
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     buf5 = torch.ops.humming.humming_gemm.default('{"shape_n": 16384, "shape_k": 5120, "pad_shape_n": 0, "pad_shape_k": 0, "num_experts": 0, "b_dtype": "uint3", "a_dtype": "bfloat16", "c_dtype": "bfloat16", "bs_dtype": "bfloat16", "input_scale_group_size": 0, "weight_scale_group_size": 128, "weight_scale_group_size_n": 0, "weight_scale_type": "group", "use_int_weight_scale": false, "use_fused_e8m0_scale": false, "has_zero_point": false, "is_fp_zero_point": false, "has_bias": false, "mma_type": "mma", "sublayer_name": ""}', '{"use_batch_invariant": false, "use_f16_accum": false, "gemm_type": "dense"}', None, buf4, arg6_1, None, None, arg7_1, None, None, None, None, None, None, None, arg8_1, top_k=1, valid_shape_m=0)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_ops.py", line 875, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self._op(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_compile.py", line 54, in inner
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return disable_fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return fn(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 432, in __torch_dispatch__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     res = func(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_ops.py", line 875, in __call__
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return self._op(*args, **kwargs)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/__init__.py", line 88, in humming_gemm
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     configs = HummingKernel.prepare_kernels(layer_config, compute_config, tuning_config)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/kernel/humming.py", line 424, in prepare_kernels
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     kernel.load_cubin()
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/kernel/humming.py", line 194, in load_cubin
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     self.kernel_id = ops.register_kernel(kernel_filename, kernel_name)
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/__init__.py", line 24, in register_kernel
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     init_humming_launcher()
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/utils.py", line 119, in init_humming_launcher
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     torch.utils.cpp_extension.load(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1927, in load
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     return _jit_compile(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]            ^^^^^^^^^^^^^
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2382, in _jit_compile
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     _write_ninja_file_and_build_library(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2543, in _write_ninja_file_and_build_library
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     _run_ninja_build(
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2910, in _run_ninja_build
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]     raise RuntimeError(message) from e
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] RuntimeError: Error building extension 'humming_launcher': [1/1] c++ launcher.o -shared -lcuda -lc10_cuda -ltorch_cuda -L/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o humming_launcher.so
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] FAILED: [code=1] humming_launcher.so
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] c++ launcher.o -shared -lcuda -lc10_cuda -ltorch_cuda -L/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o humming_launcher.so
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] /usr/bin/ld: cannot find -lcuda: No such file or directory
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] collect2: error: ld returned 1 exit status
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349] ninja: build stopped: subcommand failed.
(EngineCore pid=57211) ERROR 08-27 15:21:59 [core.py:1349]
(EngineCore pid=57211) Process EngineCore:
(EngineCore pid=57211) Traceback (most recent call last):
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2893, in _run_ninja_build
(EngineCore pid=57211)     subprocess.run(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/subprocess.py", line 571, in run
(EngineCore pid=57211)     raise CalledProcessError(retcode, process.args,
(EngineCore pid=57211) subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
(EngineCore pid=57211)
(EngineCore pid=57211) The above exception was the direct cause of the following exception:
(EngineCore pid=57211)
(EngineCore pid=57211) Traceback (most recent call last):
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=57211)     self.run()
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=57211)     self._target(*self._args, **self._kwargs)
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1353, in run_engine_core
(EngineCore pid=57211)     raise e
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=57211)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=57211)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=57211)     return func(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=57211)     super().__init__(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 143, in __init__
(EngineCore pid=57211)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=57211)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=57211)     return func(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 293, in _initialize_kv_caches
(EngineCore pid=57211)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=57211)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=57211)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 92, in collective_rpc
(EngineCore pid=57211)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=57211)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=57211)     return func(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=57211)     return func(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 504, in determine_available_memory
(EngineCore pid=57211)     self.model_runner.profile_run()
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6493, in profile_run
(EngineCore pid=57211)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=57211)                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=57211)     return func(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6156, in _dummy_run
(EngineCore pid=57211)     outputs = self.model(
(EngineCore pid=57211)               ^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=57211)     return self.runnable(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
(EngineCore pid=57211)     return self._call_impl(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
(EngineCore pid=57211)     return forward_call(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 559, in forward
(EngineCore pid=57211)     hidden_states = self.language_model.model(
(EngineCore pid=57211)                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 670, in __call__
(EngineCore pid=57211)     output = self.aot_compiled_fn(self, *args, **kwargs)
(EngineCore pid=57211)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/aot_compile.py", line 244, in __call__
(EngineCore pid=57211)     return self.fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 646, in forward
(EngineCore pid=57211)     def forward(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/caching.py", line 217, in __call__
(EngineCore pid=57211)     return self.optimized_call(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "<string>", line 277, in execution_fn
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=57211)     return self.runnable(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/compilation/piecewise_backend.py", line 380, in __call__
(EngineCore pid=57211)     return range_entry.runnable(*args)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_inductor/standalone_compile.py", line 321, in __call__
(EngineCore pid=57211)     return self.inner_fn(*args)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/aot_compile_types.py", line 221, in __call__
(EngineCore pid=57211)     return self.compiled_fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
(EngineCore pid=57211)     return fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1277, in forward
(EngineCore pid=57211)     return compiled_fn(full_args)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1141, in runtime_wrapper
(EngineCore pid=57211)     result = _codegen_runtime_wrapper(
(EngineCore pid=57211)              ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(runtime_wrapper_orchestration)", line 8, in _runtime_wrapper
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2548, in __call__
(EngineCore pid=57211)     return self.compiled_fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 763, in __call__
(EngineCore pid=57211)     return self.current_callable(inputs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/.cache/vllm/torch_compile_cache/torch_aot_compile/790c848470ff514e07f3c1414211a7322a44a3ee6ee07ed6d185e5bfcc7f316b/inductor_cache/qy/cqydeviygsfek3mfzmgsfkm4ptu4uvd6aarrqswrmvkftlzbo47w.py", line 750, in call
(EngineCore pid=57211)     buf5 = torch.ops.humming.humming_gemm.default('{"shape_n": 16384, "shape_k": 5120, "pad_shape_n": 0, "pad_shape_k": 0, "num_experts": 0, "b_dtype": "uint3", "a_dtype": "bfloat16", "c_dtype": "bfloat16", "bs_dtype": "bfloat16", "input_scale_group_size": 0, "weight_scale_group_size": 128, "weight_scale_group_size_n": 0, "weight_scale_type": "group", "use_int_weight_scale": false, "use_fused_e8m0_scale": false, "has_zero_point": false, "is_fp_zero_point": false, "has_bias": false, "mma_type": "mma", "sublayer_name": ""}', '{"use_batch_invariant": false, "use_f16_accum": false, "gemm_type": "dense"}', None, buf4, arg6_1, None, None, arg7_1, None, None, None, None, None, None, None, arg8_1, top_k=1, valid_shape_m=0)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_ops.py", line 875, in __call__
(EngineCore pid=57211)     return self._op(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_compile.py", line 54, in inner
(EngineCore pid=57211)     return disable_fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
(EngineCore pid=57211)     return fn(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 432, in __torch_dispatch__
(EngineCore pid=57211)     res = func(*args, **kwargs)
(EngineCore pid=57211)           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/_ops.py", line 875, in __call__
(EngineCore pid=57211)     return self._op(*args, **kwargs)
(EngineCore pid=57211)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/__init__.py", line 88, in humming_gemm
(EngineCore pid=57211)     configs = HummingKernel.prepare_kernels(layer_config, compute_config, tuning_config)
(EngineCore pid=57211)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/kernel/humming.py", line 424, in prepare_kernels
(EngineCore pid=57211)     kernel.load_cubin()
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/kernel/humming.py", line 194, in load_cubin
(EngineCore pid=57211)     self.kernel_id = ops.register_kernel(kernel_filename, kernel_name)
(EngineCore pid=57211)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/__init__.py", line 24, in register_kernel
(EngineCore pid=57211)     init_humming_launcher()
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/humming/ops/utils.py", line 119, in init_humming_launcher
(EngineCore pid=57211)     torch.utils.cpp_extension.load(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1927, in load
(EngineCore pid=57211)     return _jit_compile(
(EngineCore pid=57211)            ^^^^^^^^^^^^^
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2382, in _jit_compile
(EngineCore pid=57211)     _write_ninja_file_and_build_library(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2543, in _write_ninja_file_and_build_library
(EngineCore pid=57211)     _run_ninja_build(
(EngineCore pid=57211)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2910, in _run_ninja_build
(EngineCore pid=57211)     raise RuntimeError(message) from e
(EngineCore pid=57211) RuntimeError: Error building extension 'humming_launcher': [1/1] c++ launcher.o -shared -lcuda -lc10_cuda -ltorch_cuda -L/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o humming_launcher.so
(EngineCore pid=57211) FAILED: [code=1] humming_launcher.so
(EngineCore pid=57211) c++ launcher.o -shared -lcuda -lc10_cuda -ltorch_cuda -L/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o humming_launcher.so
(EngineCore pid=57211) /usr/bin/ld: cannot find -lcuda: No such file or directory
(EngineCore pid=57211) collect2: error: ld returned 1 exit status
(EngineCore pid=57211) ninja: build stopped: subcommand failed.
(EngineCore pid=57211)
[rank0]:[W827 15:22:00.342500827 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=57076) INFO 08-27 15:22:02 [utils.py:612] [shutdown] Process manager: send sigterm to process EngineCore
(APIServer pid=57076) Traceback (most recent call last):
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/bin/vllm", line 10, in <module>
(APIServer pid=57076)     sys.exit(main())
(APIServer pid=57076)              ^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 97, in main
(APIServer pid=57076)     args.dispatch_function(args)
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 152, in cmd
(APIServer pid=57076)     uvloop.run(run_server(args))
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=57076)     return __asyncio.run(
(APIServer pid=57076)            ^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=57076)     return runner.run(main)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=57076)     return self._loop.run_until_complete(task)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=57076)     return await main
(APIServer pid=57076)            ^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 764, in run_server
(APIServer pid=57076)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 778, in run_server_worker
(APIServer pid=57076)     async with build_async_engine_client(
(APIServer pid=57076)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=57076)     return await anext(self.gen)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 132, in build_async_engine_client
(APIServer pid=57076)     async with build_async_engine_client_from_engine_args(
(APIServer pid=57076)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=57076)     return await anext(self.gen)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 168, in build_async_engine_client_from_engine_args
(APIServer pid=57076)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=57076)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config
(APIServer pid=57076)     return cls(
(APIServer pid=57076)            ^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 149, in __init__
(APIServer pid=57076)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=57076)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=57076)     return func(*args, **kwargs)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client
(APIServer pid=57076)     return AsyncMPClient(*client_args)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=57076)     return func(*args, **kwargs)
(APIServer pid=57076)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 987, in __init__
(APIServer pid=57076)     super().__init__(
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 609, in __init__
(APIServer pid=57076)     with launch_core_engines(
(APIServer pid=57076)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=57076)     next(self.gen)
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1194, in launch_core_engines
(APIServer pid=57076)     wait_for_engine_startup(
(APIServer pid=57076)   File "/home/wuxiaowei/miniconda3/envs/vllm_py312/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1253, in wait_for_engine_startup
(APIServer pid=57076)     raise RuntimeError(
(APIServer pid=57076) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 29 G /Xwayland N/A |
+-----------------------------------------------------------------------------------------+

anm2211

IST Austria Distributed Algorithms and Systems Lab org about 21 hours ago

Thanks for testing it! We haven’t run a full code-agent benchmark suite (e.g. SWE-bench) on this checkpoint yet. So a proper agentic/coding evaluation would definitely be useful.

Regarding your error: the model itself actually loads successfully. The failure happens afterwards when Humming JIT-compiles its CUDA launcher:

/usr/bin/ld: cannot find -lcuda

This looks like a WSL CUDA library-path issue. Since you appear to be running under WSL, could you try:

ls -l /usr/lib/wsl/lib/libcuda.so*
export LIBRARY_PATH=/usr/lib/wsl/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

If /usr/lib/wsl/lib contains only libcuda.so.1 but no libcuda.so, you can make a userspace linker directory without modifying WSL's read-only system directory:

mkdir -p ~/.local/cuda-lib
ln -sf /usr/lib/wsl/lib/libcuda.so.1 ~/.local/cuda-lib/libcuda.so

export LIBRARY_PATH=$HOME/.local/cuda-lib:/usr/lib/wsl/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/.local/cuda-lib:/usr/lib/wsl/lib:$LD_LIBRARY_PATH

Then restart vllm serve.

modderbug

about 6 hours ago

its worked !

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment