Instructions to use PiehSoft/Qwen3.6-40B-Deckard-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Use Docker

docker model run hf.co/PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

LM Studio
Jan

vLLM

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PiehSoft/Qwen3.6-40B-Deckard-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PiehSoft/Qwen3.6-40B-Deckard-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Ollama
How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Ollama:
```
ollama run hf.co/PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
```

Unsloth Studio

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for PiehSoft/Qwen3.6-40B-Deckard-MTP to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for PiehSoft/Qwen3.6-40B-Deckard-MTP to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for PiehSoft/Qwen3.6-40B-Deckard-MTP to start chatting

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Docker Model Runner:
```
docker model run hf.co/PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M
```

Lemonade

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-40B-Deckard-MTP-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use PiehSoft/Qwen3.6-40B-Deckard-MTP with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default PiehSoft/Qwen3.6-40B-Deckard-MTP:Q4_K_M

Run Hermes

hermes

Atomic Chat

I can't get it to load with llama.cpp

by sa13ma - opened 20 days ago

Discussion

sa13ma

20 days ago

I have a 4x3060 (12GB) and 64GB DRAM
I can't load either Q6 or Q5 even with 131K context ..
On this HW -- I can load DaviAu's model ( Q6 with 131K context )
On this HW I can run Unsloth/Qwen3.6-27B Q6 with 262K context -- I have run Qwen-AgentWorld with 262K context also ( Q4_K_M)
Something is very wrong in the way it is computing memory needs -- maybe you can point out what else has to be configured -- but I never had any issues with other models on this setup.

WTPieh

PiehSoft LLC org 20 days ago

Hi there. Do you think you could go ahead and dump the logs that llama.cpp prints? I would turn on the verbosity logging, so I can see what it's printing out for memory allocation. Also, can you tell me what version of llama.cpp you're on?

sa13ma

20 days ago

I normally run it with lv 4 -- it's docker version of llama.cpp -- I don't like to compile -- I tried many different splits on the gpus --and no matter what I tried -- I couldn't load it .. and frankly loading leass than 131K context on 48GB total VRAM is not very interesting >>> here is the log with Q6 --docker compose up
[+] up 1/1
✔ Container qw36-40b-mtp Recreated 0.2s
Attaching to qw36-40b-mtp
qw36-40b-mtp | 0.00.730.037 I cmn common_param: common_params_print_info: build 9894 (a8cfdbb9e) with GNU 14.2.0 for Linux x86_64
qw36-40b-mtp | 0.00.730.040 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the -lv N CLI arg)
qw36-40b-mtp | 0.00.730.041 I cmn common_param: device_info:
qw36-40b-mtp | 0.00.829.224 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.00.940.812 I cmn common_param: - CUDA1 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.01.050.156 I cmn common_param: - CUDA2 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.01.156.657 I cmn common_param: - CUDA3 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.01.156.666 I cmn common_param: - CPU : Intel(R) Core(TM) i5-9600 CPU @ 3.10GHz (63955 MiB, 63955 MiB free)
qw36-40b-mtp | 0.01.156.726 I cmn common_param: system_info: n_threads = 6 (n_threads_batch = 6) / 6 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
qw36-40b-mtp | 0.01.156.731 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
qw36-40b-mtp | 0.01.156.787 I srv init: running without SSL
qw36-40b-mtp | 0.01.156.843 I srv init: using 8 threads for HTTP server
qw36-40b-mtp | 0.01.157.385 W srv llama_server: -----------------
qw36-40b-mtp | 0.01.157.387 W srv llama_server: Built-in tools are enabled, do not expose server to untrusted environments
qw36-40b-mtp | 0.01.157.387 W srv llama_server: This feature is EXPERIMENTAL and may be changed in the future
qw36-40b-mtp | 0.01.157.388 W srv llama_server: -----------------
qw36-40b-mtp | 0.01.157.564 I srv start: binding port with default address family
qw36-40b-mtp | 0.01.158.743 I srv load_model: loading model 'PiehSoft/Qwen3.6-40B-Deckard-MTP:Q6_K'
qw36-40b-mtp | 0.01.158.746 I srv load_model: local path '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'
qw36-40b-mtp | 0.01.235.828 I llama_model_loader: loaded meta data with 44 key-value pairs and 1290 tensors from /root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf (version GGUF V3 (latest))
qw36-40b-mtp | 0.01.235.882 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
qw36-40b-mtp | 0.01.235.891 I llama_model_loader: - kv 0: general.architecture str = qwen35
qw36-40b-mtp | 0.01.235.891 I llama_model_loader: - kv 1: general.type str = model
qw36-40b-mtp | 0.01.235.893 I llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
qw36-40b-mtp | 0.01.235.897 I llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
qw36-40b-mtp | 0.01.235.898 I llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
qw36-40b-mtp | 0.01.235.900 I llama_model_loader: - kv 5: general.name str = Qwen3.6 Vl 40B Deckard Uncensored Cla...
qw36-40b-mtp | 0.01.235.901 I llama_model_loader: - kv 6: general.finetune str = Deckard-Uncensored-ClaudeOpus250x
qw36-40b-mtp | 0.01.235.901 I llama_model_loader: - kv 7: general.basename str = Qwen3.6-vl
qw36-40b-mtp | 0.01.235.902 I llama_model_loader: - kv 8: general.size_label str = 40B
qw36-40b-mtp | 0.01.235.902 I llama_model_loader: - kv 9: qwen35.block_count u32 = 97
qw36-40b-mtp | 0.01.235.903 I llama_model_loader: - kv 10: qwen35.context_length u32 = 262144
qw36-40b-mtp | 0.01.235.903 I llama_model_loader: - kv 11: qwen35.embedding_length u32 = 5120
qw36-40b-mtp | 0.01.235.903 I llama_model_loader: - kv 12: qwen35.feed_forward_length u32 = 17408
qw36-40b-mtp | 0.01.235.904 I llama_model_loader: - kv 13: qwen35.attention.head_count u32 = 24
qw36-40b-mtp | 0.01.235.904 I llama_model_loader: - kv 14: qwen35.attention.head_count_kv u32 = 4
qw36-40b-mtp | 0.01.235.920 I llama_model_loader: - kv 15: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
qw36-40b-mtp | 0.01.235.921 I llama_model_loader: - kv 16: qwen35.rope.freq_base f32 = 10000000.000000
qw36-40b-mtp | 0.01.235.922 I llama_model_loader: - kv 17: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
qw36-40b-mtp | 0.01.235.922 I llama_model_loader: - kv 18: qwen35.attention.key_length u32 = 256
qw36-40b-mtp | 0.01.235.922 I llama_model_loader: - kv 19: qwen35.attention.value_length u32 = 256
qw36-40b-mtp | 0.01.235.923 I llama_model_loader: - kv 20: qwen35.ssm.conv_kernel u32 = 4
qw36-40b-mtp | 0.01.235.923 I llama_model_loader: - kv 21: qwen35.ssm.state_size u32 = 128
qw36-40b-mtp | 0.01.235.924 I llama_model_loader: - kv 22: qwen35.ssm.group_count u32 = 16
qw36-40b-mtp | 0.01.235.924 I llama_model_loader: - kv 23: qwen35.ssm.time_step_rank u32 = 48
qw36-40b-mtp | 0.01.235.924 I llama_model_loader: - kv 24: qwen35.ssm.inner_size u32 = 6144
qw36-40b-mtp | 0.01.235.925 I llama_model_loader: - kv 25: qwen35.full_attention_interval u32 = 4
qw36-40b-mtp | 0.01.235.925 I llama_model_loader: - kv 26: qwen35.rope.dimension_count u32 = 64
qw36-40b-mtp | 0.01.235.925 I llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2
qw36-40b-mtp | 0.01.235.926 I llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen35
qw36-40b-mtp | 0.01.264.528 I llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
qw36-40b-mtp | 0.01.272.317 I llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
qw36-40b-mtp | 0.01.297.678 I llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
qw36-40b-mtp | 0.01.297.682 I llama_model_loader: - kv 32: tokenizer.ggml.eos_token_id u32 = 248046
qw36-40b-mtp | 0.01.297.683 I llama_model_loader: - kv 33: tokenizer.ggml.padding_token_id u32 = 248044
qw36-40b-mtp | 0.01.297.683 I llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 248044
qw36-40b-mtp | 0.01.297.684 I llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = false
qw36-40b-mtp | 0.01.297.687 I llama_model_loader: - kv 36: tokenizer.chat_template str = {%- set image_count = namespace(value...
qw36-40b-mtp | 0.01.297.687 I llama_model_loader: - kv 37: general.quantization_version u32 = 2
qw36-40b-mtp | 0.01.297.688 I llama_model_loader: - kv 38: general.file_type u32 = 18
qw36-40b-mtp | 0.01.297.688 I llama_model_loader: - kv 39: quantize.imatrix.file str = w:/imx/Qwen3.6-40B-Deck-Opus-NEO-COD...
qw36-40b-mtp | 0.01.297.689 I llama_model_loader: - kv 40: quantize.imatrix.dataset str = c:/llamacpp/raw_imatrix/neo1-v2.txt
qw36-40b-mtp | 0.01.297.690 I llama_model_loader: - kv 41: quantize.imatrix.entries_count u32 = 744
qw36-40b-mtp | 0.01.297.690 I llama_model_loader: - kv 42: quantize.imatrix.chunks_count u32 = 342
qw36-40b-mtp | 0.01.297.690 I llama_model_loader: - kv 43: qwen35.nextn_predict_layers u32 = 1
qw36-40b-mtp | 0.01.297.695 I llama_model_loader: - type f32: 536 tensors
qw36-40b-mtp | 0.01.297.695 I llama_model_loader: - type q8_0: 145 tensors
qw36-40b-mtp | 0.01.297.695 I llama_model_loader: - type q6_K: 601 tensors
qw36-40b-mtp | 0.01.297.696 I llama_model_loader: - type bf16: 8 tensors
qw36-40b-mtp | 0.01.297.698 I print_info: file format = GGUF V3 (latest)
qw36-40b-mtp | 0.01.297.700 I print_info: file type = Q6_K
qw36-40b-mtp | 0.01.297.704 I print_info: file size = 30.95 GiB (6.73 BPW)
qw36-40b-mtp | 0.01.297.865 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3060) (0000:01:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.297.874 I llama_prepare_model_devices: using device CUDA1 (NVIDIA GeForce RTX 3060) (0000:02:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.297.896 I llama_prepare_model_devices: using device CUDA2 (NVIDIA GeForce RTX 3060) (0000:03:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.297.903 I llama_prepare_model_devices: using device CUDA3 (NVIDIA GeForce RTX 3060) (0000:04:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.478.319 I load: 0 unused tokens
qw36-40b-mtp | 0.01.525.854 I load: printing all EOG tokens:
qw36-40b-mtp | 0.01.525.858 I load: - 248044 ('<|endoftext|>')
qw36-40b-mtp | 0.01.525.858 I load: - 248046 ('<|im_end|>')
qw36-40b-mtp | 0.01.525.859 I load: - 248063 ('<|fim_pad|>')
qw36-40b-mtp | 0.01.525.859 I load: - 248064 ('<|repo_name|>')
qw36-40b-mtp | 0.01.525.860 I load: - 248065 ('<|file_sep|>')
qw36-40b-mtp | 0.01.526.636 I load: special tokens cache size = 33
qw36-40b-mtp | 0.01.618.713 I load: token to piece cache size = 1.7581 MB
qw36-40b-mtp | 0.01.618.732 I print_info: arch = qwen35
qw36-40b-mtp | 0.01.618.733 I print_info: vocab_only = 0
qw36-40b-mtp | 0.01.618.733 I print_info: no_alloc = 0
qw36-40b-mtp | 0.01.618.733 I print_info: n_ctx_train = 262144
qw36-40b-mtp | 0.01.618.734 I print_info: n_embd_inp = 5120
qw36-40b-mtp | 0.01.618.735 I print_info: n_embd = 5120
qw36-40b-mtp | 0.01.618.736 I print_info: n_embd_out = 5120
qw36-40b-mtp | 0.01.618.736 I print_info: n_layer = 96
qw36-40b-mtp | 0.01.618.736 I print_info: n_layer_all = 97
qw36-40b-mtp | 0.01.618.745 I print_info: n_head = 24
qw36-40b-mtp | 0.01.618.748 I print_info: n_head_kv = 4
qw36-40b-mtp | 0.01.618.748 I print_info: n_rot = 64
qw36-40b-mtp | 0.01.618.748 I print_info: n_swa = 0
qw36-40b-mtp | 0.01.618.749 I print_info: is_swa_any = 0
qw36-40b-mtp | 0.01.618.749 I print_info: n_embd_head_k = 256
qw36-40b-mtp | 0.01.618.749 I print_info: n_embd_head_v = 256
qw36-40b-mtp | 0.01.618.751 I print_info: n_gqa = 6
qw36-40b-mtp | 0.01.618.756 I print_info: n_embd_k_gqa = 1024
qw36-40b-mtp | 0.01.618.759 I print_info: n_embd_v_gqa = 1024
qw36-40b-mtp | 0.01.618.759 I print_info: f_norm_eps = 0.0e+00
qw36-40b-mtp | 0.01.618.761 I print_info: f_norm_rms_eps = 1.0e-06
qw36-40b-mtp | 0.01.618.761 I print_info: f_clamp_kqv = 0.0e+00
qw36-40b-mtp | 0.01.618.761 I print_info: f_max_alibi_bias = 0.0e+00
qw36-40b-mtp | 0.01.618.761 I print_info: f_logit_scale = 0.0e+00
qw36-40b-mtp | 0.01.618.761 I print_info: f_attn_scale = 0.0e+00
qw36-40b-mtp | 0.01.618.762 I print_info: f_attn_value_scale = 0.0000
qw36-40b-mtp | 0.01.618.764 I print_info: n_ff = 17408
qw36-40b-mtp | 0.01.618.764 I print_info: n_expert = 0
qw36-40b-mtp | 0.01.618.764 I print_info: n_expert_used = 0
qw36-40b-mtp | 0.01.618.764 I print_info: n_expert_groups = 0
qw36-40b-mtp | 0.01.618.765 I print_info: n_group_used = 0
qw36-40b-mtp | 0.01.618.765 I print_info: causal attn = 1
qw36-40b-mtp | 0.01.618.765 I print_info: pooling type = -1
qw36-40b-mtp | 0.01.618.765 I print_info: rope type = 40
qw36-40b-mtp | 0.01.618.765 I print_info: rope scaling = linear
qw36-40b-mtp | 0.01.618.766 I print_info: freq_base_train = 10000000.0
qw36-40b-mtp | 0.01.618.767 I print_info: freq_scale_train = 1
qw36-40b-mtp | 0.01.618.767 I print_info: n_ctx_orig_yarn = 262144
qw36-40b-mtp | 0.01.618.767 I print_info: rope_yarn_log_mul = 0.0000
qw36-40b-mtp | 0.01.618.768 I print_info: rope_finetuned = unknown
qw36-40b-mtp | 0.01.618.768 I print_info: mrope sections = [11, 11, 10, 0]
qw36-40b-mtp | 0.01.618.768 I print_info: ssm_d_conv = 4
qw36-40b-mtp | 0.01.618.768 I print_info: ssm_d_inner = 6144
qw36-40b-mtp | 0.01.618.769 I print_info: ssm_d_state = 128
qw36-40b-mtp | 0.01.618.769 I print_info: ssm_dt_rank = 48
qw36-40b-mtp | 0.01.618.769 I print_info: ssm_n_group = 16
qw36-40b-mtp | 0.01.618.769 I print_info: ssm_dt_b_c_rms = 0
qw36-40b-mtp | 0.01.618.770 I print_info: model type = ?B
qw36-40b-mtp | 0.01.618.771 I print_info: model params = 39.50 B
qw36-40b-mtp | 0.01.618.771 I print_info: general.name = Qwen3.6 Vl 40B Deckard Uncensored ClaudeOpus250x
qw36-40b-mtp | 0.01.618.776 I print_info: vocab type = BPE
qw36-40b-mtp | 0.01.618.777 I print_info: n_vocab = 248320
qw36-40b-mtp | 0.01.618.778 I print_info: n_merges = 247587
qw36-40b-mtp | 0.01.618.778 I print_info: BOS token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.618.778 I print_info: EOS token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.618.778 I print_info: EOT token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.618.779 I print_info: PAD token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.618.779 I print_info: LF token = 198 'Ċ'
qw36-40b-mtp | 0.01.618.779 I print_info: FIM PRE token = 248060 '<|fim_prefix|>'
qw36-40b-mtp | 0.01.618.779 I print_info: FIM SUF token = 248062 '<|fim_suffix|>'
qw36-40b-mtp | 0.01.618.780 I print_info: FIM MID token = 248061 '<|fim_middle|>'
qw36-40b-mtp | 0.01.618.780 I print_info: FIM PAD token = 248063 '<|fim_pad|>'
qw36-40b-mtp | 0.01.618.780 I print_info: FIM REP token = 248064 '<|repo_name|>'
qw36-40b-mtp | 0.01.618.780 I print_info: FIM SEP token = 248065 '<|file_sep|>'
qw36-40b-mtp | 0.01.618.780 I print_info: EOG token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.618.781 I print_info: EOG token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.618.781 I print_info: EOG token = 248063 '<|fim_pad|>'
qw36-40b-mtp | 0.01.618.781 I print_info: EOG token = 248064 '<|repo_name|>'
qw36-40b-mtp | 0.01.618.781 I print_info: EOG token = 248065 '<|file_sep|>'
qw36-40b-mtp | 0.01.618.781 I print_info: max token length = 256
qw36-40b-mtp | 0.01.618.783 I load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
qw36-40b-mtp | 0.01.655.111 I load_tensors: offloading output layer to GPU
qw36-40b-mtp | 0.01.655.116 I load_tensors: offloading 96 repeating layers to GPU
qw36-40b-mtp | 0.01.655.116 I load_tensors: offloaded 98/98 layers to GPU
qw36-40b-mtp | 0.01.655.123 I load_tensors: CPU_Mapped model buffer size = 994.63 MiB
qw36-40b-mtp | 0.01.655.124 I load_tensors: CUDA0 model buffer size = 8641.15 MiB
qw36-40b-mtp | 0.01.655.125 I load_tensors: CUDA1 model buffer size = 8341.04 MiB
qw36-40b-mtp | 0.01.655.126 I load_tensors: CUDA2 model buffer size = 8040.93 MiB
qw36-40b-mtp | 0.01.655.126 I load_tensors: CUDA3 model buffer size = 5673.13 MiB
qw36-40b-mtp | 0.24.683.448 I cmn common_init: added <|endoftext|> logit bias = -inf
qw36-40b-mtp | 0.24.683.452 I cmn common_init: added <|im_end|> logit bias = -inf
qw36-40b-mtp | 0.24.683.453 I cmn common_init_: added <|fim_pad|> logit bias = -inf
qw36-40b-mtp | 0.24.683.453 I cmn common_init_: added <|repo_name|> logit bias = -inf
qw36-40b-mtp | 0.24.683.454 I cmn common_init_: added <|file_sep|> logit bias = -inf
qw36-40b-mtp | 0.24.683.608 I llama_context: constructing llama_context
qw36-40b-mtp | 0.24.683.617 I llama_context: n_seq_max = 4
qw36-40b-mtp | 0.24.683.617 I llama_context: n_ctx = 262144
qw36-40b-mtp | 0.24.683.617 I llama_context: n_ctx_seq = 262144
qw36-40b-mtp | 0.24.683.617 I llama_context: n_batch = 2048
qw36-40b-mtp | 0.24.683.618 I llama_context: n_ubatch = 2048
qw36-40b-mtp | 0.24.683.618 I llama_context: causal_attn = 1
qw36-40b-mtp | 0.24.683.619 I llama_context: flash_attn = enabled
qw36-40b-mtp | 0.24.683.619 I llama_context: kv_unified = true
qw36-40b-mtp | 0.24.683.622 I llama_context: freq_base = 10000000.0
qw36-40b-mtp | 0.24.683.622 I llama_context: freq_scale = 1
qw36-40b-mtp | 0.24.683.622 I llama_context: n_rs_seq = 3
qw36-40b-mtp | 0.24.683.623 I llama_context: n_outputs_max = 16
qw36-40b-mtp | 0.24.686.769 I llama_context: CUDA_Host output buffer size = 3.79 MiB
qw36-40b-mtp | 0.24.691.469 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 3808.00 MiB on device 0: cudaMalloc failed: out of memory
qw36-40b-mtp | 0.24.691.478 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 3992977408
qw36-40b-mtp | 0.24.693.322 E llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
qw36-40b-mtp | 0.24.693.331 E cmn common_init_: failed to create context with model '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'
qw36-40b-mtp | 0.24.693.336 E cmn common_init_: failed to create context with model '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'
qw36-40b-mtp exited with code 139 (restarting)

The second Attemptu is with 131K docker compose up
[+] up 1/1
✔ Container qw36-40b-mtp Recreated 0.2s
Attaching to qw36-40b-mtp
qw36-40b-mtp | 0.00.677.702 I cmn common_param: common_params_print_info: build 9894 (a8cfdbb9e) with GNU 14.2.0 for Linux x86_64
qw36-40b-mtp | 0.00.677.706 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the -lv N CLI arg)
qw36-40b-mtp | 0.00.677.707 I cmn common_param: device_info:
qw36-40b-mtp | 0.00.771.323 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.00.872.986 I cmn common_param: - CUDA1 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.00.976.520 I cmn common_param: - CUDA2 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.01.079.702 I cmn common_param: - CUDA3 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
qw36-40b-mtp | 0.01.079.710 I cmn common_param: - CPU : Intel(R) Core(TM) i5-9600 CPU @ 3.10GHz (63955 MiB, 63955 MiB free)
qw36-40b-mtp | 0.01.079.772 I cmn common_param: system_info: n_threads = 6 (n_threads_batch = 6) / 6 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
qw36-40b-mtp | 0.01.079.777 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
qw36-40b-mtp | 0.01.079.805 I srv init: running without SSL
qw36-40b-mtp | 0.01.079.886 I srv init: using 8 threads for HTTP server
qw36-40b-mtp | 0.01.080.406 W srv llama_server: -----------------
qw36-40b-mtp | 0.01.080.408 W srv llama_server: Built-in tools are enabled, do not expose server to untrusted environments
qw36-40b-mtp | 0.01.080.409 W srv llama_server: This feature is EXPERIMENTAL and may be changed in the future
qw36-40b-mtp | 0.01.080.409 W srv llama_server: -----------------
qw36-40b-mtp | 0.01.080.623 I srv start: binding port with default address family
qw36-40b-mtp | 0.01.081.802 I srv load_model: loading model 'PiehSoft/Qwen3.6-40B-Deckard-MTP:Q6_K'
qw36-40b-mtp | 0.01.081.804 I srv load_model: local path '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'
qw36-40b-mtp | 0.01.159.527 I llama_model_loader: loaded meta data with 44 key-value pairs and 1290 tensors from /root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf (version GGUF V3 (latest))
qw36-40b-mtp | 0.01.159.579 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
qw36-40b-mtp | 0.01.159.592 I llama_model_loader: - kv 0: general.architecture str = qwen35
qw36-40b-mtp | 0.01.159.592 I llama_model_loader: - kv 1: general.type str = model
qw36-40b-mtp | 0.01.159.594 I llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
qw36-40b-mtp | 0.01.159.598 I llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
qw36-40b-mtp | 0.01.159.598 I llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
qw36-40b-mtp | 0.01.159.600 I llama_model_loader: - kv 5: general.name str = Qwen3.6 Vl 40B Deckard Uncensored Cla...
qw36-40b-mtp | 0.01.159.600 I llama_model_loader: - kv 6: general.finetune str = Deckard-Uncensored-ClaudeOpus250x
qw36-40b-mtp | 0.01.159.601 I llama_model_loader: - kv 7: general.basename str = Qwen3.6-vl
qw36-40b-mtp | 0.01.159.609 I llama_model_loader: - kv 8: general.size_label str = 40B
qw36-40b-mtp | 0.01.159.610 I llama_model_loader: - kv 9: qwen35.block_count u32 = 97
qw36-40b-mtp | 0.01.159.610 I llama_model_loader: - kv 10: qwen35.context_length u32 = 262144
qw36-40b-mtp | 0.01.159.611 I llama_model_loader: - kv 11: qwen35.embedding_length u32 = 5120
qw36-40b-mtp | 0.01.159.611 I llama_model_loader: - kv 12: qwen35.feed_forward_length u32 = 17408
qw36-40b-mtp | 0.01.159.611 I llama_model_loader: - kv 13: qwen35.attention.head_count u32 = 24
qw36-40b-mtp | 0.01.159.612 I llama_model_loader: - kv 14: qwen35.attention.head_count_kv u32 = 4
qw36-40b-mtp | 0.01.159.625 I llama_model_loader: - kv 15: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
qw36-40b-mtp | 0.01.159.626 I llama_model_loader: - kv 16: qwen35.rope.freq_base f32 = 10000000.000000
qw36-40b-mtp | 0.01.159.627 I llama_model_loader: - kv 17: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
qw36-40b-mtp | 0.01.159.627 I llama_model_loader: - kv 18: qwen35.attention.key_length u32 = 256
qw36-40b-mtp | 0.01.159.627 I llama_model_loader: - kv 19: qwen35.attention.value_length u32 = 256
qw36-40b-mtp | 0.01.159.628 I llama_model_loader: - kv 20: qwen35.ssm.conv_kernel u32 = 4
qw36-40b-mtp | 0.01.159.628 I llama_model_loader: - kv 21: qwen35.ssm.state_size u32 = 128
qw36-40b-mtp | 0.01.159.628 I llama_model_loader: - kv 22: qwen35.ssm.group_count u32 = 16
qw36-40b-mtp | 0.01.159.629 I llama_model_loader: - kv 23: qwen35.ssm.time_step_rank u32 = 48
qw36-40b-mtp | 0.01.159.629 I llama_model_loader: - kv 24: qwen35.ssm.inner_size u32 = 6144
qw36-40b-mtp | 0.01.159.629 I llama_model_loader: - kv 25: qwen35.full_attention_interval u32 = 4
qw36-40b-mtp | 0.01.159.630 I llama_model_loader: - kv 26: qwen35.rope.dimension_count u32 = 64
qw36-40b-mtp | 0.01.159.630 I llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2
qw36-40b-mtp | 0.01.159.630 I llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen35
qw36-40b-mtp | 0.01.187.589 I llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
qw36-40b-mtp | 0.01.195.097 I llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
qw36-40b-mtp | 0.01.220.716 I llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
qw36-40b-mtp | 0.01.220.721 I llama_model_loader: - kv 32: tokenizer.ggml.eos_token_id u32 = 248046
qw36-40b-mtp | 0.01.220.721 I llama_model_loader: - kv 33: tokenizer.ggml.padding_token_id u32 = 248044
qw36-40b-mtp | 0.01.220.722 I llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 248044
qw36-40b-mtp | 0.01.220.722 I llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = false
qw36-40b-mtp | 0.01.220.725 I llama_model_loader: - kv 36: tokenizer.chat_template str = {%- set image_count = namespace(value...
qw36-40b-mtp | 0.01.220.726 I llama_model_loader: - kv 37: general.quantization_version u32 = 2
qw36-40b-mtp | 0.01.220.726 I llama_model_loader: - kv 38: general.file_type u32 = 18
qw36-40b-mtp | 0.01.220.727 I llama_model_loader: - kv 39: quantize.imatrix.file str = w:/imx/Qwen3.6-40B-Deck-Opus-NEO-COD...
qw36-40b-mtp | 0.01.220.727 I llama_model_loader: - kv 40: quantize.imatrix.dataset str = c:/llamacpp/raw_imatrix/neo1-v2.txt
qw36-40b-mtp | 0.01.220.728 I llama_model_loader: - kv 41: quantize.imatrix.entries_count u32 = 744
qw36-40b-mtp | 0.01.220.728 I llama_model_loader: - kv 42: quantize.imatrix.chunks_count u32 = 342
qw36-40b-mtp | 0.01.220.728 I llama_model_loader: - kv 43: qwen35.nextn_predict_layers u32 = 1
qw36-40b-mtp | 0.01.220.733 I llama_model_loader: - type f32: 536 tensors
qw36-40b-mtp | 0.01.220.733 I llama_model_loader: - type q8_0: 145 tensors
qw36-40b-mtp | 0.01.220.734 I llama_model_loader: - type q6_K: 601 tensors
qw36-40b-mtp | 0.01.220.734 I llama_model_loader: - type bf16: 8 tensors
qw36-40b-mtp | 0.01.220.737 I print_info: file format = GGUF V3 (latest)
qw36-40b-mtp | 0.01.220.738 I print_info: file type = Q6_K
qw36-40b-mtp | 0.01.220.742 I print_info: file size = 30.95 GiB (6.73 BPW)
qw36-40b-mtp | 0.01.220.937 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3060) (0000:01:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.220.961 I llama_prepare_model_devices: using device CUDA1 (NVIDIA GeForce RTX 3060) (0000:02:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.220.983 I llama_prepare_model_devices: using device CUDA2 (NVIDIA GeForce RTX 3060) (0000:03:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.220.990 I llama_prepare_model_devices: using device CUDA3 (NVIDIA GeForce RTX 3060) (0000:04:00.0) - 11785 MiB free
qw36-40b-mtp | 0.01.404.853 I load: 0 unused tokens
qw36-40b-mtp | 0.01.450.006 I load: printing all EOG tokens:
qw36-40b-mtp | 0.01.450.011 I load: - 248044 ('<|endoftext|>')
qw36-40b-mtp | 0.01.450.011 I load: - 248046 ('<|im_end|>')
qw36-40b-mtp | 0.01.450.012 I load: - 248063 ('<|fim_pad|>')
qw36-40b-mtp | 0.01.450.012 I load: - 248064 ('<|repo_name|>')
qw36-40b-mtp | 0.01.450.012 I load: - 248065 ('<|file_sep|>')
qw36-40b-mtp | 0.01.450.772 I load: special tokens cache size = 33
qw36-40b-mtp | 0.01.541.686 I load: token to piece cache size = 1.7581 MB
qw36-40b-mtp | 0.01.541.704 I print_info: arch = qwen35
qw36-40b-mtp | 0.01.541.705 I print_info: vocab_only = 0
qw36-40b-mtp | 0.01.541.705 I print_info: no_alloc = 0
qw36-40b-mtp | 0.01.541.705 I print_info: n_ctx_train = 262144
qw36-40b-mtp | 0.01.541.706 I print_info: n_embd_inp = 5120
qw36-40b-mtp | 0.01.541.707 I print_info: n_embd = 5120
qw36-40b-mtp | 0.01.541.708 I print_info: n_embd_out = 5120
qw36-40b-mtp | 0.01.541.709 I print_info: n_layer = 96
qw36-40b-mtp | 0.01.541.709 I print_info: n_layer_all = 97
qw36-40b-mtp | 0.01.541.717 I print_info: n_head = 24
qw36-40b-mtp | 0.01.541.719 I print_info: n_head_kv = 4
qw36-40b-mtp | 0.01.541.719 I print_info: n_rot = 64
qw36-40b-mtp | 0.01.541.719 I print_info: n_swa = 0
qw36-40b-mtp | 0.01.541.720 I print_info: is_swa_any = 0
qw36-40b-mtp | 0.01.541.721 I print_info: n_embd_head_k = 256
qw36-40b-mtp | 0.01.541.721 I print_info: n_embd_head_v = 256
qw36-40b-mtp | 0.01.541.723 I print_info: n_gqa = 6
qw36-40b-mtp | 0.01.541.727 I print_info: n_embd_k_gqa = 1024
qw36-40b-mtp | 0.01.541.730 I print_info: n_embd_v_gqa = 1024
qw36-40b-mtp | 0.01.541.730 I print_info: f_norm_eps = 0.0e+00
qw36-40b-mtp | 0.01.541.732 I print_info: f_norm_rms_eps = 1.0e-06
qw36-40b-mtp | 0.01.541.732 I print_info: f_clamp_kqv = 0.0e+00
qw36-40b-mtp | 0.01.541.733 I print_info: f_max_alibi_bias = 0.0e+00
qw36-40b-mtp | 0.01.541.733 I print_info: f_logit_scale = 0.0e+00
qw36-40b-mtp | 0.01.541.733 I print_info: f_attn_scale = 0.0e+00
qw36-40b-mtp | 0.01.541.733 I print_info: f_attn_value_scale = 0.0000
qw36-40b-mtp | 0.01.541.735 I print_info: n_ff = 17408
qw36-40b-mtp | 0.01.541.735 I print_info: n_expert = 0
qw36-40b-mtp | 0.01.541.735 I print_info: n_expert_used = 0
qw36-40b-mtp | 0.01.541.735 I print_info: n_expert_groups = 0
qw36-40b-mtp | 0.01.541.736 I print_info: n_group_used = 0
qw36-40b-mtp | 0.01.541.736 I print_info: causal attn = 1
qw36-40b-mtp | 0.01.541.736 I print_info: pooling type = -1
qw36-40b-mtp | 0.01.541.736 I print_info: rope type = 40
qw36-40b-mtp | 0.01.541.736 I print_info: rope scaling = linear
qw36-40b-mtp | 0.01.541.752 I print_info: freq_base_train = 10000000.0
qw36-40b-mtp | 0.01.541.752 I print_info: freq_scale_train = 1
qw36-40b-mtp | 0.01.541.753 I print_info: n_ctx_orig_yarn = 262144
qw36-40b-mtp | 0.01.541.753 I print_info: rope_yarn_log_mul = 0.0000
qw36-40b-mtp | 0.01.541.753 I print_info: rope_finetuned = unknown
qw36-40b-mtp | 0.01.541.753 I print_info: mrope sections = [11, 11, 10, 0]
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_d_conv = 4
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_d_inner = 6144
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_d_state = 128
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_dt_rank = 48
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_n_group = 16
qw36-40b-mtp | 0.01.541.754 I print_info: ssm_dt_b_c_rms = 0
qw36-40b-mtp | 0.01.541.755 I print_info: model type = ?B
qw36-40b-mtp | 0.01.541.756 I print_info: model params = 39.50 B
qw36-40b-mtp | 0.01.541.756 I print_info: general.name = Qwen3.6 Vl 40B Deckard Uncensored ClaudeOpus250x
qw36-40b-mtp | 0.01.541.760 I print_info: vocab type = BPE
qw36-40b-mtp | 0.01.541.761 I print_info: n_vocab = 248320
qw36-40b-mtp | 0.01.541.761 I print_info: n_merges = 247587
qw36-40b-mtp | 0.01.541.761 I print_info: BOS token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.541.761 I print_info: EOS token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.541.762 I print_info: EOT token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.541.762 I print_info: PAD token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.541.762 I print_info: LF token = 198 'Ċ'
qw36-40b-mtp | 0.01.541.762 I print_info: FIM PRE token = 248060 '<|fim_prefix|>'
qw36-40b-mtp | 0.01.541.763 I print_info: FIM SUF token = 248062 '<|fim_suffix|>'
qw36-40b-mtp | 0.01.541.763 I print_info: FIM MID token = 248061 '<|fim_middle|>'
qw36-40b-mtp | 0.01.541.763 I print_info: FIM PAD token = 248063 '<|fim_pad|>'
qw36-40b-mtp | 0.01.541.763 I print_info: FIM REP token = 248064 '<|repo_name|>'
qw36-40b-mtp | 0.01.541.763 I print_info: FIM SEP token = 248065 '<|file_sep|>'
qw36-40b-mtp | 0.01.541.764 I print_info: EOG token = 248044 '<|endoftext|>'
qw36-40b-mtp | 0.01.541.764 I print_info: EOG token = 248046 '<|im_end|>'
qw36-40b-mtp | 0.01.541.764 I print_info: EOG token = 248063 '<|fim_pad|>'
qw36-40b-mtp | 0.01.541.764 I print_info: EOG token = 248064 '<|repo_name|>'
qw36-40b-mtp | 0.01.541.764 I print_info: EOG token = 248065 '<|file_sep|>'
qw36-40b-mtp | 0.01.541.765 I print_info: max token length = 256
qw36-40b-mtp | 0.01.541.766 I load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
qw36-40b-mtp | 0.01.578.039 I load_tensors: offloading output layer to GPU
qw36-40b-mtp | 0.01.578.045 I load_tensors: offloading 96 repeating layers to GPU
qw36-40b-mtp | 0.01.578.046 I load_tensors: offloaded 98/98 layers to GPU
qw36-40b-mtp | 0.01.578.053 I load_tensors: CPU_Mapped model buffer size = 994.63 MiB
qw36-40b-mtp | 0.01.578.054 I load_tensors: CUDA0 model buffer size = 8641.15 MiB
qw36-40b-mtp | 0.01.578.054 I load_tensors: CUDA1 model buffer size = 8341.04 MiB
qw36-40b-mtp | 0.01.578.055 I load_tensors: CUDA2 model buffer size = 7149.46 MiB
qw36-40b-mtp | 0.01.578.056 I load_tensors: CUDA3 model buffer size = 6564.60 MiB
qw36-40b-mtp | 0.24.589.124 I cmn common_init: added <|endoftext|> logit bias = -inf
qw36-40b-mtp | 0.24.589.127 I cmn common_init: added <|im_end|> logit bias = -inf
qw36-40b-mtp | 0.24.589.128 I cmn common_init_: added <|fim_pad|> logit bias = -inf
qw36-40b-mtp | 0.24.589.129 I cmn common_init_: added <|repo_name|> logit bias = -inf
qw36-40b-mtp | 0.24.589.129 I cmn common_init_: added <|file_sep|> logit bias = -inf
qw36-40b-mtp | 0.24.589.238 I llama_context: constructing llama_context
qw36-40b-mtp | 0.24.589.245 I llama_context: n_seq_max = 4
qw36-40b-mtp | 0.24.589.246 I llama_context: n_ctx = 131072
qw36-40b-mtp | 0.24.589.246 I llama_context: n_ctx_seq = 131072
qw36-40b-mtp | 0.24.589.246 I llama_context: n_batch = 2048
qw36-40b-mtp | 0.24.589.246 I llama_context: n_ubatch = 2048
qw36-40b-mtp | 0.24.589.246 I llama_context: causal_attn = 1
qw36-40b-mtp | 0.24.589.272 I llama_context: flash_attn = enabled
qw36-40b-mtp | 0.24.589.273 I llama_context: kv_unified = true
qw36-40b-mtp | 0.24.589.276 I llama_context: freq_base = 10000000.0
qw36-40b-mtp | 0.24.589.276 I llama_context: freq_scale = 1
qw36-40b-mtp | 0.24.589.277 I llama_context: n_rs_seq = 3
qw36-40b-mtp | 0.24.589.277 I llama_context: n_outputs_max = 16
qw36-40b-mtp | 0.24.589.277 I llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
qw36-40b-mtp | 0.24.592.268 I llama_context: CUDA_Host output buffer size = 3.79 MiB
qw36-40b-mtp | 0.24.593.740 I llama_kv_cache: CUDA0 KV buffer size = 1904.00 MiB
qw36-40b-mtp | 0.24.613.710 I llama_kv_cache: CUDA1 KV buffer size = 1904.00 MiB
qw36-40b-mtp | 0.24.629.135 I llama_kv_cache: CUDA2 KV buffer size = 1632.00 MiB
qw36-40b-mtp | 0.24.634.452 I llama_kv_cache: CUDA3 KV buffer size = 1088.00 MiB
qw36-40b-mtp | 0.24.637.813 I llama_kv_cache: size = 6528.00 MiB (131072 cells, 24 layers, 4/1 seqs), K (q8_0): 3264.00 MiB, V (q8_0): 3264.00 MiB
qw36-40b-mtp | 0.24.637.816 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
qw36-40b-mtp | 0.24.637.817 I llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
qw36-40b-mtp | 0.24.641.480 I llama_memory_recurrent: CUDA0 RS buffer size = 1097.25 MiB
qw36-40b-mtp | 0.24.644.960 I llama_memory_recurrent: CUDA1 RS buffer size = 1047.38 MiB
qw36-40b-mtp | 0.24.647.947 I llama_memory_recurrent: CUDA2 RS buffer size = 897.75 MiB
qw36-40b-mtp | 0.24.649.806 I llama_memory_recurrent: CUDA3 RS buffer size = 548.62 MiB
qw36-40b-mtp | 0.24.649.819 I llama_memory_recurrent: size = 3591.00 MiB ( 4 cells, 96 layers, 4 seqs 3 rs_seq), R (f32): 135.00 MiB, S (f32): 3456.00 MiB
qw36-40b-mtp | 0.24.649.881 I llama_context: pipeline parallelism enabled
qw36-40b-mtp | 0.24.649.883 I sched_reserve: reserving ...
qw36-40b-mtp | 0.24.684.290 I sched_reserve: resolving fused Gated Delta Net support:
qw36-40b-mtp | 0.24.687.306 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
qw36-40b-mtp | 0.24.690.851 I sched_reserve: fused Gated Delta Net (chunked) enabled
qw36-40b-mtp | 0.24.695.890 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 3041.31 MiB on device 0: cudaMalloc failed: out of memory
qw36-40b-mtp | 0.24.695.898 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 3189047808
qw36-40b-mtp | 0.24.695.899 E graph_reserve: failed to allocate compute buffers
qw36-40b-mtp | 0.24.695.906 W sched_reserve: compute buffer allocation failed, retrying without pipeline parallelism
qw36-40b-mtp | 0.24.703.369 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1344.33 MiB on device 0: cudaMalloc failed: out of memory
qw36-40b-mtp | 0.24.703.374 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 1409630336
qw36-40b-mtp | 0.24.703.375 E graph_reserve: failed to allocate compute buffers
qw36-40b-mtp | 0.24.719.501 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
qw36-40b-mtp | 0.24.719.510 E cmn common_init_: failed to create context with model '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'
qw36-40b-mtp | 0.24.719.514 E cmn common_init_: failed to create context with model '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q6_K.gguf'

WTPieh

PiehSoft LLC org 20 days ago

•

edited 20 days ago

Can you run your rig with this test and see if it loads well, and attach the logs? The MTP head isn’t free. It takes up its own KV cache allocation, and I don’t see anywhere in your logs to indicate that you were getting that allocation because I think it failed beforehand. from the logs, it seems like it’s trying to put everything onto CUDA zero, which then fails because it cannot have that continuous allocation for the contiguous blocks. I’m thinking a distribution split will probably be the better idea, but this is a hypothesis. The only difference between mine and DavidAU’s model is the MTP drafter head.

--parallel 1 --n-ubatch 512 --ctx-size 32768 -ts 0.7,1,1,1

sa13ma

20 days ago

I have been running 3x3060s and 2x5060 setups with llama.cpp for a while with a number of models (qwen3.6-27B-MTP .. also )
I have noticed that llama.cpp wants to allocate a lot of memory on the last GPU .. so I ended up doing -ts 1.1,1.1,1.1,0.7 to make it fit!
You are asking to do the reverse and I had tried it but with large context of 131K . .and it didn't load.
I tried many different flavors and none worked.
Ony thing that loaded it DavidAu model was not setting -ts at all -- and it allocated itself and loaded (with 131K context)
I think with small context of 32K i think it will load, I will give it a try .
-- ubatch I don't think is going to make a diff because it's not making it that far
Not knowing much when I looked at the numbers it was trying allocate and where, it seemed to me, it didn't care for any of the parameters, it was doing it's own thing - that's why I posted.

WTPieh

PiehSoft LLC org 20 days ago

Gotcha. Considering the fact that -ts auto distributes and that's the only thing that loads DaviAu's model, and mine won't load at any split, it's probably in the indicator here. It's looking like the MTP head is a fixed cost rather than the split you haven't tried yet, which is what I was hypothesizing.
Two things I'd like you to try real quick:

Run mine with no -ts, --ctx-size 32768 --parallel 1. Mirror exactly what worked for DavidAU, but just shrink the context size so it clears initialization. If it loads, the full memory breakdown will show you whether the drafter head is allocating its own KV cache. In my experience, I've seen it pull roughly 3.8 GB for the draft KV, but I'm unsure of what it looks like on other rigs. We can't confirm that it's allocating at all because it's dying before it even shows.
If it loads at 32,000 context, walk the context up and then tell me where it stops. The gap between where mine caps out and where DavidAU's runs clean on 131,000 context is probably going to be the head's footprint. You're right that the ubatch probably won't help because you're dying at graph reserve before it matters, but I figured it would at least reduce the footprint a little bit if you were trying to run it at that higher context.

sa13ma

20 days ago

I was able to load -- and upto 192K Ctx .. I think the --parallel 1 setting is the only one that mattered.
FWIW -- only breakdown of -ts 1,1.1,1.1,0.7 works -- there is still about 3.8GB of unused VRAM -- but the way differejnt parts are allocating is very suboptimal.
llama.cpp wants at least 4GB after weights are loaded -- most of it on last GPU -- then I think mtv wants about 4.5GB .. mostly on gpu0 -- this is very suboptimal. >> you can see the logs
I tried everything .. but could not get more than this loaded .. it's not nice .. because there is at least 3.8GB unused . maybe it's used for scratch, I don't know >>>>

qw36-40b-mtp | 0.01.972.488 I load_tensors: offloaded 98/98 layers to GPU
qw36-40b-mtp | 0.01.972.494 I load_tensors: CPU_Mapped model buffer size = 833.59 MiB
qw36-40b-mtp | 0.01.972.495 I load_tensors: CUDA0 model buffer size = 6757.96 MiB
qw36-40b-mtp | 0.01.972.496 I load_tensors: CUDA1 model buffer size = 6888.50 MiB
qw36-40b-mtp | 0.01.972.496 I load_tensors: CUDA2 model buffer size = 6888.50 MiB
qw36-40b-mtp | 0.01.972.497 I load_tensors: CUDA3 model buffer size = 5940.46 MiB
qw36-40b-mtp | 0.22.973.703 I cmn common_init_: added <|endoftext|> logit bias = -inf
qw36-40b-mtp | 0.22.973.708 I cmn common_init_: added <|im_end|> logit bias = -inf
qw36-40b-mtp | 0.22.973.709 I cmn common_init_: added <|fim_pad|> logit bias = -inf
qw36-40b-mtp | 0.22.973.710 I cmn common_init_: added <|repo_name|> logit bias = -inf
qw36-40b-mtp | 0.22.973.710 I cmn common_init_: added <|file_sep|> logit bias = -inf
qw36-40b-mtp | 0.22.973.832 I llama_context: constructing llama_context
qw36-40b-mtp | 0.22.973.838 I llama_context: n_seq_max = 1
qw36-40b-mtp | 0.22.973.839 I llama_context: n_ctx = 192000
qw36-40b-mtp | 0.22.973.839 I llama_context: n_ctx_seq = 192000
qw36-40b-mtp | 0.22.973.840 I llama_context: n_batch = 2048
qw36-40b-mtp | 0.22.973.841 I llama_context: n_ubatch = 2048
qw36-40b-mtp | 0.22.973.841 I llama_context: causal_attn = 1
qw36-40b-mtp | 0.22.973.842 I llama_context: flash_attn = enabled
qw36-40b-mtp | 0.22.973.842 I llama_context: kv_unified = false
qw36-40b-mtp | 0.22.973.845 I llama_context: freq_base = 10000000.0
qw36-40b-mtp | 0.22.973.845 I llama_context: freq_scale = 1
qw36-40b-mtp | 0.22.973.845 I llama_context: n_rs_seq = 3
qw36-40b-mtp | 0.22.973.846 I llama_context: n_outputs_max = 4
qw36-40b-mtp | 0.22.973.846 I llama_context: n_ctx_seq (192000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
qw36-40b-mtp | 0.22.976.960 I llama_context: CUDA_Host output buffer size = 0.95 MiB
qw36-40b-mtp | 0.22.978.941 I llama_kv_cache: CUDA0 KV buffer size = 2390.62 MiB
qw36-40b-mtp | 0.23.000.834 I llama_kv_cache: CUDA1 KV buffer size = 2789.06 MiB
qw36-40b-mtp | 0.23.022.674 I llama_kv_cache: CUDA2 KV buffer size = 2789.06 MiB
qw36-40b-mtp | 0.23.031.538 I llama_kv_cache: CUDA3 KV buffer size = 1593.75 MiB
qw36-40b-mtp | 0.23.036.447 I llama_kv_cache: size = 9562.50 MiB (192000 cells, 24 layers, 1/1 seqs), K (q8_0): 4781.25 MiB, V (q8_0): 4781.25 MiB
qw36-40b-mtp | 0.23.036.450 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
qw36-40b-mtp | 0.23.036.450 I llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
qw36-40b-mtp | 0.23.037.438 I llama_memory_recurrent: CUDA0 RS buffer size = 249.38 MiB
qw36-40b-mtp | 0.23.038.329 I llama_memory_recurrent: CUDA1 RS buffer size = 249.38 MiB
qw36-40b-mtp | 0.23.039.226 I llama_memory_recurrent: CUDA2 RS buffer size = 249.38 MiB
qw36-40b-mtp | 0.23.039.787 I llama_memory_recurrent: CUDA3 RS buffer size = 149.62 MiB
qw36-40b-mtp | 0.23.039.810 I llama_memory_recurrent: size = 897.75 MiB ( 1 cells, 96 layers, 1 seqs 3 rs_seq), R (f32): 33.75 MiB, S (f32): 864.00 MiB
qw36-40b-mtp | 0.23.039.868 I llama_context: pipeline parallelism enabled
qw36-40b-mtp | 0.23.039.872 I sched_reserve: reserving ...
qw36-40b-mtp | 0.23.074.644 I sched_reserve: resolving fused Gated Delta Net support:
qw36-40b-mtp | 0.23.077.804 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
qw36-40b-mtp | 0.23.081.484 I sched_reserve: fused Gated Delta Net (chunked) enabled
qw36-40b-mtp | 0.23.087.643 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4231.31 MiB on device 0: cudaMalloc failed: out of memory
qw36-40b-mtp | 0.23.087.651 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 4436853248
qw36-40b-mtp | 0.23.087.653 E graph_reserve: failed to allocate compute buffers
qw36-40b-mtp | 0.23.087.660 W sched_reserve: compute buffer allocation failed, retrying without pipeline parallelism
qw36-40b-mtp | 0.23.420.041 I sched_reserve: CUDA0 compute buffer size = 1820.33 MiB
qw36-40b-mtp | 0.23.420.047 I sched_reserve: CUDA1 compute buffer size = 1820.33 MiB
qw36-40b-mtp | 0.23.420.048 I sched_reserve: CUDA2 compute buffer size = 1820.33 MiB
qw36-40b-mtp | 0.23.420.049 I sched_reserve: CUDA3 compute buffer size = 1820.33 MiB
qw36-40b-mtp | 0.23.420.049 I sched_reserve: CUDA_Host compute buffer size = 830.34 MiB
qw36-40b-mtp | 0.23.420.050 I sched_reserve: graph nodes = 6415
qw36-40b-mtp | 0.23.420.050 I sched_reserve: graph splits = 5
qw36-40b-mtp | 0.23.420.053 I sched_reserve: reserve took 380.18 ms, sched copies = 1
qw36-40b-mtp | 0.23.420.165 I srv load_model: creating MTP draft context against the target model '/root/.cache/huggingface/hub/models--PiehSoft--Qwen3.6-40B-Deckard-MTP/snapshots/57db5e2e9ef3ef758add9585aeb21dc9e4021dd7/Qwen3.6-40B-Deckard-MTP-Q5_K_M.gguf'
qw36-40b-mtp | 0.23.420.209 I llama_context: constructing llama_context
qw36-40b-mtp | 0.23.420.212 I llama_context: n_seq_max = 1
qw36-40b-mtp | 0.23.420.212 I llama_context: n_ctx = 192000
qw36-40b-mtp | 0.23.420.212 I llama_context: n_ctx_seq = 192000
qw36-40b-mtp | 0.23.420.213 I llama_context: n_batch = 2048
qw36-40b-mtp | 0.23.420.214 I llama_context: n_ubatch = 2048
qw36-40b-mtp | 0.23.420.214 I llama_context: causal_attn = 1
qw36-40b-mtp | 0.23.420.215 I llama_context: flash_attn = enabled
qw36-40b-mtp | 0.23.420.215 I llama_context: kv_unified = false
qw36-40b-mtp | 0.23.420.216 I llama_context: freq_base = 10000000.0
qw36-40b-mtp | 0.23.420.217 I llama_context: freq_scale = 1
qw36-40b-mtp | 0.23.420.217 I llama_context: n_rs_seq = 0
qw36-40b-mtp | 0.23.420.217 I llama_context: n_outputs_max = 1
qw36-40b-mtp | 0.23.420.217 I llama_context: n_ctx_seq (192000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
qw36-40b-mtp | 0.23.420.452 I llama_context: CUDA_Host output buffer size = 0.95 MiB
qw36-40b-mtp | 0.23.425.016 I llama_kv_cache: CUDA3 KV buffer size = 750.00 MiB
qw36-40b-mtp | 0.23.427.356 I llama_kv_cache: size = 750.00 MiB (192000 cells, 1 layers, 1/1 seqs), K (f16): 375.00 MiB, V (f16): 375.00 MiB
qw36-40b-mtp | 0.23.427.361 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
qw36-40b-mtp | 0.23.427.362 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
qw36-40b-mtp | 0.23.427.437 I llama_context: pipeline parallelism enabled
qw36-40b-mtp | 0.23.427.439 I sched_reserve: reserving ...
qw36-40b-mtp | 0.23.462.872 I sched_reserve: resolving fused Gated Delta Net support:
qw36-40b-mtp | 0.23.463.025 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
qw36-40b-mtp | 0.23.464.273 I sched_reserve: fused Gated Delta Net (chunked) enabled
qw36-40b-mtp | 0.23.465.658 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 3808.25 MiB on device 3: cudaMalloc failed: out of memory
qw36-40b-mtp | 0.23.465.662 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA3 buffer of size 3993240064
qw36-40b-mtp | 0.23.465.663 E graph_reserve: failed to allocate compute buffers
qw36-40b-mtp | 0.23.465.671 W sched_reserve: compute buffer allocation failed, retrying without pipeline parallelism
qw36-40b-mtp | 0.23.782.102 I sched_reserve: CUDA3 compute buffer size = 1022.06 MiB
qw36-40b-mtp | 0.23.782.110 I sched_reserve: CUDA_Host compute buffer size = 830.07 MiB
qw36-40b-mtp | 0.23.782.111 I sched_reserve: graph nodes = 50
qw36-40b-mtp | 0.23.782.111 I sched_reserve: graph splits = 2
qw36-40b-mtp | 0.23.782.112 I sched_reserve: reserve took 354.67 ms, sched copies = 1
qw36-40b-mtp | 0.23.968.941 I cmn common_conte: the context supports bounded partial sequence removal
qw36-40b-mtp | 0.24.002.208 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 192000, kv_unified = 'false'
qw36-40b-mtp | 0.24.002.233 I spec common_specu: adding speculative implementation 'draft-mtp'
qw36-40b-mtp | 0.24.002.236 I spec common_specu: - n_max=3, n_min=0, p_min=0.75, n_embd=5120, backend_sampling=1
qw36-40b-mtp | 0.24.002.237 I spec common_specu: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
qw36-40b-mtp | 0.24.005.723 I sched_reserve: reserving ...
qw36-40b-mtp | 0.24.603.775 I sched_reserve: CUDA3 compute buffer size = 1022.06 MiB
qw36-40b-mtp | 0.24.603.783 I sched_reserve: CUDA_Host compute buffer size = 830.07 MiB
qw36-40b-mtp | 0.24.603.783 I sched_reserve: graph nodes = 56
qw36-40b-mtp | 0.24.603.783 I sched_reserve: graph splits = 2
qw36-40b-mtp | 0.24.603.784 I sched_reserve: reserve took 598.02 ms, sched copies = 1
qw36-40b-mtp | 0.24.617.021 I srv load_model: speculative decoding context initialized
qw36-40b-mtp | 0.24.617.025 I slot load_model: id 0 | task -1 | new slot, n_ctx = 192000
qw36-40b-mtp | 0.24.617.168 I srv load_model: prompt cache is enabled, size limit: 32768 MiB
qw36-40b-mtp | 0.24.617.170 I srv load_model: use --cache-ram 0 to disable the prompt cache
qw36-40b-mtp | 0.24.617.171 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
qw36-40b-mtp | 0.24.617.171 I srv load_model: context checkpoints enabled, max = 64, min spacing = 8192
qw36-40b-mtp | 0.24.617.194 I srv init: idle slots will be saved to prompt cache upon starting a new task
qw36-40b-mtp | 0.24.636.101 I srv init: init: chat template, example_format: '<|im_start|>system
qw36-40b-mtp | You are a helpful assistant<|im_end|>
qw36-40b-mtp | <|im_start|>user
qw36-40b-mtp | Hello<|im_end|>
qw36-40b-mtp | <|im_start|>assistant
qw36-40b-mtp |

WTPieh changed discussion status to closed 11 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment