Instructions to use ubergarm/Qwen3-Coder-Next-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ubergarm/Qwen3-Coder-Next-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Use Docker

docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use ubergarm/Qwen3-Coder-Next-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/Qwen3-Coder-Next-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/Qwen3-Coder-Next-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Ollama
How to use ubergarm/Qwen3-Coder-Next-GGUF with Ollama:
```
ollama run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
```

Unsloth Studio

How to use ubergarm/Qwen3-Coder-Next-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

How to use ubergarm/Qwen3-Coder-Next-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ubergarm/Qwen3-Coder-Next-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ubergarm/Qwen3-Coder-Next-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
```

Lemonade

How to use ubergarm/Qwen3-Coder-Next-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Run and chat with the model

lemonade run user.Qwen3-Coder-Next-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use ubergarm/Qwen3-Coder-Next-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ubergarm/Qwen3-Coder-Next-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ubergarm/Qwen3-Coder-Next-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Quant requests?

by ubergarm - opened Feb 22

Discussion

ubergarm

Owner Feb 22

•

edited Feb 22

As qwen3next architecture support only landed in ik_llama.cpp more recently along with qwen35moe, I haven't done as many quants here for Qwen3-Coder-Next.

If there is a target size in which you're interested e.g. full offload on 32GB 5090 etc, let me know. This model seems pretty fast with long context which is nice for a zippy local vibe coding experience for more simple code changes and workloads.

I've not been able to get down below ~2.3bpw or so using imatrix as it gives an error. It seems to be able to handle ffn_down_exps but explodes on ffn_gate_exps when providing imatrix, and without imatrix it throws that The result will be garbage, so bailing out. I could probably try to override that and force it to work without imatrix, or maybe go back and run a much larger corpus to collect more importance data as these Qwen models have been difficult for imatrix like this before. I've tested iq1_kt, iq2_kt, and iq1_s which all fail similarly.

[   9/ 843]                blk.0.ssm_norm.weight - [  128,     1,     1,     1], type =    f32, size =    0.000 MB
[  10/ 843]                 blk.0.ssm_out.weight - [ 4096,  2048,     1,     1], type =   bf16, Using custom type q8_0 for tensor blk.0.ssm_out.weight converting to q8_0 .. size =    16.00 MiB ->     8.50 MiB
[  11/ 843]           blk.0.ffn_down_exps.weight - [  512,  2048,   512,     1], type =   bf16, Using custom type iq1_kt for tensor blk.0.ffn_down_exps.weight converting to iq1_kt .. size =  1024.00 MiB ->   116.00 MiB
[  12/ 843]           blk.0.ffn_gate_exps.weight - [ 2048,   512,   512,     1], type =   bf16, Using custom type iq1_kt for tensor blk.0.ffn_gate_exps.weight converting to iq1_kt .. Oops: jbest = -1 for cluster 98 with 1160 points
/home/w/projects/ik_llama.cpp/ggml/src/iqk/iqk_quantize.cpp:8334: GGML_ASSERT(false) failed
Oops: jbest = -1 for cluster 162 with 1224 points
/home/w/projects/ik_llama.cpp/ggml/src/iqk/iqk_quantize.cpp:8334: GGML_ASSERT(false) failed
Oops: jbest = -1 for cluster 157 with 872 points

rhinocerosowllegolas

Feb 23

My setup’s a bit odd 😅 I’m running an M4 Pro MacBook with 24GB of RAM, and I’m using ik_cpp (CPU-only) because it lets me load larger models without pushing memory pressure into the yellow (and hitting swap).

Also, thanks so much for the quants. I really appreciate the work you put into them. I’m going to try out both versions and see how they perform on my end!

rhinocerosowllegolas

Feb 23

By the way, are you planning to release Qwen Next Instruct/Thinking quants as well? I’d definitely be interested in trying those out if you do.

ubergarm

Owner Feb 23

Thanks curious to hear how they work out for you, I know the smaller one is probably a touch big for 24GB. Does it have some kinda eGPU too or 24GB total unified memory or something?

By the way, are you planning to release Qwen Next Instruct/Thinking quants as well?

Do you specifically mean these two: https://huggingface.co/collections/Qwen/qwen3-next ?

I was not considering it given there are quite a few available already and I haven't heard as much talk about them, but they might be good candidates for lower RAM+VRAM systems for sure! I'll see if anyone else did ik_llama.cpp quants for them yet or not and think about it

Dsturb

Feb 23

Hi ubergarm,

https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fqwen3-coder-next-oddly-usable-at-aggressive-quantization-v0-q9q4nsw11rkg1.png%3Fwidth%3D3200%26format%3Dpng%26auto%3Dwebp%26s%3D5932e14267173413e275e01539ae4d848ee99077

Could you have a look here? In fact a 32GB version that beats or is similar with IQ3_XXS would be great. Currently I get around 3000 PP and 100-115 TG with partial offloading.

ubergarm

Owner Feb 23

@Dsturb

I assume you have a single 5090 32GB VRAM ? Yes, these recent qwen3next models seem to hold up quite well to quantization given the right recipes. I'll take a look.

@rhinocerosowllegolas

I'm not sure how mac does with turboderp's exllamav3 , but if it works there are probably some of the strongest low BPW EXL3 quants here: https://huggingface.co/turboderp/Qwen3-Next-80B-A3B-Instruct-exl3

igor255

Feb 23

IQ3_K would be good, sitting inbetween 2 quants provided.
I also have 5090, running about 1500 for prompt processing, and 50 token generation on 40gb quants.

igor255

Feb 23

I'm quite happy with IQ4_KSS you provided, getting 1200 prompt processing, 50 token generation on my PC. Quite usable for local coding agents. Thanks!

ubergarm

Owner Feb 23

@Dsturb @igor255

I did some fishing and came up with something very very tight for 32GB smol-IQ3_KS 30.728 GiB (3.313 BPW). I had to knock down to iq6_k for attn/ssm/shexp but still larger tensors than used by the UD. You can see the perplexity is looking good too on the graph: https://huggingface.co/ubergarm/Qwen3-Coder-Next-GGUF/blob/main/images/perplexity.png

igor255

Feb 24

•

edited Feb 24

Qwen3-Coder-Next-smol-IQ3_KS.gguf
Prompt 2142.12 t/s
Generation 79.97 t/s

Qwen3-Coder-Next-IQ4_KSS.gguf
Prompt 1280.50 t/s
Generation 50.66 t/s

My results, running with -c 65536 --n-cpu-moe {8 or 18} --no-mmap --no-warmup -ger --merge-qkv -ub 2048 -b 2048 --jinja -ngl 99 -fa on

Windows, Ryzen 5 5700x3d, 32gb ddr4, RTX 5090.

And additionally tested on linux, extra free vram gives a nice boost.
Qwen3-Coder-Next-smol-IQ3_KS.gguf
Prompt 3080 t/s
Generation 108 t/s

Qwen3-Coder-Next-IQ4_KSS.gguf
Prompt 1760 t/s
Generation 72.9 t/s

rhinocerosowllegolas

Feb 24

Hi! @ubergam
My system has 24 GB of unified memory, with roughly 10 GB used by the OS and apps — so about 14 GB available for inference.

For quantized models, I’m seeing:

~12 tokens/sec with IQ4_KSS
~20 tokens/sec with IQ3_KS

~64 tokens/sec PP for both.

I checked the Qwen next quants collection and couldn't find any ik_cpp-based quantized versions (e.g., IQ4_KSS, IQ3_KS, etc.) — which is why I reached out. I’m also pretty sure I can’t run exl3 quants efficiently on CPU alone (no GPU support).

Thanks anyway for the help — really appreciate it!

Let me know if you'd like help finding compatible quants or optimizing for CPU-only inference! (Rephrased by Qwen Coder ;])

gkubon

May 9

Requesting mainline q4 with mtp layers very much please! Current Q4 beats all new models for my usecases.

ubergarm

Owner May 11

@gkubon

If you're doing full GPU offload, also check out turboderp's exllamav3 with DFlash support as those EXL3 quants are mighty strong for the size: https://www.reddit.com/r/LocalLLaMA/comments/1t9voxs/exllamav3_major_updates/

Also, fwiw, unsloth amusingly released mainline compatible MTP quants for Qwen3.6's despite the mainline PR still being marked draft haha. won't be as good as ik quants but if you're stuck on mainline go for it.

gkubon

May 13

Still going 3090 + 64gb ddr5 (cpu-moe). Currently fishing for another gpu to offload fully and use graph split mode of ik_lllama. Qwen 3.5 and 3.6 works but lacks the quality of coder next for my work. Eager to try MTP speedups. Thanks for your work - its amazing!

ubergarm

Owner May 13

•

edited May 13

@gkubon

I'm surprised old Qwen3-Coder-Next is better for your workload than Qwen3.6-27B.

My daily driver with pi harness on single 3090TI full offload with 128k context including mmproj multimodal support and MTP is this one: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/blob/main/Qwen3.6-27B-MTP-IQ4_KS.gguf

I run it like this:

#!/usr/bin/env bash

model=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/Qwen3.6-27B-MTP-IQ4_KS.gguf
# https://huggingface.co/ggml-org/Qwen3.6-27B-GGUF/blob/main/mmproj-Qwen3.6-27B-Q8_0.gguf
mmproj=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/mmproj-Qwen3.6-27B-Q8_0.gguf

CUDA_VISIBLE_DEVICES="0" \
./build/bin/llama-server \
  --model "$model" \
  --alias "Qwen3.6-27B" \
  -c 131072 \
  -ctk q8_0 -ctv q8_0 \
  --merge-qkv \
  -muge \
  -ngl 99 \
  -t 1 \
  -tb 1 \
  --host 127.0.0.1 \
  --port 8080 \
  --parallel 1 \
  --jinja \
  --no-mmap \
  --ctx-checkpoints 32 \
  -cram 32768 \
  -mtp --draft-max 4 --draft-p-min 0.0 \
  --no-mmproj-offload \
  --mmproj "$mmproj" \
  --image-min-tokens 1024 \
  --image-max-tokens 4096

I have a local branch that enables -tm 16 so i can use 16 threads only when processing multimodal, otherwise 1 (best for full GPU offload typically). I may try to PR that we'll see.

I'm seeing ~85 tok/sec decode on coding generation, and maybe 1200+ pp on long enough prompts. With pi harness it is lite weight and even if it makes a mistkae now and then it keeps going very quickly if you give it some feedback (e.g. let it compile, use playwright to view html websites as mmproj image) etc.

gkubon

May 13

•

edited May 13

Thanks for sharing this!!

Reporting back same setup on single 3090 sample workload

  1 | Context | Prefill t/s | Decode t/s | Draft Acceptance |
  2 |---------|------------|------------|-----------------|
  3 | 24-33t  | 179-229    | 81-82      | 73-75%          |
  4 | 50-61t  | 293-298    | 40-57      | 30-61%          |
  5 | 100-112t| 412-464    | 58-70      | 67-88%          |
  6 | 160-232t| 521-601    | 47-54      | 48-61%          |
  7 | 260-311t| 551-671    | 45-59      | 48-50%          |
  8 | 517-575t| 712-807    | 46-50      | 44-50%          |
  9 | 776-989t| 762-816    | 51-65      | 50-74%          |
 10 | 1228-1278t| 772-797 | 45-56      | 46-64%          |
 11 | 1847t   | 763        | 51         | 61%             |
 12 | 2667t   | 747        | 50         | 60%             |
 13 | 5110t   | 868        | 50         | 53%             |
 14 | 5763t   | 815        | 48         | 56%             |
 15 | 5856t   | 858        | 50         | 55%             |
 16 | 6696t   | 789        | 53         | 65%             |
 17 | 12060t  | 1052       | 46         | 38%             |
 18 | 13279t  | 1129       | 52         | 42%             |
 19 | 26747t  | 1028       | 58         | 65%             |
 20 | 5913t*  | 310        | 76         | 70%             |

gkubon

May 14

•

edited May 14

@ubergarm

Gave it a day and it got stuck in loops few times. Attaching logs, now back to qwen-coder-next :P

LLM output at ~50k context

The user wants to run the build in foreground mode instead of background. Let me check if the previous background 
    shell already did this or if there are still 3 running builds. Let me use only 1 process this time and use 
    is_background=false this time.

    The issue was is_background=false.

    I see.
    The issue. Let's cancel all. The problem with the. I need. The problem: User said not. So: Use is_background. 
    But is_background is not is_background=true is not_background is not. User's current working: Wait the: No! Use!
     Wait is. Use background? Yes is is the same?
    Background the shell command in a file! is the the output in! So in this a process, that to background tasks the
     and use? But no not I'd stop use and a way that task task a it that tasks or with use no output can run or 
    shell run this, with, command it no shell commands stop
    So and background no task run use command commands tasks shell? tasks with not that! that are
    ! stop to command are can background or: Stop not this
    Background it commands no background I task are background with in or to to? Yes run but, task task but
    Use with with, command stop background but can: or and are that but or this background that use that
    to not tasks tasks background I shell it a! Stop is tasks stop stop: in it commands is I'll are shell tasks not?
     shell this but but run to to task no with with! tasks background with? This the can not task background
    it that background stop background that. that's or! So task use
    It are? but and commands stop to: The user run tasks task! shell tasks
    Background and. shell use use but background this background stop in but but! a with that stop that stop shell! 
    Use or this commands I commands: this: This it background commands
    but run is tasks
    Stop no are to tasks it stop with task task not not the tasks but task in is with with with that's task tasks 
    use not or.

    use shell no a and? this background can to to!

    but background that is
    Wait shell background! task shell stop run tasks stop to and
    this stop can that is can task! Stop? with stop commands with. tasks background the! tasks background commands 
    background commands background that a background not or background: But: No a a in it I are it background use
    in tasks run task and commands is commands with tasks task it this tasks to! No run tasks tasks that!

    is are. this or that tasks with! not not stop no? shell! can can and! This background with with commands the to
    task background I commands but? stop commands use. background stop background I I commands stop is! no that in 
    task
    Use
    It no the tasks but or or that use that but task tasks this a a a? But shell but. Use not tasks tasks
    that: background run to tasks this run not and! in. Stop run: background that! Stop
    task this to tasks background I I'll is is with background commands no are shell the tasks background not 
    background to: can background! tasks with with and commands with use not not stop with and the with stop tasks 
    tasks it that task it commands task tasks can! this background that commands task use that? stop to. in! task 
    shell or it! The that
    that commands stop
    use but but to tasks shell tasks? This stop? This are I. Stop run shell! not not can the not run shell but
    No is run background I that a background or but run
    but task no use background a: or commands is! background tasks use background: Stop and: that background stop 
    that to tasks in task it stop this stop tasks. this tasks not is is task with and in tasks this can the! But 
    run.

    ? and commands! commands background commands stop is I not or this and background with with commands with are 
    shell not
    Wait not with that commands
    Stop I! background? a. stop but the tasks but use task use no stop a are it's task can not tasks no that 
    background a tasks stop tasks stop! Stop to: are! No that
    in stop shell no to tasks task tasks this to commands or to. or that to tasks with background I background a
    not can. But task but but! background with. This this background use background shell the is run background task

config

***@***:/work/projects/ik_llama.cpp$ cat /etc/systemd/system/ik_llama-server.service 
[Unit]
Description=ik_llama.cpp server (Qwen3.6-27B Inference)
After=network.target

[Service]
Type=simple
User=***
Group=***
WorkingDirectory=/work
ExecStart=/work/ik_llama-server \
  --model /work/models/Qwen3.6-27B-GGUF/Qwen3.6-27B-MTP-IQ4_KS.gguf \
  --alias "Qwen3.6-27B" \
  -c 131072 \
  -ctk q8_0 -ctv q8_0 \
  --merge-qkv \
  -muge \
  -ngl 99 \
  -t 1 \
  -tb 1 \
  --host 127.0.0.1 \
  --port 8080 \
  --parallel 1 \
  --jinja \
  --no-mmap \
  --ctx-checkpoints 32 \
  -cram 32768 \
  -mtp --draft-max 4 --draft-p-min 0.0 \
  --no-mmproj-offload \
  --mmproj /work/models/Qwen3.6-27B-GGUF/mmproj-Qwen3.6-27B-Q8_0.gguf \
  --image-min-tokens 1024 \
  --image-max-tokens 4096
Restart=always
RestartSec=3
StandardOutput=journal
StandardError=journal
Environment=LD_LIBRARY_PATH=/usr/local/lib:/usr/lib

[Install]
WantedBy=multi-user.target
***@***:/work/projects/ik_llama.cpp$ git log --oneline 
949bb8f1 (HEAD -> main, origin/main, origin/HEAD) More MTP tweaks (#1792)

ik_llama logs

May 14 13:22:36 epsilon ik_llama-server[1941]: ======== Prompt cache: cache size: 61417, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 1.00, cache_ram_similarity: 0.50
May 14 13:22:36 epsilon ik_llama-server[1941]:  - looking for better prompt, base f_keep = 1.000, sim = 1.000, n_keep = 0, n_discarded_prompt = 0
May 14 13:22:36 epsilon ik_llama-server[1941]:  - cache state: 3 prompts, 9251.152 MiB (limits: 32768.000 MiB, 0 tokens, 147026 est)
May 14 13:22:36 epsilon ik_llama-server[1941]:    - prompt 0x77b204674ec0:   17373 tokens,       0 discarded, checkpoints: 32,  5554.537 MiB
May 14 13:22:36 epsilon ik_llama-server[1941]:    - prompt 0x632edd9bc950:    1536 tokens,       0 discarded, checkpoints:  0,   203.827 MiB
May 14 13:22:36 epsilon ik_llama-server[1941]:    - prompt 0x77b1d80e2ba0:   22600 tokens,       0 discarded, checkpoints: 17,  3492.788 MiB
May 14 13:22:36 epsilon ik_llama-server[1941]: prompt cache load took 7.33 ms
May 14 13:22:36 epsilon ik_llama-server[1941]: INFO [   launch_slot_with_task] slot is processing task | tid="131637731364864" timestamp=1778757756 id_slot=0 id_task=24832
May 14 13:22:36 epsilon ik_llama-server[1941]: ======== Cache: cache_size = 61417, n_past0 =  60602, n_past1 =  60602, n_past_prompt1 = 60602,  n_past2 =  60602, n_past_prompt2 =  60602
May 14 13:22:36 epsilon ik_llama-server[1941]: Common part does not match fully
May 14 13:22:36 epsilon ik_llama-server[1941]: cache :
May 14 13:22:36 epsilon ik_llama-server[1941]: <|im_start|>assistant
May 14 13:22:36 epsilon ik_llama-server[1941]: <think>
May 14 13:22:36 epsilon ik_llama-server[1941]: The user wants me to SSH into the remote host and run docker-compose up.
May 14 13:22:36 epsilon ik_llama-server[1941]: </think>
May 14 13:22:36 epsilon ik_llama-server[1941]: <tool_call>
May 14 13:22:36 epsilon ik_llama-server[1941]: <function=
May 14 13:22:36 epsilon ik_llama-server[1941]: prompt:
May 14 13:22:36 epsilon ik_llama-server[1941]: <|im_start|>assistant
May 14 13:22:36 epsilon ik_llama-server[1941]: <tool_call>
May 14 13:22:36 epsilon ik_llama-server[1941]: <function=run_shell_command>
May 14 13:22:36 epsilon ik_llama-server[1941]: <parameter=command>
May 14 13:22:36 epsilon ik_llama-server[1941]: ssh ubuntu@***
May 14 13:22:36 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | n_past = 60602, slot.prompt.tokens.size() = 61417, seq_id = 0, pos_min = 61416
May 14 13:22:36 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | restored context checkpoint took  13.97 ms (pos_min = 60598, pos_max = 60598, n_tokens = 60599, n_past = 60599, size = 150.090 MiB)
May 14 13:22:36 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | erased invalidated context checkpoint (pos_min = 60928, pos_max = 60928, size = 150.092 MiB)
May 14 13:22:36 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | erased invalidated context checkpoint (pos_min = 61085, pos_max = 61085, size = 150.094 MiB)
May 14 13:22:36 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | erased invalidated context checkpoint (pos_min = 61280, pos_max = 61280, size = 150.095 MiB)
May 14 13:22:37 epsilon ik_llama-server[1941]: slot apply_checkp: id  0 | task 24832 | erased invalidated context checkpoint (pos_min = 61416, pos_max = 61416, size = 150.096 MiB)
May 14 13:22:37 epsilon ik_llama-server[1941]: INFO [    batch_pending_prompt] kv cache rm [p0, end) | tid="131637731364864" timestamp=1778757757 id_slot=0 id_task=24832 p0=60599
May 14 13:22:38 epsilon ik_llama-server[1941]: slot create_check: id  0 | task 24832 | created context checkpoint 25 of 32 (pos_min = 61383, pos_max = 61383, n_tokens = 61384, size = 150.096 MiB, took 61.19 ms)
May 14 13:22:38 epsilon ik_llama-server[1941]: INFO [    batch_pending_prompt] kv cache rm [p0, end) | tid="131637731364864" timestamp=1778757758 id_slot=0 id_task=24832 p0=61384
May 14 13:22:38 epsilon ik_llama-server[1941]: reasoning-budget: activated, budget=2147483647 tokens
May 14 13:23:45 epsilon ik_llama-server[1941]: INFO [      log_server_request] request | tid="131636260839424" timestamp=1778757825 remote_addr="127.0.0.1" remote_port=51310 status=200 method="POST" path="/v1/chat/completions" params={}
May 14 13:23:45 epsilon ik_llama-server[1941]: srv          stop: cancel task, id_task = 24832
May 14 13:23:45 epsilon ik_llama-server[1941]: INFO [           release_slots] slot released | tid="131637731364864" timestamp=1778757825 id_slot=0 id_task=24832 n_ctx=131072 n_past=62311 n_system_tokens=0 n_cache_tokens=62311 truncated=false
May 14 13:23:45 epsilon ik_llama-server[1941]: INFO [              slots_idle] all slots are idle | tid="131637731364864" timestamp=1778757825

ubergarm

Owner May 14

•

edited May 14

@gkubon

Interesting thanks for sharing the LLM output, I've seen something like that before psure. If you want to try it again, make this one change:

# old setting
# -mtp --draft-max 4 --draft-p-min 0.0 \
-mtp --draft-max 4 --draft-p-min 0.50 \

While draft-p-min being 0 should be fine (the main model should correct the draft model), anecdotally I'd seen some wonky output like that before when using 0 before.

Also, given you're on a 3090, I'm assuming you know of LACT for linux undervolt / overclock ? It allows you to run smoother without thermal / power throttling by avoiding p-state 0 and limiting the max boost clock. i get better performance, keep the card cooler, and its less oscillating noise. holler if you're interested in more links on that.

Cheers!

PS:

That -tm got merged into main, full command here for using mmproj as well for visual feedback still fitting it all in 24GB VRAM with 128k context: https://github.com/ikawrakow/ik_llama.cpp/pull/1797#issuecomment-4442151972

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment