bartowski

/

Qwen3.8-27B-GGUF

like 103

I am experiencing intermittent premature EOS (End of Sequence) truncation when running Qwen3.8-27B in an agentic/tool-calling workflow.

Despite using the community-vetted fixed chat template (froggeric/Qwen-Fixed-Chat-Templates), enabling --reasoning on, and dynamically aligning the --reasoning-budget with the reasoning_effort, the model occasionally stops generating mid-thought or mid-tool-call.

When this happens, the llama-server logs show a clean exit with truncated = 0, indicating the server believes it finished normally, but the output is clearly cut off from the client's perspective.

Environment:

• Hardware: Kaggle Notebook (2x NVIDIA T4 GPUs, 16GB VRAM each)
• OS: Linux (Kaggle Docker environment)
• llama.cpp build: b10435 / b10451 (CUDA 12.x pre-compiled)
• Models Tested:
  • bartowski/Qwen3.8-27B-GGUF → Qwen3.8-27B-Q4_K_M.gguf (Vision: mmproj-Qwen3.8-27B-bf16.gguf)
  • unsloth/Qwen3.8-27B-GGUF → Qwen3.8-27B-UD-Q4_K_XL.gguf (Vision: mmproj-BF16.gguf)
• Chat Template: froggeric/Qwen-Fixed-Chat-Templates (chat_template.jinja)

Exact llama-server Command Used:
(Extracted dynamically from my Python orchestration script)

llama-server \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-bf16.gguf \
  --chat-template-file /tmp/qwen3.8_fixed.jinja \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 131072 \
  --gpu-layers 999 \
  --jinja \
  -fa on \
  --no-mmap \
  --threads 4 \
  --temp 1.0 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.0 \
  --repeat-penalty 1.0 \
  --repeat-last-n 256 \
  --presence-penalty 0.0 \
  --threads-batch 4 \
  -b 2048 \
  -ub 1024 \
  -np 1 \
  --split-mode layer \
  -ts 1,1 \
  --reasoning on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --defrag-thold 0.1 \
  --reasoning-preserve \
  --reasoning-budget 2048 \
  --n-predict 98304 \
  --reasoning-budget-message "Reasoning budget exhausted — answering now." \
  --chat-template-kwargs '{"reasoning_effort": "low"}'

Key Configuration Notes:

1. Split Mode: Running with --split-mode layer -ts 1,1 across 2x T4 GPUs.
2. Reasoning: Explicitly set to on with a strict budget of 2048 tokens to enforce concise thinking, paired with reasoning_effort: "low" in the template kwargs.
3. Template: Using the froggeric fixed template to avoid the known official Qwen 3.8 "empty think poisoning" and tool-parsing bugs.
4. Both GGUF sources tested: The issue occurs with both bartowski and unsloth quantizations, suggesting it is not quantization-specific.