localslm/Qwen3.5-9B-MTP-Uncensored

Uncensored Qwen3.5-9B-MTP Q8_0 GGUF

Example Hybrid Setup: CPU (LLM) + GPU (Vision, ~4GB VRAM)

This setup launches a hybrid CPU/GPU vision-enabled server using llama.cpp designed for maximum performance on budget hardware:

• Fast CPU Text Generation (~20+ tokens/sec): Leverages Multi-Token Prediction (MTP) speculative decoding while disabling model "thinking/reasoning" steps.
• Instant Image Preprocessing (~1 second): Offloads the multimodal projector to even a low-end GPU (4GB VRAM). Even with a massive budget of 4,096 visual tokens, image processing takes under a second on GPU — whereas running vision encoding on CPU alone would take up to 5 minutes.
• Context & WebUI: Features a 16K context window and an OpenAI-compatible local API/WebUI server on port 8080.

llama-server.exe ^
    -m "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\model.gguf" ^
    --mmproj "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\mmproj.gguf" ^
    --spec-type draft-mtp --spec-draft-n-max 6 ^
    --image-min-tokens 4096 ^
    --image-max-tokens 4096 ^
    -t 6 --threads-batch 6 ^
    -b 4224 -ub 4224 ^
    --no-mmap --mlock ^
    -ngl 0 ^
    --mmproj-offload ^
    -fa on ^
    -c 16384 ^
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 ^
    --presence-penalty 1.5 --repeat-penalty 1.0 ^
    --chat-template-kwargs "{\"enable_thinking\": false}" ^
    --jinja ^
    --metrics ^
    --host 127.0.0.1 --port 8080

This command launches llama-server.exe (the server binary provided by llama.cpp) to host a local REST API / web interface for a vision-enabled Qwen3.5-9B on Windows:

1. Model & Multimodal Files

• -m "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\model.gguf"
  Path to the main GGUF language model file (Qwen 3.5 9B).
• --mmproj "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\mmproj.gguf"
  Path to the multimodal projector file (vision), allowing the model to process image inputs alongside text.

2. Speculative Decoding & Multimodal Settings

• --spec-type draft-mtp
  Enables Multi-Token Prediction (MTP) speculative decoding. By predicting multiple tokens per step natively, it dramatically boosts CPU token generation speed (up to 20+ tokens/sec).
• --spec-draft-n-max 6
  Sets the maximum number of draft tokens generated per speculative decoding step to 6.
• --image-min-tokens 4096 & --image-max-tokens 4096
  Forces image processing to use a fixed allocation of 4,096 visual tokens per image input, ensuring maximum visual detail and OCR precision.

3. CPU, Hardware, & Memory Management

• -t 6
  Allocates 6 CPU threads for token generation.
• --threads-batch 6
  Allocates 6 CPU threads for prompt processing/batch processing.
• -b 4224
  Sets the maximum prompt/logical batch size to 4,224 tokens.
• -ub 4224
  Sets the micro-batch size (ubatch) to 4,224 tokens, ensuring batch processing is not split into smaller sub-batches.
• --no-mmap
  Disables memory mapping (mmap). The entire model is loaded directly into RAM at startup rather than read lazily from disk.
• --mlock
  Locks model memory in physical RAM to prevent the operating system from swapping it out to disk/pagefile.
• -ngl 0
  Set GPU offload layers to 0. This forces the main LLM to run entirely on the CPU.
• --mmproj-offload
  Crucial optimization: Offloads the heavy vision encoder to the GPU. Even on a cheap GPU with 4GB VRAM, processing a huge 4,096 visual token budget completes in < 1 second, whereas CPU-only vision processing would bottleneck the system for ~5 minutes.
• -fa on
  Enables FlashAttention (or optimized attention kernels) to reduce RAM usage and improve processing speed.

4. Context & Generation (Sampling) Parameters

• -c 16384
  Sets the maximum context window size to 16,384 tokens.
• --temp 0.7
  Sets generation temperature to 0.7 (balances creativity and coherence).
• --top-p 0.8
  Applies nucleus sampling, considering tokens within the top 80% cumulative probability.
• --top-k 20
  Restricts token selection to the top 20 candidate tokens at each step.
• --min-p 0.0
  Sets Min-P sampling threshold to 0.0 (effectively disabling Min-P filtering).
• --presence-penalty 1.5
  Applies a strong penalty for introducing already mentioned tokens, encouraging the model to introduce new topics.
• --repeat-penalty 1.0
  Sets repetition penalty to 1.0 (neutral / standard repetition penalty disabled).

5. Template & Server Configuration

• --chat-template-kwargs "{\"enable_thinking\": false}"
  Passes custom key-value parameters to the model's chat template. Disables reasoning/thinking blocks to minimize generation overhead and maximize speed.
• --jinja
  Enables Jinja2 templating support to process modern complex chat templates included with models.
• --metrics
  Enables an endpoint for system metrics monitoring (e.g., tokens per second, memory usage).
• --host 127.0.0.1
  Binds the server locally to localhost (127.0.0.1), preventing external network access.
• --port 8080
  Runs the server API on port 8080 (accessible via http://127.0.0.1:8080).