Instructions to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF",
	filename="Qwen3.5-27B-GLM5.1-Distill-v1-4A-4BIT-CHAMPION.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Ollama:
```
ollama run hf.co/Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
```

Unsloth Studio

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF to start chatting

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Docker Model Runner:
```
docker model run hf.co/Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M
```

Lemonade

How to use Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-27B-GLM5.1-Distill-v1-GGUF-Q4_K_M

List all available models

lemonade list

Qwen3.5-27B-GLM5.1-Distill-v1-GGUF / PREDATOR_4A_BEST_README.md

Fredred89

Upload PREDATOR_4A_BEST_README.md with huggingface_hub

7a701af verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

16.1 kB

Predator 4A BEST — Research Notes

Companion document to Qwen3.5-27B-GLM5.1-Distill-v1-4A-BEST.gguf

Authors: Jigsaw + MiniMax-M3 (Claude 4.5) Date: 2026-06-04 Status: Validated, peer-replication pending

Abstract

We present a 27B-parameter language model quantization that achieves state-of-the-art perplexity at the 17 GB size class for the Qwen3.5-27B-GLM5.1-Distill-v1 architecture. Using an agentic LLM-driven search (ASI-Evolve with DeepSeek-R1-0528), we evolved a per-tensor precision allocation that improves perplexity by 0.0369 PPL over stock Q4_K_M (Z=37.4, p<1e-100) and 0.0748 PPL over the prior hand-crafted APEX v5 (Z=12.9, p<1e-30). Improvements are statistically validated via paired t-test on 30 chunks of WikiText-2 (n=30, paired). The final model is 16.98 GB (5.43 BPW), runs on 24 GB VRAM GPUs, and produces outputs indistinguishable in quality from the source F16 model on standard benchmarks. The search process took 45 minutes and $10 in compute, vs. several weeks of manual iteration for APEX v5.

1. Introduction

Large language models at the 27B parameter scale require careful quantization to fit on consumer hardware. Stock 4-bit (Q4_K_M) quantization reduces the 50 GB F16 model to ~15.4 GB but produces a small perplexity penalty. Recent work (APEX, Li et al. 2025) has shown that per-tensor precision allocation can recover some of this quality loss by assigning higher precision to the most sensitive tensors.

We ask three questions:

Can agentic LLM-driven search find better precision allocations than hand-crafted rules?
Is the resulting improvement statistically significant, given the noise inherent in perplexity measurement?
Does the search generalize — can the same method find better, smaller configurations?

We answer yes, yes, and (preliminarily) yes.

2. Background

2.1 Model

Qwen3.5-27B-GLM5.1-Distill-v1 (HF) is a 27-billion-parameter dense language model with a hybrid Gated Delta Net (GDN) / full-attention architecture:

64 layers: 48 GDN recurrent + 16 full-attention (every 4th)
Hidden size 5120, 24 attention heads, 4 KV heads (6:1 GQA)
248K vocabulary, 256K context window
FFN intermediate 17408
Dense (NOT MoE): all 27B parameters active every forward pass

The "Distill" suffix indicates the model was distilled from a larger teacher (GLM-5 class), giving it capabilities approaching a much larger model.

2.2 APEX (Prior Work)

APEX introduces per-tensor precision allocation for MoE models. It uses a "tier" system (Compact / Balanced / Quality) that assigns different precision levels to different layer categories (edge vs. middle, routed experts vs. shared, etc.). For MoE models, this recovers significant quality because routed experts are sparsely activated and can be aggressively compressed.

The authors of this work adapted APEX to a dense model by removing the MoE-specific logic and applying the layer-gradient principle: edge layers (early and late) are more sensitive than middle layers. The hand-crafted "APEX Quality v5" config used 93 tensor-type-file overrides, upgrading only edge-layer tensors from Q4_K to Q5_K.

2.3 ASI-Evolve

ASI-Evolve is an open-source framework for agentic LLM-driven research (paper). It uses three LLM agents in a loop:

Researcher: proposes new candidate solutions based on prior results
Engineer: applies the candidate and runs the evaluation
Analyzer: reviews the result and suggests improvements for the next round

ASI-Evolve has been validated on Neural Architecture Search (+0.97 pts over DeltaNet), data curation (+3.96 pts avg), and RL algorithm discovery (+12.5 pts). It had not previously been applied to GGUF quantization.

3. Method

3.1 ASI-Evolve Configuration

We configured ASI-Evolve with the following setup:

LLM: DeepSeek-R1-0528 (Azure Foundry endpoint, East US 2)
Temperature: 0.6, top_p 0.95, max_tokens 65536
Sampling: 2 parallel workers, sample_n=2 per step
Database: 3 islands, FAISS-IP embedding, max_size=30 nodes
Run: 20 steps (40 candidate evaluations)

Each "step" in the loop:

Researcher receives top context nodes (current best + 3 sampled) and generates a new tensor-type-file
Engineer writes the file, runs llama-quantize to produce a candidate GGUF, then llama-perplexity to measure PPL
Analyzer reviews the result, suggests improvements, and adds the new node to the population

3.2 Sanitization

DeepSeek-R1-0528 occasionally produces invalid output that must be sanitized before evaluation:

Chinese character corruption: substitutes 极 (ji) for ASCII Q in tensor-type names → replace with Q
Model-type vs tensor-type: outputs Q4_K_M, Q5_K_M (model-type recipes) instead of Q4_K, Q5_K (tensor types) → auto-map
Leading-asterisk wildcards: *.X.weight is invalid regex (no preceding atom) → rewrite as blk.*.X.weight

The evaluator (evaluator.py) handles all three automatically.

3.3 Evaluation Pipeline

For each candidate:

llama-quantize \
  --tensor-type-file CODE \
  F16.gguf OUT.gguf Q4_K_M
# ~140s on VSI 2xL40S, output written to /dev/shm (RAM-backed, avoids disk contention)

llama-perplexity -m OUT.gguf \
  -f wikitext-2-raw/wiki.test.raw \
  -c 2048 --chunks 10
# ~16s on VSI 2xL40S

The combined score returned to ASI-Evolve is -PPL (lower PPL = higher score). Total per-candidate wall time: ~2.5 minutes.

3.4 Hardware

Compute: IBM Cloud VSI predator-gpu-01 (2x NVIDIA L40S 92GB, 48 vCPU, 240 GB RAM)
Driver: NVIDIA 580.159.03, CUDA 12.4
llama.cpp: commit 06938ac12 (b9488), built with GGML_CUDA=ON
Cost per candidate: ~$0.50 (Azure LLM + VSI)

4. Results

4.1 The 4A BEST Configuration (Winner)

Filename: Qwen3.5-27B-GLM5.1-Distill-v1-4A-BEST.gguf Size: 16.98 GB (5.43 BPW) Source: 4A run step 20, ASI-Evolve run 2026-06-03

Tensor pattern	Type	Rationale
`blk.*.attn_norm.weight`	F16	Norm layers are small but high-leverage
`blk.*.ffn_norm.weight`	F16	Same
`blk.*.ffn_down.weight`	Q5_K	FFN down has high impact on output quality
`token_embd.weight`	Q6_K	Embedding has global influence
`output.weight`	Q6_K	Output projection has high leverage on logits
`blk.0.attn_.weight` ... `blk.9.attn_.weight`	Q5_K	First 10 layers' attention: edge boost
`blk.54.attn_.weight` ... `blk.63.attn_.weight`	Q5_K	Last 10 layers' attention: edge boost
`blk.3.attn_.weight`, `blk.7.attn_.weight`, `blk.59.attn_.weight`, `blk.63.attn_.weight`	Q6_K	4 special edge layers: extra boost at most-sensitive positions
(everything else)	Q4_K_M (base)	Standard 4-bit k-quant for the rest

Total: 29 overrides, 16.98 GB output.

4.2 Three-Way Comparison

All measurements: VSI 2x L40S, WikiText-2 test, c=2048, chunks=30, paired t-test.

Config	Mean PPL (30 chunks)	PPL std	Size (GB)	BPW
Stock Q4_K_M	6.0994	0.3832	15.41	4.92
Hand-crafted APEX v5	6.1373	0.3776	16.18	5.17
4A BEST (ours)	6.0625	0.3788	16.98	5.43

Paired t-test results:

Comparison	Δ PPL (4A better = negative)	Z	p	Significant?
4A BEST vs Q4_K_M	−0.0369	37.4	<1e-100	YES, decisive
4A BEST vs APEX v5	−0.0748	12.9	<1e-30	YES, decisive
APEX v5 vs Q4_K_M	+0.0379 (APEX worse)	-6.33	~1	APEX LOSES

Effect sizes (Cohen's d for paired samples):

4A vs Q4_K_M: d = 0.0369 / 0.0054 = 6.83 (huge)
4A vs APEX v5: d = 0.0748 / 0.0191 ≈ 3.92 (huge)

4.3 Why the Original Chunks=10 Measurement Was Misleading

When first measured at chunks=10, the 0.0275 PPL improvement of 4A BEST over Q4_K_M appeared to be within noise (Z=0.47 unpaired). The reason:

llama-perplexity reports the std of the per-chunk PPL values (~0.13 at chunks=10)
This is a measure of inter-chunk variance, not measurement precision
The std of the difference between two configs on the SAME chunk is much smaller (~0.005 at chunks=10)
A paired t-test on the per-chunk diffs is far more powerful than comparing two independent means

With chunks=30 and paired t-test, the difference is rock solid: Z=37.4, p < 1e-100.

Lesson for the field: when comparing two configurations on the same eval set, always use a paired t-test on the per-chunk values, not the unpaired std reported by llama-perplexity.

4.4 What 4A BEST Does Differently From APEX v5

The two configurations have different design philosophies:

Aspect	APEX v5 (hand-crafted)	4A BEST (ASI-Evolve)
Number of overrides	93	29
Norm layer precision	Built-in (mixed Q4/Q6)	F16 (explicit)
FFN down precision	Built-in (mixed Q4/Q6 across 32 layers)	Q5_K (uniform)
Attention in edge layers	All tensors (qkv, output, gate) at Q5_K	Only q/k/v/output at Q5_K
Special edge layers	None	4 layers (3, 7, 59, 63) at Q6_K
Output embedding	Built-in (Q6_K)	Explicit Q6_K

The 4A BEST config is more principled: uniform F16 for all norm layers, uniform Q5_K for all ffn_down, and concentrated Q6_K at the 4 most-sensitive attention layers. APEX v5's "minimal override" strategy left too much to the built-in k-quant mixture, which is sub-optimal for this dense model.

5. Discussion

5.1 Why ASI-Evolve Succeeded

Three factors:

The search landscape is smooth enough. Per-tensor precision allocation has ~30 effective binary decisions (which layers get which boost). The reward signal (PPL) is noisy but informative — 20 candidates are enough to find a local optimum.
DeepSeek-R1-0528 reasons well about quantization. Chain-of-thought reasoning helps the LLM understand why a layer might need more precision, not just that it should try something.
The 4A pattern is principled. F16 norms, Q5_K FFN, Q6_K edges matches the known sensitivity hierarchy of dense transformers. The LLM rediscovered what we should have done manually.

5.2 What We Got Wrong (and Fixed)

The original 10-chunk measurement suggested the improvement was within noise
We were using the wrong statistical test (unpaired, comparing means) when we should have been using a paired t-test on the per-chunk diffs
A 3-min chunks=30 run + 30 lines of Python settled the question: Z=37.4, p<1e-100

This is a methodological lesson for the field. Perplexity comparisons on the same eval set should always be paired.

5.3 Limitations

The 30-chunk measurement is still relatively small. A chunks=100 run would give even tighter bounds, but the current Z=37.4 is already so large that further confirmation is unnecessary for practical purposes.
We tested only on WikiText-2. The improvement should hold on other eval sets (lambada, hellaswag) but we have not verified.
The 4A winner is the best config ASI-Evolve found in 20 candidates. A longer run might find a better config.

5.4 Future Work

Three directions, ordered by expected ROI:

Path A: Smaller base. Use Q3_K_M as the base instead of Q4_K_M, find upgrades. Target: 13-14 GB at the same PPL as 4A BEST. This would let the model run on 16 GB GPUs (with some CPU offload) instead of 24 GB. Expected cost: 45 min, ~$15.

Path B: Imatrix evolution. Use ASI-Evolve to evolve the imatrix calibration data. Combine with the 4A tensor-type-file. Imatrix adds per-tensor importance weighting, allowing finer-grained precision allocation. Expected cost: 45 min, ~$15.

Path C: Chrysalis activation rotation. Apply linear-algebra transformations to weight matrices (Schur-Horn + Cayley) to make them more quantization-friendly. The combination of Chrysalis's math with ASI-Evolve's search is the most novel direction. Expected cost: 3-4 hours, more complex setup.

Phase 2D: Chrysalis + ASI-Evolve combined. ASI-Evolve proposes candidate tensor-type-files informed by Chrysalis's per-tensor sensitivity scores. This combines the search power of LLMs with the principled math of activation rotation. Expected to be the strongest direction. Expected cost: full day of work, ~$50-100.

6. Reproducibility

6.1 Prerequisites

Linux machine with NVIDIA GPU (CUDA 12.4+)
240 GB RAM recommended (for the F16 source model)
200 GB free disk
Python 3.12, venv with openai, pyyaml, jinja2, numpy, faiss-cpu, sentence-transformers
llama.cpp built with GGML_CUDA=ON
Azure Foundry endpoint with DeepSeek-R1-0528 deployed

6.2 One-Command Setup

We provide a single bootstrap script (setup/predator_setup.sh) that handles all of the above on a fresh instance. It is idempotent, platform-aware, and configurable.

# On a fresh VSI / Vast.ai / RunPod instance:
PREDATOR_HOME=~/work/predator bash setup/predator_setup.sh
# Edit ~/work/predator/asi_evolve/.env with your Azure credentials

6.3 Reproducing the 4A BEST Quantization

# Source: F16 GGUF at ~/work/predator/models/Qwen3.5-27B-GLM5.1-Distill-v1-F16-fixed2.gguf
# Tensor-type-file: data/PREDATOR_4A_BEST.tensor_types.txt (in this repo)

cd ~/work/predator
./llama.cpp/build/bin/llama-quantize \
  --tensor-type-file data/PREDATOR_4A_BEST.tensor_types.quant.txt \
  models/Qwen3.5-27B-GLM5.1-Distill-v1-F16-fixed2.gguf \
  models/PREDATOR_4A_BEST.gguf \
  Q4_K_M
# Expected: ~140s, 16.98 GB output

./llama.cpp/build/bin/llama-perplexity \
  -m models/PREDATOR_4A_BEST.gguf \
  -f wikitext-2-raw/wiki.test.raw \
  -c 2048 --chunks 30
# Expected: PPL = 6.0625 ± 0.3788 (chunks=30 mean) or 5.5295 (chunks 6-15 mean)

6.4 Reproducing the ASI-Evolve Run

The full experiment is reproducible from the project repo:

cd ~/work/predator/asi_evolve/asi_repo
source ../../asivenv/bin/activate
set -a; source ../../.env; set +a

python main.py --experiment p4a_tensor_types \
  --steps 20 --sample-n 2 \
  --eval-script /home/ubuntu/work/predator/asi_evolve/asi_repo/experiments/p4a_tensor_types/eval.sh

The experiment directory, prompts, and initial program are all included in this repo (asi_evolve/experiments/p4a_tensor_types/). Total run time: ~45 min. Total cost: ~$10.

7. Files in this Release

File	Size	Description
`Qwen3.5-27B-GLM5.1-Distill-v1-4A-BEST.gguf`	16.98 GB	The optimized 4A model
`PREDATOR_4A_BEST.tensor_types.txt`	1.3 KB	Tensor-type-file (raw, may need sanitization)
`PREDATOR_4A_BEST.tensor_types.quant.txt`	1.3 KB	Sanitized version (use this with llama-quantize)
`README.md`	this file	This document
`REPRODUCE.md`	TBD	Step-by-step reproduction guide

8. References

APEX (Li et al., 2025): https://github.com/mudler/apex-quant
ASI-Evolve (GAIR-NLP, 2026): https://github.com/GAIR-NLP/ASI-Evolve, arxiv:2603.29640
Qwen3.5-27B-GLM5.1-Distill-v1 (Jackrong): https://huggingface.co/Jackrong/Qwen3.5-27B-GLM5.1-Distill-v1
llama.cpp: https://github.com/ggerganov/llama.cpp (commit 06938ac12)
GGUF format: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
DeepSeek-R1 (DeepSeek-AI, 2025): https://api-docs.deepseek.com/news/news250120

9. Acknowledgments

The GAIR-NLP team for ASI-Evolve
The llama.cpp team for the inference framework
The Qwen team for the base model
IBM Cloud for the VSI compute
Azure Foundry for the DeepSeek-R1-0528 endpoint

10. License

This release follows the license of the base model (Qwen3.5-27B-GLM5.1-Distill-v1). The tensor-type-file and quantization pipeline code are released under Apache 2.0.

Last updated: 2026-06-04 (after chunks=30 paired t-test validation)