Instructions to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Melvin56/GLM-Z1-9B-0414-abliterated-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Melvin56/GLM-Z1-9B-0414-abliterated-GGUF", dtype="auto")

llama-cpp-python

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Melvin56/GLM-Z1-9B-0414-abliterated-GGUF",
	filename="glm-z1-9b-0414-abliterated-BF16.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

SGLang

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Ollama:
```
ollama run hf.co/Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
```

Unsloth Studio

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Melvin56/GLM-Z1-9B-0414-abliterated-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Melvin56/GLM-Z1-9B-0414-abliterated-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Melvin56/GLM-Z1-9B-0414-abliterated-GGUF to start chatting

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new
Docker Model Runner
How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Docker Model Runner:
```
docker model run hf.co/Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M
```

Lemonade

How to use Melvin56/GLM-Z1-9B-0414-abliterated-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Melvin56/GLM-Z1-9B-0414-abliterated-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.GLM-Z1-9B-0414-abliterated-GGUF-Q4_K_M

List all available models

lemonade list

Melvin56 commited on Apr 22, 2025

Commit

a47c1b0

1 Parent(s): 8182055

Update README.md

Browse files

Files changed (1) hide show

README.md +22 -131

README.md CHANGED Viewed

@@ -11,144 +11,35 @@ tags:
 - abliterated
 - uncensored
 ---
-# huihui-ai/GLM-Z1-9B-0414-abliterated
-This is an uncensored version of [THUDM/GLM-Z1-9B-0414](https://huggingface.co/THUDM/GLM-Z1-9B-0414) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it).
-This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens.
-## Use with transformers
 ```
-from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TextStreamer
-import torch
-import os
-import signal
-cpu_count = os.cpu_count()
-print(f"Number of CPU cores in the system: {cpu_count}")
-half_cpu_count = cpu_count // 2
-os.environ["MKL_NUM_THREADS"] = str(half_cpu_count)
-os.environ["OMP_NUM_THREADS"] = str(half_cpu_count)
-torch.set_num_threads(half_cpu_count)
-print(f"PyTorch threads: {torch.get_num_threads()}")
-print(f"MKL threads: {os.getenv('MKL_NUM_THREADS')}")
-print(f"OMP threads: {os.getenv('OMP_NUM_THREADS')}")
-# Load the model and tokenizer
-NEW_MODEL_ID = "huihui-ai/GLM-Z1-9B-0414-abliterated"
-print(f"Load Model {NEW_MODEL_ID} ... ")
-quant_config_4 = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-    bnb_4bit_use_double_quant=True,
-    llm_int8_enable_fp32_cpu_offload=True,
-)
-quant_config_2 = BitsAndBytesConfig(
-    load_in_2bit=True,
-    bnb_2bit_compute_dtype=torch.bfloat16,
-    llm_int8_enable_fp32_cpu_offload=True
-)
-model = AutoModelForCausalLM.from_pretrained(
-    NEW_MODEL_ID,
-    device_map="auto",
-    trust_remote_code=True,
-    quantization_config=quant_config_2,
-    torch_dtype=torch.bfloat16
-)
-tokenizer = AutoTokenizer.from_pretrained(NEW_MODEL_ID, trust_remote_code=True)
-if tokenizer.pad_token is None:
-    tokenizer.pad_token = tokenizer.eos_token
-tokenizer.pad_token_id = tokenizer.eos_token_id
-initial_messages = [{"role": "system", "content": "You are a helpful assistant."}]
-messages = initial_messages.copy()
-class CustomTextStreamer(TextStreamer):
-    def __init__(self, tokenizer, skip_prompt=True, skip_special_tokens=True):
-        super().__init__(tokenizer, skip_prompt=skip_prompt, skip_special_tokens=skip_special_tokens)
-        self.generated_text = ""
-        self.stop_flag = False
-    def on_finalized_text(self, text: str, stream_end: bool = False):
-        self.generated_text += text
-        print(text, end="", flush=True)
-        if self.stop_flag:
-            raise StopIteration
-    def stop_generation(self):
-        self.stop_flag = True
-def generate_stream(model, tokenizer, messages, max_new_tokens):
-    input_ids = tokenizer.apply_chat_template(
-        messages,
-        tokenize=True,
-        add_generation_prompt=True,
-        return_tensors="pt"
-    )
-    attention_mask = torch.ones_like(input_ids, dtype=torch.long)
-    tokens = input_ids.to(model.device)
-    attention_mask = attention_mask.to(model.device)
-    streamer = CustomTextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
-    def signal_handler(sig, frame):
-        streamer.stop_generation()
-        print("\n[Generation stopped by user with Ctrl+C]")
-    signal.signal(signal.SIGINT, signal_handler)
-    print("Response: ", end="", flush=True)
-    try:
-        generated_ids = model.generate(
-            tokens,
-            attention_mask=attention_mask,
-            use_cache=False,
-            max_new_tokens=max_new_tokens,
-            do_sample=True,
-            pad_token_id=tokenizer.pad_token_id,
-            streamer=streamer
-        )
-        del generated_ids
-    except StopIteration:
-        print("\n[Stopped by user]")
-    del input_ids, attention_mask
-    torch.cuda.empty_cache()
-    signal.signal(signal.SIGINT, signal.SIG_DFL)
-    return streamer.generated_text, streamer.stop_flag
-while True:
-    user_input = input("User: ").strip()
-    if user_input.lower() == "/exit":
-        print("Exiting chat.")
-        break
-    if user_input.lower() == "/clear":
-        messages = initial_messages.copy()
-        print("Chat history cleared. Starting a new conversation.")
-        continue
-    if not user_input:
-        print("Input cannot be empty. Please enter something.")
-        continue
-    messages.append({"role": "user", "content": user_input})
-    response, stop_flag = generate_stream(model, tokenizer, messages, 8192)
-    if stop_flag:
-        continue
-    messages.append({"role": "assistant", "content": response})
 ```
-### Donation
-If you like it, please click 'like' and follow us for more updates.
-You can follow [x.com/support_huihui](https://x.com/support_huihui) to get the latest model information from huihui.ai.
-##### Your donation helps us continue our further development and improvement, a cup of coffee can do it.
-- bitcoin（BTC):
 ```
-  bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
 ```

 - abliterated
 - uncensored
 ---
+# Melvin56/GLM-Z1-9B-0414-abliterated-GGUF
+Original Model : [huihui-ai/GLM-Z1-9B-0414-abliterated](https://huggingface.co/huihui-ai/GLM-Z1-9B-0414-abliterated)
+Llama.cpp build: 1d735c0b (5165)
+I used imatrix to create all these quants using this [Dataset](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8).
+With Llama.cpp(Ollama and LM Studio were not tested), you'll need to add these [specific commands](https://github.com/ggml-org/llama.cpp/issues/12946) :
 ```
+--override-kv glm4.rope.dimension_count=int:64 \
+--override-kv tokenizer.ggml.eos_token_id=int:151336 \
+--chat-template chatglm4
 ```
+|               | CPU (AVX2) | CPU (ARM NEON) | Metal | cuBLAS | rocBLAS | SYCL | CLBlast | Vulkan | Kompute |
+| :------------ | :---------: | :------------: | :---: | :----: | :-----: | :---: | :------: | :----: | :------: |
+| K-quants      |      ✅     |       ✅      |   ✅  |   ✅   |    ✅   |  ✅  |   ✅ 🐢5  |  ✅ 🐢5 |    ❌    |
+| I-quants      |    ✅ 🐢4   |     ✅ 🐢4    |  ✅ 🐢4 |   ✅   |    ✅   | Partial¹ |    ❌    |   ❌  |    ❌    |
 ```
+✅: feature works
+🚫: feature does not work
+❓: unknown, please contribute if you can test it youself
+🐢: feature is slow
+¹: IQ3_S and IQ1_S, see #5886
+²: Only with -ngl 0
+³: Inference is 50% slower
+⁴: Slower than K-quants of comparable size
+⁵: Slower than cuBLAS/rocBLAS on similar cards
+⁶: Only q8_0 and iq4_nl
 ```