Instructions to use Terminator278/Qwen2.5-Coder-3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Terminator278/Qwen2.5-Coder-3B-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Terminator278/Qwen2.5-Coder-3B-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Terminator278/Qwen2.5-Coder-3B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Terminator278/Qwen2.5-Coder-3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

SGLang

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Terminator278/Qwen2.5-Coder-3B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Terminator278/Qwen2.5-Coder-3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Terminator278/Qwen2.5-Coder-3B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Terminator278/Qwen2.5-Coder-3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Ollama:
```
ollama run hf.co/Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
```

Unsloth Studio

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Terminator278/Qwen2.5-Coder-3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Terminator278/Qwen2.5-Coder-3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Terminator278/Qwen2.5-Coder-3B-GGUF to start chatting

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Docker Model Runner:
```
docker model run hf.co/Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M
```

Lemonade

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen2.5-Coder-3B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Terminator278/Qwen2.5-Coder-3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Terminator278/Qwen2.5-Coder-3B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Terminator278

prithivMLmods commited on Jun 14

Commit

8a3f508

0 Parent(s):

Duplicate from prithivMLmods/Qwen2.5-Coder-3B-GGUF

Browse files

Co-authored-by: Prithiv Sakthi <prithivMLmods@users.noreply.huggingface.co>

Files changed (6) hide show

.gitattributes +39 -0
Qwen2.5-Coder-3B.F16.gguf +3 -0
Qwen2.5-Coder-3B.Q4_K_M.gguf +3 -0
Qwen2.5-Coder-3B.Q5_K_M.gguf +3 -0
Qwen2.5-Coder-3B.Q8_0.gguf +3 -0
README.md +121 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,39 @@

+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Qwen2.5-Coder-3B.F16.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen2.5-Coder-3B.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen2.5-Coder-3B.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen2.5-Coder-3B.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

Qwen2.5-Coder-3B.F16.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ecd2bd21870d32716e70283da37a3cdc3ecebbb3a5b402313d06271578f9a9c1
+size 6178316608

Qwen2.5-Coder-3B.Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:103a3893057a6ad794c00cd38ab6ce8c3b9b88e9403ee8d8ec3adf561cdbb30a
+size 1929902400

Qwen2.5-Coder-3B.Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:91458254ecabf3c971ccecf430239b64e6f9f19e125442a7b7de2baab97a94d3
+size 2224814400

Qwen2.5-Coder-3B.Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d346e4c6291bd9d9674aec198c7646322d0a26ff262305858c9ed269613f3666
+size 3285475648

README.md ADDED Viewed

	@@ -0,0 +1,121 @@

+---
+license: creativeml-openrail-m
+language:
+- en
+base_model:
+- Qwen/Qwen2.5-Coder-3B
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- Qwen
+- '2.5'
+- Coder
+- F16
+- 16-bit
+- Q4
+- Q5
+- Q8
+- Llama-cpp
+---
+## Qwen2.5-Coder-3B-GGUF
+| File Name                         | Size    | Description                                                       |
+|-----------------------------------|---------|-------------------------------------------------------------------|
+| `.gitattributes`                  | 1.77kB  | Configuration file for Git and LFS handling.                      |
+| `Qwen2.5-Coder-3B.F16.gguf`       | 6.18GB  | Full-precision (16-bit) model for coding tasks.                   |
+| `Qwen2.5-Coder-3B.Q4_K_M.gguf`    | 1.93GB  | Quantized 4-bit model (medium variant) for reduced resource usage.|
+| `Qwen2.5-Coder-3B.Q5_K_M.gguf`    | 2.22GB  | Quantized 5-bit model (medium variant) balancing size and accuracy.|
+| `Qwen2.5-Coder-3B.Q8_0.gguf`      | 3.29GB  | Quantized 8-bit model for improved accuracy in coding tasks.      |
+| `README.md`                       | 42B     | Initial README with basic information.                            |
+# Run with Ollama 🦙
+## Overview
+Ollama is a powerful tool that allows you to run machine learning models effortlessly. This guide will help you download, install, and run your own GGUF models in just a few minutes.
+## Table of Contents
+- [Download and Install Ollama](#download-and-install-ollama)
+- [Steps to Run GGUF Models](#steps-to-run-gguf-models)
+  - [1. Create the Model File](#1-create-the-model-file)
+  - [2. Add the Template Command](#2-add-the-template-command)
+  - [3. Create and Patch the Model](#3-create-and-patch-the-model)
+- [Running the Model](#running-the-model)
+- [Sample Usage](#sample-usage)
+## Download and Install Ollama🦙
+To get started, download Ollama from [https://ollama.com/download](https://ollama.com/download) and install it on your Windows or Mac system.
+## Steps to Run GGUF Models
+### 1. Create the Model File
+First, create a model file and name it appropriately. For example, you can name your model file `metallama`.
+### 2. Add the Template Command
+In your model file, include a `FROM` line that specifies the base model file you want to use. For instance:
+```bash
+FROM Llama-3.2-1B.F16.gguf
+```
+Ensure that the model file is in the same directory as your script.
+### 3. Create and Patch the Model
+Open your terminal and run the following command to create and patch your model:
+```bash
+ollama create metallama -f ./metallama
+```
+Once the process is successful, you will see a confirmation message.
+To verify that the model was created successfully, you can list all models with:
+```bash
+ollama list
+```
+Make sure that `metallama` appears in the list of models.
+---
+## Running the Model
+To run your newly created model, use the following command in your terminal:
+```bash
+ollama run metallama
+```
+### Sample Usage
+In the command prompt, you can execute:
+```bash
+D:\>ollama run metallama
+```
+You can interact with the model like this:
+```plaintext
+>>> write a mini passage about space x
+Space X, the private aerospace company founded by Elon Musk, is revolutionizing the field of space exploration.
+With its ambitious goals to make humanity a multi-planetary species and establish a sustainable human presence in
+the cosmos, Space X has become a leading player in the industry. The company's spacecraft, like the Falcon 9, have
+demonstrated remarkable capabilities, allowing for the transport of crews and cargo into space with unprecedented
+efficiency. As technology continues to advance, the possibility of establishing permanent colonies on Mars becomes
+increasingly feasible, thanks in part to the success of reusable rockets that can launch multiple times without
+sustaining significant damage. The journey towards becoming a multi-planetary species is underway, and Space X
+plays a pivotal role in pushing the boundaries of human exploration and settlement.
+```
+---
+## Conclusion
+With these simple steps, you can easily download, install, and run your own models using Ollama. Whether you're exploring the capabilities of Llama or building your own custom models, Ollama makes it accessible and efficient.
+- This README provides clear instructions and structured information to help users navigate the process of using Ollama effectively. Adjust any sections as needed based on your specific requirements or additional details you may want to include.