Instructions to use educatedlucifer12/OpenMythos-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use educatedlucifer12/OpenMythos-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="educatedlucifer12/OpenMythos-GGUF",
	filename="OpenMythos-27B-Q4_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use educatedlucifer12/OpenMythos-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf educatedlucifer12/OpenMythos-GGUF:Q6_K
# Run inference directly in the terminal:
llama cli -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf educatedlucifer12/OpenMythos-GGUF:Q6_K
# Run inference directly in the terminal:
llama cli -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf educatedlucifer12/OpenMythos-GGUF:Q6_K
# Run inference directly in the terminal:
./llama-cli -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf educatedlucifer12/OpenMythos-GGUF:Q6_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Use Docker

docker model run hf.co/educatedlucifer12/OpenMythos-GGUF:Q6_K

LM Studio
Jan
Ollama
How to use educatedlucifer12/OpenMythos-GGUF with Ollama:
```
ollama run hf.co/educatedlucifer12/OpenMythos-GGUF:Q6_K
```

Unsloth Studio

How to use educatedlucifer12/OpenMythos-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for educatedlucifer12/OpenMythos-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for educatedlucifer12/OpenMythos-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for educatedlucifer12/OpenMythos-GGUF to start chatting

How to use educatedlucifer12/OpenMythos-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "educatedlucifer12/OpenMythos-GGUF:Q6_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use educatedlucifer12/OpenMythos-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf educatedlucifer12/OpenMythos-GGUF:Q6_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default educatedlucifer12/OpenMythos-GGUF:Q6_K

Run Hermes

hermes

Atomic Chat new
Docker Model Runner
How to use educatedlucifer12/OpenMythos-GGUF with Docker Model Runner:
```
docker model run hf.co/educatedlucifer12/OpenMythos-GGUF:Q6_K
```

Lemonade

How to use educatedlucifer12/OpenMythos-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull educatedlucifer12/OpenMythos-GGUF:Q6_K

Run and chat with the model

lemonade run user.OpenMythos-GGUF-Q6_K

List all available models

lemonade list

educatedlucifer12

jabbatheduck commited on 6 days ago

Commit

58e33b1

0 Parent(s):

Duplicate from jabbatheduck/OpenMythos-GGUF

Browse files

Co-authored-by: Jeremy <jabbatheduck@users.noreply.huggingface.co>

Files changed (7) hide show

.gitattributes +43 -0
OpenMythos-27B-Q4_K.gguf +3 -0
OpenMythos-27B-Q5_K.gguf +3 -0
OpenMythos-27B-Q6_K.gguf +3 -0
README.md +136 -0
seceval-1781809723.json +0 -0
seceval-1781811525.json +0 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,43 @@

+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+OpenMythos-F16.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-Q5_k.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-27B-Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-27B-Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
+OpenMythos-27B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

OpenMythos-27B-Q4_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d3c3398ba8b512aa59074efcaa34d7dcabed7381b68a05ae6bcce77e14d7db45
+size 16547399872

OpenMythos-27B-Q5_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9569a64a89a49064cb3d55127453ce810fb860643c4ea467e993177a5becbdf0
+size 19231099072

OpenMythos-27B-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b225034431666b0e4723d42efe0aae088c54a84e42ea5f9547f19cb29e2593c
+size 22082529472

README.md ADDED Viewed

	@@ -0,0 +1,136 @@

+---
+license: apache-2.0
+tags:
+- gguf
+- qwen3.5
+- openmythos
+- build-small-hackathon
+datasets:
+- build-small-hackathon/CVE_Vulnerailities_Detailed
+- himanshu17HF/ArvixImport-Filtered-Final
+base_model:
+- build-small-hackathon/OpenMythos
+- Qwen/Qwen3.6-27B
+---
+# OpenMythos 27B - GGUF
+GGUF quantisation of [build-small-hackathon/OpenMythos](https://huggingface.co/build-small-hackathon/OpenMythos),
+a fine-tune of [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B).
+Converted with `convert_hf_to_gguf.py --no-mtp` from llama.cpp build 9658.
+The fine-tune does not include MTP head weights (dropped during training), so MTP
+is not available in this GGUF.
+## Available Quantisations
+| File | Size | Type |
+|------|------|------|
+| OpenMythos-27B-F16.gguf | 53.8 GB | F16 |
+| OpenMythos-27B-Q5_K.gguf | 18.3 GB | Q5_K_M |
+| OpenMythos-27B-Q4_K.gguf | 15.4 GB | Q4_K_M |
+| OpenMythos-27B-Q6_K.gguf | 21.2 GB | Q6_K |
+## Benchmark
+Evaluated with [SecEval](https://github.com/XuanwuAI/SecEval) (commit 7aef317) on 2189
+multiple-choice security questions. Backend: llama.cpp OpenAI-compatible server, fully
+offloaded to GPU. No chain-of-thought / reasoning enabled (`enable_thinking=false`).
+Prompt formatted with a system prompt requesting letter-only answers (no explanation).
+| Set | Model | Score |
+|-----|-------|-------|
+| A | OpenMythos-27B-Q5_K | 1703 / 2189 (77.8%) |
+| B | VulnLLM-R-7B | 1315 / 2189 (60.1%) |
+### OpenMythos-27B-Q5_K test parameters
+- model: `OpenMythos-27B-Q5_K.gguf`
+- inference: `temp=0.2`, `top_p=0.8`, `top_k=20`, `min_p=0.05`, `repeat_penalty=1.02`
+- benchmark script: `/mnt/storage/SecEval-tmp/run_bench.py`
+- output: `seceval-1781809723.json`
+- prompt speed: 282 tok/s | generation speed: 68 tok/s
+#### Per-topic scores
+| Topic | Score |
+|-------|-------|
+| PenTest | 84.2% |
+| MemorySafety | 83.3% |
+| WebSecurity | 82.7% |
+| Vulnerability | 77.8% |
+| NetworkSecurity | 77.4% |
+| SoftwareSecurity | 75.0% |
+| ApplicationSecurity | 74.8% |
+| SystemSecurity | 73.6% |
+| Cryptography | 71.4% |
+### VulnLLM-R-7B test parameters
+- model: `VulnLLM-R-7B.Q6_K.gguf`
+- inference: same settings as above
+- output: `seceval-1781811525.json`
+- prompt speed: 148 tok/s | generation speed: 39 tok/s
+#### Per-topic scores
+| Topic | Score |
+|-------|-------|
+| PenTest | 70.9% |
+| WebSecurity | 66.4% |
+| Vulnerability | 58.7% |
+| NetworkSecurity | 58.3% |
+| SystemSecurity | 56.4% |
+| SoftwareSecurity | 54.7% |
+| ApplicationSecurity | 54.7% |
+| MemorySafety | 54.2% |
+| Cryptography | 28.6% |
+Full detailed results are included in this repo: `seceval-1781809723.json` and
+`seceval-1781811525.json`.
+## Usage
+### llama-server (recommended)
+```ini
+[OpenMythos-27B]
+model = /mnt/storage/models/OpenMythos/OpenMythos-27B-Q5_K.gguf
+chat-template-file = /mnt/storage/llama-server/chat_template-v15.jinja
+ctx-size = 65536
+cache-type-k = q8_0
+cache-type-v = q8_0
+cache-prompt = on
+cache-reuse = 2048
+batch-size = 4096
+ubatch-size = 4096
+kv-unified = on
+parallel = 1
+gpu-layers = all
+temp = 0.2
+top-p = 0.8
+top-k = 20
+min-p = 0.05
+presence-penalty = 0.2
+repeat-penalty = 1.02
+spec-type = ngram-mod
+spec-draft-n-max = 5
+reasoning-format = deepseek
+swa-checkpoints = 5
+```
+### llama-cli
+```bash
+/mnt/storage/llama.cpp/build/bin/llama-cli \
+  -m /mnt/storage/models/OpenMythos/OpenMythos-27B-Q5_K.gguf \
+  --chat-template-file /mnt/storage/llama-server/chat_template-v15.jinja \
+  -c 65536 -b 4096 --ubatch-size 4096 \
+  --cache-type-k q8_0 --cache-type-v q8_0 \
+  --kv-unified -t 8 -fa \
+  --temp 0.2 --top-p 0.8 --top-k 20 --min-p 0.05 \
+  --presence-penalty 0.2 --repeat-penalty 1.02 \
+  -ngl all \
+  -p "Your prompt here"
+```

seceval-1781809723.json ADDED Viewed

The diff for this file is too large to render. See raw diff

seceval-1781811525.json ADDED Viewed

The diff for this file is too large to render. See raw diff