Instructions to use Doses-AI/Pestle-27B-Ternary-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Doses-AI/Pestle-27B-Ternary-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Doses-AI/Pestle-27B-Ternary-GGUF
# Run inference directly in the terminal:
llama cli -hf Doses-AI/Pestle-27B-Ternary-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Doses-AI/Pestle-27B-Ternary-GGUF
# Run inference directly in the terminal:
llama cli -hf Doses-AI/Pestle-27B-Ternary-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Doses-AI/Pestle-27B-Ternary-GGUF
# Run inference directly in the terminal:
./llama-cli -hf Doses-AI/Pestle-27B-Ternary-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Doses-AI/Pestle-27B-Ternary-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Doses-AI/Pestle-27B-Ternary-GGUF

Use Docker

docker model run hf.co/Doses-AI/Pestle-27B-Ternary-GGUF

LM Studio
Jan

vLLM

How to use Doses-AI/Pestle-27B-Ternary-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Doses-AI/Pestle-27B-Ternary-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Doses-AI/Pestle-27B-Ternary-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Doses-AI/Pestle-27B-Ternary-GGUF

Ollama
How to use Doses-AI/Pestle-27B-Ternary-GGUF with Ollama:
```
ollama run hf.co/Doses-AI/Pestle-27B-Ternary-GGUF
```

Unsloth Studio

How to use Doses-AI/Pestle-27B-Ternary-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Doses-AI/Pestle-27B-Ternary-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Doses-AI/Pestle-27B-Ternary-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Doses-AI/Pestle-27B-Ternary-GGUF to start chatting

How to use Doses-AI/Pestle-27B-Ternary-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Doses-AI/Pestle-27B-Ternary-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Doses-AI/Pestle-27B-Ternary-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Doses-AI/Pestle-27B-Ternary-GGUF with Docker Model Runner:
```
docker model run hf.co/Doses-AI/Pestle-27B-Ternary-GGUF
```

Lemonade

How to use Doses-AI/Pestle-27B-Ternary-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Doses-AI/Pestle-27B-Ternary-GGUF

Run and chat with the model

lemonade run user.Pestle-27B-Ternary-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Doses-AI/Pestle-27B-Ternary-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Doses-AI/Pestle-27B-Ternary-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Doses-AI/Pestle-27B-Ternary-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Doses-AI/Pestle-27B-Ternary-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Doses-AI/Pestle-27B-Ternary-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Doses-AI/Pestle-27B-Ternary-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Pestle-27B-Ternary-GGUF

Run locally - Benchmarks - Whitepaper - Model details - Responsible use

A trained 27B ternary medical model: 6.75 GB weight payload, one 8.48 GB runnable GGUF.

8.2× weight compression vs Qwen3.6-27B FP16 at 96.48% median medical capability retention across 17 matched measures · 101.9% of MedGemma-27B’s medical mean · ~10 tok/s on an 18 GB M3 MacBook

vs IQ2_XXS: 10% smaller · ahead or tied on 9/12 shared text-medical metrics · 12% higher shared medical score/GB

Highlights

Full 27B-class model in one 8.48 GB GGUF.
Strong medical performance across clinical knowledge, biomedical QA, and pharmaceutical retrieval.
Runs locally with Mortar on Apple Silicon, NVIDIA CUDA, and CPU fallback.
Text generation, tool-capable chat templates, and optional vision input.
Apache 2.0 licensed.

Benchmarks

Only completed, validated Pestle runs are shown. All locally measured medical runs for Pestle, Qwen3.6-27B FP16, Bonsai-27B Ternary, and Qwen3.6-27B IQ2_XXS used thinking disabled with greedy decoding (temperature=0, top_p=1, top_k=0), no repetition penalty, and seed 20260805. Published external reference cells are identified separately.

Complete comparison matrix

The categorized launch tables and their recorded values are preserved below.

Text medical benchmark comparison

Benchmark	Pestle-27B-Ternary	Qwen3.6-27B FP16	MedGemma 27B FP16	Bonsai 27B Ternary	Qwen3.6-27B IQ2_XXS
BPW (nominal / deployed)	1.79 / 2.52	16.00 / 16.00	16.00 / 16.00	1.71 / 2.13	2.06 / 2.80
Runnable model artifact (GB)	8.48	55.56	54.02	7.17	9.39
Clinical knowledge & reasoning
MedQA	89.79	93.87	89.8 best-of-5 / 87.7 zero-shot (text)	78.24	89.40
MedXpertQA	32.49	41.10	25.7 (text)	20.82	31.22
MedMCQA	68.85	73.70	74.2 (text)	58.59	68.40
MMLU Anatomy	86.67	92.59	83.7 (text)	68.15	84.44
MMLU Clinical Knowledge	90.19	92.45	86.0 (text)	84.53	90.19
MMLU College Biology	96.53	97.92	96.5 (text)	94.44	97.92
MMLU College Medicine	89.60	90.17	86.1 (text)	84.39	88.44
MMLU Medical Genetics	96.00	99.00	97.0 (text)	89.00	98.00
MMLU Professional Medicine	94.85	95.96	93.4 (text)	88.97	93.38
MMLU Virology	56.02	56.02	53.6 (text)	50.60	56.02
ChemBench	61.72	68.73	--	61.33	--
Biomedical QA & Retrieval
PubMedQA (accuracy)	76.70	79.50	77.2 (multimodal)	77.70	78.30
PubMedQA (macro F1)	62.78	64.61	--	62.17	57.74
BioASQ (token F1)	75.28	76.65	--	75.46	--
BioASQ (exact match)	55.94	59.49	--	54.12	--
PharmaRAG (MRR)	90.15	93.91	--	92.60	--
PharmaRAG (nDCG@10)	84.62	90.84	--	89.43	--

_{Size note: Pestle's trained weight payload is 6.75 GB before packaging as one 8,480,707,488-byte runnable GGUF. “Representation BPW” reports the underlying nominal or idealized low-bit representation: Pestle's 1.79-BPW low-bit base before its BF16 overlay, Bonsai's 1.71-BPW ideal ternary representation, and IQ2_XXS's 2.06-BPW quantized block format. “Effective BPW” and decimal GB use runnable or officially downloadable artifacts. Pestle's runnable GGUF equals 2.52 effective BPW over 26,895,998,464 text parameters. Qwen3.6-27B FP16 uses the official 55,562,855,904-byte safetensors payload; MedGemma 27B text FP16 uses 54,018,098,888 bytes across its official shards; Ternary Bonsai uses the downloadable 7,165,121,600-byte Q2_0 GGUF (2.13 deployed BPW); and IQ2_XXS uses the official 9,388,779,744-byte GGUF (2.79 effective BPW, displayed as 2.80).}

Medical intelligence density

Medical intelligence density comparison across Pestle-27B-Ternary, Qwen3.6-27B IQ2_XXS, Bonsai-27B Ternary, Qwen3.6-27B FP16, and MedGemma-27B FP16

_{Medical intelligence density on 10 shared text-only medical benchmarks, defined as D = -log₂(1 - score/100) / downloadable GB. Pestle records 0.275 at 8.48 GB; Qwen3.6-27B IQ2_XXS records 0.245 at 9.39 GB; Bonsai-27B Ternary records 0.234 using a 7.79 GB package comprising its 7.17 GB GGUF and 0.63 GB optional Q8 vision projector; Qwen3.6-27B FP16 records 0.046 at 55.56 GB; and MedGemma-27B FP16 records 0.041 at 54.02 GB. Higher is better.}

_{Metric source: Prism ML's Bonsai.}

General benchmark comparison

Benchmark	Pestle-27B-Ternary	Qwen3.6-27B FP16	Bonsai 27B Ternary	Qwen3.6-27B Q4_K_XL	Qwen3.6-27B IQ2_XXS	Gemma4-31B FP16
Knowledge & reasoning
MMLU-Redux	83.53	93.5	88.05	93.35	88.93	93.60
MuSR	70.50	72.88	65.87	73.01	66.99	71.03
Math
GSM8K	93.25	95.30	96.06	96.66	89.90	97.57
Coding
HumanEval+	87.20	95.12	93.90	95.73	91.46	96.34
MBPP+	83.33	83.33	81.22	83.86	78.89	84.39
Instruction following
IFEval (prompt-strict)	81.70	88.91	85.03	88.83	84.03	90.57

Pestle-27B-Ternary medical and general benchmark landscape

_{Medical-text benchmarks are shown above broader capability context. Solid connections denote locally paired Pestle and Qwen FP16 measurements.}

Experimental vision

Optional vision uses the separately downloaded projection file and is not included in the text-medical headline comparisons.

Benchmark	Pestle-27B-Ternary	Qwen3.6-27B FP16	MedGemma 27B FP16
CXR14 adjudicated (macro F1)	34.28	--	45.3 (multimodal)
CheXpert expert validation (macro F1)	51.03	--	49.9 (multimodal)
VQA-RAD (tokenized F1)	53.49	66.69	46.7 (multimodal)
SLAKE (tokenized F1)	54.81	76.60	70.3 (multimodal)

Deployment performance

Pestle-27B-Ternary deployment performance and deterministic agreement

Canonical generation protocol

Pestle results use thinking disabled, deterministic greedy decoding (temperature=0, top_p=1, top_k=0), no repetition penalty, and seed 20260805. The locally measured medical Qwen BF16, Bonsai, and IQ2_XXS comparison runs use the same non-thinking sampling protocol. Dataset-specific output ceilings and finish-state handling are documented in the technical report.

Model details

	Pestle-27B-Ternary
Base model	Qwen/Qwen3.6-27B
Architecture	Qwen3.6 (`qwen35` in GGUF metadata)
Parameters	26.9B text parameters
Format	Single-file Pestle ternary GGUF with a BF16 final decoder block
Representation / effective deployed BPW	1.79 / 2.52
Trained weight payload	6.75 GB before GGUF packaging
Runnable text artifact	`pestle-27b-ternary.gguf` (8.48 GB)
Training	Medical-domain optimization and knowledge distillation; one NVIDIA H100, approximately 40 hours
Context length	Up to 262K tokens; practical context depends on available memory
Chat template	Qwen3.6 template, embedded in the GGUF and published as `chat_template.jinja`
Inputs	Text; optional images with `mmproj-pestle-27b-ternary.gguf` (0.93 GB)
Hardware	Apple Silicon via Metal; NVIDIA GPUs via CUDA; CPU fallback
Runtime	Mortar, with a llama.cpp-compatible interface
License	Apache 2.0

The published GGUF is the complete runnable model. No separate overlay assembly or weight conversion is required.

Run locally

1. Download

hf auth login

hf download Doses-AI/Pestle-27B-Ternary-GGUF \
  pestle-27b-ternary.gguf \
  --local-dir models/Pestle-27B-Ternary

For optional vision input, download the matching projection too:

hf download Doses-AI/Pestle-27B-Ternary-GGUF \
  mmproj-pestle-27b-ternary.gguf \
  --local-dir models/Pestle-27B-Ternary

2. Build Mortar

git clone https://github.com/DosesAI/mortar.cpp.git
cd mortar.cpp
./scripts/build-mortar.sh

The build script selects Metal on macOS, CUDA on Linux when a compatible NVIDIA GPU is visible, and CPU otherwise.

For a CPU-only build on macOS or Linux:

./scripts/build-mortar.sh --cpu
./mortar --cpu-only \
  --model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf

On Windows PowerShell:

.\scripts\build-mortar.ps1 -Backend cpu
.\mortar.ps1 --cpu-only `
  --model ..\models\Pestle-27B-Ternary\pestle-27b-ternary.gguf

CPU-only inference uses the same GGUF without conversion. It requires enough system RAM for the model and context and is substantially slower than Metal or CUDA inference.

3. Chat

./mortar \
  --model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf

Or start with a prompt:

./mortar \
  --model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf \
  --prompt "Explain the mechanism of action of metformin."

Mortar applies the validated Pestle runtime and deterministic defaults automatically.

Recommended generation settings

Setting	Value used for reported evaluations
Thinking	Off
Temperature	0
Top-p	1
Top-k	0
Repetition penalty	1
Seed	20260805

The model supports thinking mode; enable it only when desired and treat it as a different evaluation/deployment configuration.

Architecture

Pestle is a trained ternary medical model developed through Doses AI’s medical-domain optimization and knowledge distillation. It preserves the Qwen3.6-27B hybrid-attention architecture while applying Doses AI ternary compression across the language model. The release combines a compact ternary core with a matching-parent BF16 final decoder block and ships as a single runnable GGUF.

Optional vision input uses the separately downloadable matching projection file.

Intended use

Pestle is a research preview intended for evaluation and development of locally hosted medical-text, biomedical retrieval, pharmaceutical, coding, and general assistant systems. It is not released for clinical use.

Responsible use

Pestle is a research preview for evaluation and development. It is not a medical device and is not intended to make diagnosis, treatment, prescribing, triage, or patient-management decisions.

Outputs may be inaccurate, incomplete, biased, or confidently wrong.
Medical outputs require review by qualified professionals and validation for the intended setting.
Do not send identifiable patient information to an environment that is not approved to process it.
Local execution can support privacy and data-residency goals, but does not by itself establish regulatory compliance.
Long outputs can occasionally become repetitive; production systems should use sensible generation limits and monitoring.

Citation

@misc{dosesai2026pestle,
  title        = {Pestle-27B-Ternary},
  author       = {{Doses AI}},
  year         = {2026},
  howpublished = {Hugging Face model release},
  url          = {https://huggingface.co/Doses-AI/Pestle-27B-Ternary-GGUF}
}

Model tree for Doses-AI/Pestle-27B-Ternary-GGUF

Base model

Qwen/Qwen3.6-27B

Quantized

(715)

this model

Evaluation results

Accuracy on MedQA
self-reported

89.790
Mean accuracy on MMLU medical subjects
self-reported

86.890
Mean accuracy on MMLU-Redux 2.0
self-reported

83.530
Token F1 on BioASQ
self-reported

75.280
MRR on PharmaRAG
self-reported

90.150
pass@1 on HumanEval+
self-reported

87.200
pass@1 on MBPP+
self-reported

83.330
Accuracy on GSM8K
self-reported

93.250

Doses-AI
/

Pestle-27B-Ternary-GGUF

Pestle-27B-Ternary-GGUF

Highlights

Benchmarks

Complete comparison matrix

Text medical benchmark comparison

Medical intelligence density

General benchmark comparison

Experimental vision

Deployment performance

Canonical generation protocol

Model details

Run locally

1. Download

2. Build Mortar

3. Chat

Recommended generation settings

Architecture

Intended use

Responsible use

Citation

Links

Model tree for Doses-AI/Pestle-27B-Ternary-GGUF

Evaluation results