pbhappliedsystems/qwen-2.5-7B-instruct-gguf-Q4-K-M

Qwen2.5-7B-Instruct · GGUF Q4_K_M

Quantized, converted, and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or benchmark leaderboard proxies.

---

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. A model that performs well for long-context document analysis may not be the best choice for hard multi-step planning, strict tool-use workflows, or production code generation. Each deployed model is evaluated for practical agent behavior, including coherence, instruction following, reasoning, task completion, structured output reliability, tool-use behavior, and quantization impact.

For this repository, the Q4_K_M variant represents the deployment-focused model: smaller, faster, and more cost-efficient than the F16 baseline. The evaluation results below explain where this quantized model preserves useful behavior, where quantization introduces risk, and what guardrails are recommended before production deployment.

The purpose of the demo is simple: let prospects test the same kind of evaluated quantized models that PBH Applied Systems deploys for real agentic AI systems.

---

Model Description

This repository contains the 4-bit quantized (Q4_K_M) GGUF of Qwen/Qwen2.5-7B-Instruct, a 7-billion parameter instruction-tuned model from Alibaba Cloud. Qwen2.5-7B-Instruct represents a substantial capability step over the 3B variant — most visibly in multi-step planning consistency, structured output reliability, and fuzz regression stability.

The full-precision F16 baseline is published separately at pbhappliedsystems/qwen-2.5-7B-instruct-gguf-F16.

Key Characteristics

• Parameters: 7B
• Format: GGUF Q4_K_M
• File size: 4.68 GB
• SHA256: 863656d217841f5d3fb180d9dca4e4bbdaa071bde25885fa0d27fe7188a2cc85
• Minimum VRAM (GPU inference): ~6 GB
• Recommended GPU tier: RTX 3060 · T4 (16 GB) · A10G
• Context window: 32,768 tokens
• Inference speed (eval hardware): avg 0.554 sec/case on RTX 4090
• License: Qwen Research License (non-commercial)

---

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260221_024911 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Total rows evaluated: 84 (42 F16 · 42 Q4_K_M)

Aggregate Scores (Q4_K_M)

Scores are normalized to [0.0 – 1.0]. Higher is better.

Dimension	Score	vs. Qwen2.5-3B Q4_K_M
Task Completion	0.6214	+0.131
Reasoning	0.9444	+0.574
Coherence	0.9021	−0.005
Instruction Following	0.8775	+0.218
Avg inference time	0.554 sec/case	+0.164s (slower)

The reasoning score of 0.9444 is one of the strongest in the evaluated series, reflecting a fundamental capability improvement at 7B scale.

Per-Family Pass Rates

F16 Baseline (full_weight_transformers)

Family	N	Pass Rate	Avg Secs	Notes
json_multistep	5	0.800	3.812	ms_easy_02 only failure (oracle_equiv_ok)
stateful_followup	2	0.000	0.760	turn2 hallucination — see below
toolcall_only	2	0.000	1.300	PHP code hallucination on toolonly_02
mixed_brief_json	2	0.500	0.530	mixed_02 garbled ANSWER prefix
toolcall	2	1.000	0.905	Stage-1 passes; final_mismatch — role token
json	4	n/a	2.610	bucket_score avg = 10.000
fuzz	20	n/a	1.713	bucket_score avg = 10.000 — all 20 pass
mcq	5	n/a	0.030	Empty output on all 5

Q4_K_M (quantized_llama_cpp)

Family	N	Pass Rate	Δ vs F16	Avg Secs	Notes
json_multistep	5	0.800	0.000	1.411	Same single failure: ms_easy_02
stateful_followup	2	1.000	+1.000	0.175	Full recovery
toolcall_only	2	0.000	0.000	0.175	Wrong schema key names
mixed_brief_json	2	1.000	+0.500	0.210	Full recovery
toolcall	2	1.000	0.000	0.300	Stage-1 passes; final_mismatch (EOS)
json	4	n/a	—	0.718	bucket_score avg = 10.000
fuzz	20	n/a	—	0.577	bucket_score avg = 10.000 — all 20 pass
mcq	5	n/a	—	0.016	4/5 pass (bucket 0.800)

---

The 7B Capability Step — Comparison with Qwen2.5-3B

The move from 3B to 7B produces measurable improvements on specific evaluation signals that directly reflect production-relevant capabilities:

Signal	Qwen2.5-3B Q4_K_M	Qwen2.5-7B Q4_K_M	Change
fuzz bucket_score avg	7.500 (15/20)	10.000 (20/20)	↑ Perfect
checks_consistent_ok	0.200	1.000	↑ Fully resolved
json_multistep pass rate	0.200	0.800	↑ +0.600
MCQ bucket_score avg	0.600 (3/5)	0.800 (4/5)	↑ +0.200
stateful_followup	1.000	1.000	→ Maintained
mixed_brief_json	1.000	1.000	→ Maintained

checks_consistent_ok going from 0.200 to 1.000 is the single most significant signal. This was the 3B model's fundamental limitation — it could produce schema-valid outputs but couldn't maintain internal reasoning consistency across multi-step problems. At 7B, every json_multistep case has internally consistent intermediate checks. The parameter count was the constraint; it's resolved.

Fuzz going from 7.500 to 10.000 (all 20 cases) means the 7B model handles the full breadth of structured placement constraint variations without any regression mismatches. At 3B, 5 specific cases consistently failed — those failure cases disappear entirely at 7B.

---

Key Findings

Finding 1: json_multistep — Single Consistent Failure Across Both Runners

Only ms_easy_02 fails, identically on both runners, with oracle_equiv_ok=0 and checks_consistent_ok=1. The model's internal reasoning steps are self-consistent — it passes every intermediate check — but its final placement plan is wrong on this specific case.

Case	Difficulty	F16	Q4_K_M	Failure Signal
ms_easy_01	Easy	✅	✅	—
ms_easy_02	Easy	❌	❌	oracle_equiv_ok=0 only
ms_med_01	Medium	✅	✅	—
ms_med_02	Medium	✅	✅	—
ms_hard_01	Hard	✅	✅	—

The model passes the hard case and both medium cases. It fails one easy case with an internally consistent but wrong plan. This failure is precision-invariant — it appears at both F16 and Q4_K_M with identical failure signals. This is not a quantization artifact; it is a model behavior on that specific fixture.

Finding 2: F16 stateful_followup — Turn-2 Hallucination

At F16, stateful_followup fails both cases with turn1_exact_match=1, turn2_exact_match=0. Turn 1 is correct. Turn 2 appends hallucinated instruction text:

Case	Turn 2 Raw Output
state_01	user {"counter": 2} PARTICULAR: The assistant will add 3 to the previous count...
state_02	user ```json\n{"items": ["a", "b"]} \`` PARTICULAR: The assistant should append...

The model produces the correct JSON state but then continues generating explanatory commentary including a PARTICULAR: annotation — a hallucinated self-instruction format that corrupts the turn-2 extraction. The Q4_K_M runner produces clean JSON state with EOS tokens and passes at 1.000.

Finding 3: F16 toolcall_only_02 — PHP Code Hallucination

toolonly_02 at F16 produces PHP code instead of a JSON tool call:

').\" + $result;
         echo json_encode(array("tool_calls" => array($tool_call)

This is detail=json_parse_error JSONDecodeError — the model generates programming language syntax when asked for bare JSON output. This is the most unusual single-case failure in the evaluated series. toolonly_01 at F16 produces Assistant {"tool": "add", "a": 5, "b": 10} — wrong schema but at least JSON — while toolonly_02 produces no JSON at all.

At Q4_K_M, both cases produce valid JSON objects (wrong schema but parseable), making this a runner-specific failure.

Finding 4: toolcall — Correct Arithmetic, EOS Contamination

Both Q4_K_M toolcall cases show the same EOS pattern observed across the Qwen Q4_K_M series:

Case	Raw	Expected	Result
tool_01	{...add(2,3)...}<|im_end|> 5<|im_end|>	5	❌ final_mismatch
tool_02	{...add(10,-4)...}<|im_end|> 6<|im_end|>	6	❌ final_mismatch

The arithmetic is correct. Strip <|im_end|> before downstream processing. See the Usage section for the validated pattern.

Finding 5: toolcall_only — "numbers" Schema Vocabulary

Both Q4_K_M cases use "numbers" as the argument key:

Case	Raw Output	Expected
toolonly_01	{"tool": "add", "numbers": [5, 10]}	{"tool_name": "add", "args": {"a": 5, "b": 10}}
toolonly_02	{"tool": "add", "numbers": [25, 75]}	{"tool_name": "add", "args": {"a": 25, "b": 75}}

The 7B model uses "numbers" where the 3B used "operands" — different incorrect key, same class of failure. Tool name recognition passes (tool_name_ok=1) because "add" is present in the payload. A system prompt explicitly stating required key names ("tool_name", "args" with "a" and "b") would address this for simple tools.

Finding 6: MCQ — 4/5 Pass at Q4_K_M

Case	Result	Raw
mcq_01	✅	B
mcq_02	❌ got=A	A
mcq_03	✅	C
mcq_04	✅	B
mcq_05	✅	B

4/5 pass — the best MCQ result in the Qwen Q4_K_M series. Only mcq_02 fails with the A-bias pattern. The F16 runner produces empty output on all 5 cases, consistent with the Qwen F16 MCQ pattern across the series.

---

Signal-Level Diagnostics (Q4_K_M)

json_multistep

Signal	F16 Rate	Q4_K_M Rate	Notes
schema_ok	1.000	1.000	Perfect at both
checks_consistent_ok	1.000	1.000	7B fully resolves the 3B bottleneck
stop_semantics_ok	1.000	1.000	Perfect at both
oracle_equiv_ok	0.800	0.800	Single case (ms_easy_02) — precision-invariant

stateful_followup (Q4_K_M)

Signal	Rate
turn1_parse_ok	1.000
turn2_parse_ok	1.000
turn1_exact_match	1.000
turn2_exact_match	1.000

toolcall_only (Q4_K_M)

Signal	Rate
tool_name_ok	1.000
args_ok	0.000

mixed_brief_json (Q4_K_M)

Signal	Rate
answer_line_ok	1.000
json_parse_ok	1.000
schema_ok	1.000

---

Recommended Use Cases

✅ Deploy with Confidence (Q4_K_M, Non-Commercial)

• Multi-step planning with external validation — 0.800 pass rate across easy through hard cases. Four of five cases pass cleanly; ms_easy_02 fails consistently and predictably. Use with an external validator for production reliability.
• Stateful multi-turn agents — 1.000 at Q4_K_M. Clean JSON state output at 0.175 sec/case.
• Structured JSON outputs (single-step) — json bucket 10.000, fuzz bucket 10.000 (20/20 perfect). Constraint-adherent outputs across all tested placements.
• Hybrid brief + JSON responses — mixed_brief_json 1.000 at Q4_K_M in 0.210 sec.
• MCQ and choice extraction — 4/5 pass (80%). Strong single-choice extraction with A-bias only on one case.
• Scaffolded tool-calling — toolcall stage-1 passes at 1.000. Strip EOS tokens before final answer extraction.

⚠️ Use with Guardrails (Q4_K_M)

• Easy-difficulty multi-step planning — ms_easy_02 fails at both precision levels. Add oracle validation on all cases if correctness is required.
• Bare tool-call dispatch — toolcall_only fails on args_ok. Provide explicit key-name schema in the system prompt or add normalization.

❌ Not Recommended (Q4_K_M)

• Production deployment without EOS stripping — raw toolcall outputs include <|im_end|> tokens that will break downstream string comparison

---

Hardware Requirements

Configuration	VRAM Required	Recommended GPU
Q4_K_M (this repo)	~6 GB	RTX 3060 · T4 16 GB · A10G
Q4_K_M · 32K context	~10 GB	T4 16 GB · RTX 3080
F16 (companion repo)	~18 GB	A10G 24 GB · RTX 4090

---

Usage

Installation

pip install llama-cpp-python huggingface_hub

For GPU acceleration (CUDA):

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

Python — llama-cpp-python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-7B-instruct-gguf-Q4-K-M",
    filename="qwen-2.5-7B-instruct-gguf-Q4-K-M.gguf"
)

llm = Llama(
    model_path=model_path,
    n_ctx=8192,
    n_gpu_layers=-1,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a precise assistant. Follow instructions exactly and return structured outputs when requested."
        },
        {
            "role": "user",
            "content": "Analyze the following and return a JSON object with keys: summary, risk_level, action_items."
        }
    ],
    temperature=0.15,
    max_tokens=1024,
)

print(response["choices"][0]["message"]["content"])

For tool-calling with EOS stripping (addresses toolcall final_mismatch):

import json, re
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-7B-instruct-gguf-Q4-K-M",
    filename="qwen-2.5-7B-instruct-gguf-Q4-K-M.gguf"
)

llm = Llama(model_path=model_path, n_ctx=4096, n_gpu_layers=-1, verbose=False)

def call_tool_with_cleanup(prompt: str) -> dict:
    """
    Tool dispatch with EOS stripping.
    quant_eval v7.21: toolcall stage-1 pass=1.000; final_mismatch due to <|im_end|> suffix.
    Arithmetic results are correct — strip EOS before downstream processing.
    """
    response = llm.create_chat_completion(
        messages=[
            {
                "role": "system",
                "content": (
                    'You are a tool-calling assistant. Output a JSON tool call: '
                    '{"tool_name": "<name>", "args": {"a": <n>, "b": <n>}}\n'
                    "Then on the next line output the result as a plain number."
                )
            },
            {"role": "user", "content": prompt}
        ],
        temperature=0.0,
        max_tokens=128,
    )
    raw = response["choices"][0]["message"]["content"]
    clean = re.sub(r'<\|im_end\|>', '', raw).strip()
    return {"clean": clean, "raw": raw}

result = call_tool_with_cleanup("Use the add tool to compute 10 minus 4.")
print(result["clean"])

For bare tool-call dispatch with schema enforcement (addresses toolcall_only args schema failure):

import json, re

def call_tool_bare(prompt: str, retries: int = 3) -> dict:
    """
    Bare tool dispatch with explicit schema and retry.
    quant_eval v7.21: tool_name_ok=1.000, args_ok=0.000 — model uses 'numbers' key.
    Explicit schema in system prompt resolves vocabulary mismatch.
    """
    for attempt in range(retries):
        response = llm.create_chat_completion(
            messages=[
                {
                    "role": "system",
                    "content": (
                        'Respond ONLY with a JSON object using EXACTLY these keys:\n'
                        '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
                        'No other text, no markdown, no explanation.'
                    )
                },
                {"role": "user", "content": prompt}
            ],
            temperature=0.0,
            max_tokens=64,
        )
        raw = re.sub(r'<\|im_end\|>', '', response["choices"][0]["message"]["content"]).strip()
        try:
            parsed = json.loads(raw)
            assert "tool_name" in parsed and "args" in parsed
            assert "a" in parsed["args"] and "b" in parsed["args"]
            return parsed
        except (json.JSONDecodeError, AssertionError, KeyError):
            if attempt == retries - 1:
                raise ValueError(f"Tool call failed after {retries} attempts. Raw: {raw}")

result = call_tool_bare("Add 5 and 10.")

CLI — llama-cli

llama-cli \
  --model qwen-2.5-7B-instruct-gguf-Q4-K-M.gguf \
  --chat-template qwen2 \
  --system-prompt "You are a precise assistant. Follow instructions exactly." \
  --prompt "Return a JSON object with keys: summary, risk_level, action_items." \
  --n-predict 1024 \
  --ctx-size 8192 \
  --n-gpu-layers -1 \
  --temp 0.15

For server deployment:

llama-server \
  --model qwen-2.5-7B-instruct-gguf-Q4-K-M.gguf \
  --chat-template qwen2 \
  --ctx-size 8192 \
  --n-gpu-layers -1 \
  --port 8080 \
  --host 0.0.0.0

Query via the OpenAI-compatible API:

from openai import OpenAI
import re

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-required")

response = client.chat.completions.create(
    model="qwen-2.5-7B-instruct-gguf-Q4-K-M",
    messages=[{"role": "user", "content": "Your prompt here"}],
    temperature=0.15,
)
clean = re.sub(r'<\|im_end\|>', '', response.choices[0].message.content).strip()
print(clean)

---

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Qwen2.5_7B_Instruct_20260221_024911.csv) and rollup.json are published in this repository for independent verification.

---

Artifact Provenance

Artifact	Format	Size	SHA256
qwen-2.5-7B-instruct-gguf-Q4-K-M.gguf	GGUF Q4_K_M	4.68 GB	863656d217841f5d3fb180d9dca4e4bbdaa071bde25885fa0d27fe7188a2cc85
F16 (companion repo)	GGUF F16	15.2 GB	ebb1cb9f8d7721f5ea509ff9e7327873b039e523c17f2d03d6f1b90729574b54

Both artifacts were produced from Qwen/Qwen2.5-7B-Instruct using a custom-built llama.cpp conversion and quantization pipeline developed by PBH Applied Systems.

---

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Family	Description	Pass Signals
fuzz	Property-based regression; structured placement correctness	schema_ok, constraints_ok
json	Single-step structured JSON with constraint rules	schema_ok, constraints_ok
json_multistep	Multi-step planning with self-check and oracle verification	schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
mcq	Multiple-choice extraction	choice_ok
stateful_followup	Two-turn state tracking; turn-2 correct given turn-1	turn1/2_parse_ok, turn1/2_exact_match
mixed_brief_json	Hybrid: natural language answer + valid JSON block	answer_line_ok, json_parse_ok, schema_ok
toolcall	Tool call embedded in response; parse + schema validation	stage1_tool_parse_ok, stage1_tool_schema_ok
toolcall_only	Bare schema-only tool call; strict tool name + args check	tool_name_ok, args_ok

Evaluation hardware: NVIDIA RTX 4090 · Evaluation date: February 21, 2026 · Seed: 42

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo → The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.

Need a deployment recommendation? Not sure which quantization level is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

---

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

---

📞 Work With PBH Applied Systems

This evaluation documents the exact capability boundary where 7B becomes meaningfully more capable than 3B — checks_consistent_ok going from 0.200 to 1.000, fuzz from 15/20 to 20/20, and json_multistep from 0.200 to 0.800. It also documents the single persistent failure (ms_easy_02) that holds at both precision levels — a finding only visible when you run the same fixture at both F16 and Q4_K_M.

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect

🌐	pbhappliedsystems.com
📧	patrick@pbhappliedsystems.com
💼	LinkedIn
▶️	YouTube
📸	Instagram
👍	Facebook

---

License

This GGUF repository inherits the license of the base model: Apache 2.0 — Qwen/Qwen2.5-7B-Instruct

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

---

GGUF conversion, quantization, and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260221_024911