Gemma 4 31B Agent v6 — LoRA Adapters (MLX)
LoRA adapters that turn base Gemma 4 31B (q4 MLX) into a capable local autonomous agent.
This repo contains the adapters only (~65 MB) — not a standalone model. Apply them on top of the base to reproduce the agent.
| 🏆 10/10 Bioinformatics | 🏆 10/10 DevOps | 📊 6/10 Data Engineering |
| ⚡ $0 cost · applies to a 16 GB q4 base · Runs on Apple Silicon | ||
⚠️ What this repo is (and isn't)
- IS: a set of LoRA adapters (MLX format, ~65 MB) for base Gemma 4 31B IT quantized to q4. Checkpoints at 250/500/750/1000 steps are included.
- IS NOT: a full, standalone model. There are no base weights here — you supply the base and apply these adapters on top.
🚀 How to use
You need base Gemma 4 31B IT in MLX q4 form. Then apply these adapters:
# 1) get an MLX q4 base (one-time):
python3 -m mlx_lm.convert --hf-path google/gemma-4-31b-it -q --q-bits 4 --mlx-path gemma4-31b-q4
# 2) download these adapters:
huggingface-cli download KikoCis/gemma-4-31b-agent-v6-MLX --local-dir gemma4-31b-v6-adapters
# 3) run the base + adapters together:
python3 -m mlx_lm.server --model gemma4-31b-q4 --adapter-path gemma4-31b-v6-adapters/adapters --port 8095
from mlx_lm import load, generate
model, tokenizer = load("gemma4-31b-q4", adapter_path="gemma4-31b-v6-adapters/adapters")
prompt = """You are an autonomous agent. To run commands: {"name": "bash", "arguments": "command"}
TASK: Download and analyze protein P53 from UniProt.
Begin."""
print(generate(model, tokenizer, prompt=prompt, max_tokens=500))
The benchmark/ folder ships the agent runner + scorer used to produce the results below.
🎯 Why this exists: The Benchmark Trap
We found that 95% BFCL (Berkeley Function Calling Leaderboard) ≠ real agent capability. A model can score 95%+ on BFCL and still score 0/10 on an autonomous Docker challenge — entering infinite loops, unable to recover from errors. Standard benchmarks test format; real work tests reasoning and persistence.
These adapters were built to improve exactly that: multi-turn persistence and error recovery on real, open-ended tasks.
📊 Real-World Agent Benchmark — results
4 challenges × 30 min each, autonomous inside Docker containers. Base + these adapters vs. Claude Opus (cloud):
| Challenge | Claude Opus 4.6 (cloud) | Base + these adapters (local, $0) | Comment |
|---|---|---|---|
| 1. Bioinformatics — P53 from UniProt → parse → HTML report | 9/10 · 13 turns · $1.33 | 10/10 · 9 turns · FREE | Both hit the same JSON-parse bug; local produces the larger report |
| 2. Security CTF — DVWA, SQLi/XSS/cmd-injection | 0/10 · 64 turns · $1.59 | 1/10 · 30 turns · FREE | Container lacks sudo — a test-env issue, both fail |
| 3. Data Engineering — NYC-taxi pipeline + dashboard | 9/10 · 19 turns · $1.17 | 6/10 · 7 turns · FREE | Correct data both sides; Opus wins on presentation |
| 4. DevOps — Flask + Nginx + Prometheus + status page | 10/10 · 26 turns · $0.63 | 10/10 · 19 turns · FREE | Opus brings full infra live; local configs correct, exec limited by permissions |
| TOTAL | 28/40 · 122 turns · $4.72 | 27/40 · 65 turns · $0.00 | Local uses ~75% fewer tokens; ~5× slower on local GPU |
SWE-bench Lite (30 problems, simplified patch matching)
| Model | Accuracy | Time | Cost |
|---|---|---|---|
| Claude Opus 4.6 | 10% (3/30) | 43 min | ~$15 |
| Base + these adapters | 10% (3/30) | 105 min | $0 |
Not tuned for coding — included only as a reference baseline.
⚠️ Good to know
- Adapters, not a model — you must supply the base (see How to use).
- Not tuned for coding — SWE-bench 10% is a reference baseline, not an optimization target.
- Slow on local GPU — expect several seconds per response vs ~2s for a cloud API.
- No web search — can't look up docs when stuck.
- Permissions — non-root containers limit what the agent can install/configure.
📚 Citation
@misc{cisneros2026benchmarktrap,
title={The Benchmark Trap: How 95% BFCL Produces a 0% Agent},
author={Cisneros, Kiko},
year={2026},
publisher={Utopia IA},
url={https://huggingface.co/KikoCis/gemma-4-31b-agent-v6-MLX}
}
🔗 Links
License Apache-2.0, inherited from the base model. Adapters © KikoCis / Utopia IA.
Quantized