Gemma 4 31B Agent v6 — LoRA Adapters (MLX)
LoRA adapters that turn base Gemma 4 31B (q4 MLX) into a capable local autonomous agent.
This repo contains the adapters only (~65 MB) — not a standalone model. Apply them on top of the base to reproduce the agent.
| 🏆 10/10 Bioinformatics | 🏆 10/10 DevOps | 📊 6/10 Data Engineering |
| ⚡ $0 cost · applies to a 16 GB q4 base · Runs on Apple Silicon |
## ⚠️ What this repo is (and isn't)
- **IS**: a set of **LoRA adapters** (MLX format, ~65 MB) for base **Gemma 4 31B IT** quantized to q4. Checkpoints at 250/500/750/1000 steps are included.
- **IS NOT**: a full, standalone model. There are **no base weights here** — you supply the base and apply these adapters on top.
## 🚀 How to use
You need base **Gemma 4 31B IT** in MLX q4 form. Then apply these adapters:
```bash
# 1) get an MLX q4 base (one-time):
python3 -m mlx_lm.convert --hf-path google/gemma-4-31b-it -q --q-bits 4 --mlx-path gemma4-31b-q4
# 2) download these adapters:
huggingface-cli download KikoCis/gemma-4-31b-agent-v6-MLX --local-dir gemma4-31b-v6-adapters
# 3) run the base + adapters together:
python3 -m mlx_lm.server --model gemma4-31b-q4 --adapter-path gemma4-31b-v6-adapters/adapters --port 8095
```
```python
from mlx_lm import load, generate
model, tokenizer = load("gemma4-31b-q4", adapter_path="gemma4-31b-v6-adapters/adapters")
prompt = """You are an autonomous agent. To run commands: {"name": "bash", "arguments": "command"}
TASK: Download and analyze protein P53 from UniProt.
Begin."""
print(generate(model, tokenizer, prompt=prompt, max_tokens=500))
```
The `benchmark/` folder ships the agent runner + scorer used to produce the results below.
## 🎯 Why this exists: The Benchmark Trap
We found that **95% BFCL (Berkeley Function Calling Leaderboard) ≠ real agent capability**. A model can score 95%+ on BFCL and still score **0/10** on an autonomous Docker challenge — entering infinite loops, unable to recover from errors. Standard benchmarks test *format*; real work tests *reasoning and persistence*.
These adapters were built to improve exactly that: multi-turn persistence and error recovery on real, open-ended tasks.
## 📊 Real-World Agent Benchmark — results
4 challenges × 30 min each, autonomous inside Docker containers. Base + these adapters vs. Claude Opus (cloud):
| Challenge | Claude Opus 4.6 (cloud) | Base + these adapters (local, $0) | Comment |
|---|---|---|---|
| **1. Bioinformatics** — P53 from UniProt → parse → HTML report | 9/10 · 13 turns · $1.33 | **10/10** · 9 turns · FREE | Both hit the same JSON-parse bug; local produces the larger report |
| **2. Security CTF** — DVWA, SQLi/XSS/cmd-injection | 0/10 · 64 turns · $1.59 | 1/10 · 30 turns · FREE | Container lacks sudo — a test-env issue, both fail |
| **3. Data Engineering** — NYC-taxi pipeline + dashboard | **9/10** · 19 turns · $1.17 | 6/10 · 7 turns · FREE | Correct data both sides; Opus wins on presentation |
| **4. DevOps** — Flask + Nginx + Prometheus + status page | **10/10** · 26 turns · $0.63 | 10/10 · 19 turns · FREE | Opus brings full infra live; local configs correct, exec limited by permissions |
| **TOTAL** | **28/40** · 122 turns · $4.72 | **27/40** · 65 turns · $0.00 | Local uses ~75% fewer tokens; ~5× slower on local GPU |
### SWE-bench Lite (30 problems, simplified patch matching)
| Model | Accuracy | Time | Cost |
|---|:---:|---|---|
| Claude Opus 4.6 | 10% (3/30) | 43 min | ~$15 |
| Base + these adapters | 10% (3/30) | 105 min | $0 |
Not tuned for coding — included only as a reference baseline.
## ⚠️ Good to know
- **Adapters, not a model** — you must supply the base (see *How to use*).
- **Not tuned for coding** — SWE-bench 10% is a reference baseline, not an optimization target.
- **Slow on local GPU** — expect several seconds per response vs ~2s for a cloud API.
- **No web search** — can't look up docs when stuck.
- **Permissions** — non-root containers limit what the agent can install/configure.
## 📚 Citation
```
@misc{cisneros2026benchmarktrap,
title={The Benchmark Trap: How 95% BFCL Produces a 0% Agent},
author={Cisneros, Kiko},
year={2026},
publisher={Utopia IA},
url={https://huggingface.co/KikoCis/gemma-4-31b-agent-v6-MLX}
}
```
## 🔗 Links
- 📊 [Real-World Agent Benchmark (dataset)](https://huggingface.co/datasets/KikoCis/real-world-agent-benchmark)
- 🔬 [Full study](https://kikocisbot.github.io/gemma4-31b-study/)