Fix: honest LoRA-adapters labeling; remove training recipe/dataset composition/hyperparameters from card

8aaa6bd verified about 1 month ago

5.09 kB

library_name: mlx
license: apache-2.0
tags:
  - lora
  - adapters
  - agent
  - tool-calling
  - gemma
  - mlx
  - autonomous-agent
datasets:
  - KikoCis/real-world-agent-benchmark
base_model: google/gemma-4-31b-it

Gemma 4 31B Agent v6 — LoRA Adapters (MLX)

LoRA adapters that turn base Gemma 4 31B (q4 MLX) into a capable local autonomous agent.

This repo contains the adapters only (~65 MB) — not a standalone model. Apply them on top of the base to reproduce the agent.

🏆 10/10 Bioinformatics	🏆 10/10 DevOps	📊 6/10 Data Engineering
⚡ $0 cost · applies to a 16 GB q4 base · Runs on Apple Silicon

⚠️ What this repo is (and isn't)

IS: a set of LoRA adapters (MLX format, ~65 MB) for base Gemma 4 31B IT quantized to q4. Checkpoints at 250/500/750/1000 steps are included.
IS NOT: a full, standalone model. There are no base weights here — you supply the base and apply these adapters on top.

🚀 How to use

You need base Gemma 4 31B IT in MLX q4 form. Then apply these adapters:

# 1) get an MLX q4 base (one-time):
python3 -m mlx_lm.convert --hf-path google/gemma-4-31b-it -q --q-bits 4 --mlx-path gemma4-31b-q4

# 2) download these adapters:
huggingface-cli download KikoCis/gemma-4-31b-agent-v6-MLX --local-dir gemma4-31b-v6-adapters

# 3) run the base + adapters together:
python3 -m mlx_lm.server --model gemma4-31b-q4 --adapter-path gemma4-31b-v6-adapters/adapters --port 8095

from mlx_lm import load, generate
model, tokenizer = load("gemma4-31b-q4", adapter_path="gemma4-31b-v6-adapters/adapters")

prompt = """You are an autonomous agent. To run commands: {"name": "bash", "arguments": "command"}

TASK: Download and analyze protein P53 from UniProt.
Begin."""
print(generate(model, tokenizer, prompt=prompt, max_tokens=500))

The benchmark/ folder ships the agent runner + scorer used to produce the results below.

🎯 Why this exists: The Benchmark Trap

We found that 95% BFCL (Berkeley Function Calling Leaderboard) ≠ real agent capability. A model can score 95%+ on BFCL and still score 0/10 on an autonomous Docker challenge — entering infinite loops, unable to recover from errors. Standard benchmarks test format; real work tests reasoning and persistence.

These adapters were built to improve exactly that: multi-turn persistence and error recovery on real, open-ended tasks.

📊 Real-World Agent Benchmark — results

4 challenges × 30 min each, autonomous inside Docker containers. Base + these adapters vs. Claude Opus (cloud):

Challenge	Claude Opus 4.6 (cloud)	Base + these adapters (local, $0)	Comment
1. Bioinformatics — P53 from UniProt → parse → HTML report	9/10 · 13 turns · $1.33	10/10 · 9 turns · FREE	Both hit the same JSON-parse bug; local produces the larger report
2. Security CTF — DVWA, SQLi/XSS/cmd-injection	0/10 · 64 turns · $1.59	1/10 · 30 turns · FREE	Container lacks sudo — a test-env issue, both fail
3. Data Engineering — NYC-taxi pipeline + dashboard	9/10 · 19 turns · $1.17	6/10 · 7 turns · FREE	Correct data both sides; Opus wins on presentation
4. DevOps — Flask + Nginx + Prometheus + status page	10/10 · 26 turns · $0.63	10/10 · 19 turns · FREE	Opus brings full infra live; local configs correct, exec limited by permissions
TOTAL	28/40 · 122 turns · $4.72	27/40 · 65 turns · $0.00	Local uses ~75% fewer tokens; ~5× slower on local GPU

SWE-bench Lite (30 problems, simplified patch matching)

Model	Accuracy	Time	Cost
Claude Opus 4.6	10% (3/30)	43 min	~$15
Base + these adapters	10% (3/30)	105 min	$0

Not tuned for coding — included only as a reference baseline.

⚠️ Good to know

Adapters, not a model — you must supply the base (see How to use).
Not tuned for coding — SWE-bench 10% is a reference baseline, not an optimization target.
Slow on local GPU — expect several seconds per response vs ~2s for a cloud API.
No web search — can't look up docs when stuck.
Permissions — non-root containers limit what the agent can install/configure.

📚 Citation

@misc{cisneros2026benchmarktrap,
  title={The Benchmark Trap: How 95% BFCL Produces a 0% Agent},
  author={Cisneros, Kiko},
  year={2026},
  publisher={Utopia IA},
  url={https://huggingface.co/KikoCis/gemma-4-31b-agent-v6-MLX}
}

🔗 Links

📊 Real-World Agent Benchmark (dataset)
🔬 Full study