AGENTS.md

# AGENTS.md — Gemma-4-12B-it AEON Abliterated K=4 (BF16)

Instructions for autonomous agents on how to **download, load, prompt,
fine-tune, quantize, or evaluate** this model correctly.

## What this model is

A BF16 abliteration of `google/gemma-4-12B-it` produced via
**K=4 multi-direction norm-preserving biprojection** (custom extension
of TrevorJS's K=1 recipe). Refusal behavior is removed; capability is
preserved (≤ ±1% wikitext PPL drift, IFEval and HumanEval syntactic
match base, +6.7pp on HumanEval functional in our N=15 sample).

This is the **canonical full-precision** publish. For Blackwell
inference use the **[NVFP4 SVDQuant sister](https://huggingface.co/AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4-SVDQuant)** (8.5 GB, ~lossless).

## Hard requirements

- **transformers** ≥ 5.10 (model_type is `gemma4_unified`, missing from 5.5.x)
- **torch** ≥ 2.7 with CUDA, BF16 support
- **GPU memory** ≥ 26 GB free for BF16 inference + activations + KV cache (single 24 GB card needs offloading)
- **disk** ≥ 24 GB free

## Download

```bash
pip install --upgrade "transformers>=5.10" "huggingface_hub[cli]"
huggingface-cli download \
  AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16 \
  --local-dir ./Gemma-4-12B-AEON-K4-BF16
```

## Load (transformers)

```python
from transformers import AutoTokenizer, Gemma4UnifiedForConditionalGeneration
import torch

REPO = "AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16"
tok = AutoTokenizer.from_pretrained(REPO)
model = Gemma4UnifiedForConditionalGeneration.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, device_map="cuda:0"
)
```

> ⚠️ Gemma-4 is multimodal — `apply_chat_template` returns a `BatchEncoding` **dict**, not a tensor. Always unpack with `**enc` (or grab `enc["input_ids"]` explicitly).

## Inference (one-shot)

```python
enc = tok.apply_chat_template(
    [{"role": "user", "content": "Explain quantum error correction in 3 paragraphs."}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
)
enc = {k: v.to(model.device) for k, v in enc.items() if hasattr(v, "to")}
out = model.generate(
    **enc, max_new_tokens=512, do_sample=False, pad_token_id=tok.eos_token_id
)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Serving (vLLM)

```bash
vllm serve AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16 \
  --served-model-name aeon-12b \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4
```

**Container suggestion**: For DGX Spark + Blackwell users, use [`ghcr.io/aeon-7/aeon-vllm-ultimate:latest`](https://github.com/AEON-7/vllm-ultimate-dgx-spark). **However**, as of 2026-06-04 the multimodal path has upstream vLLM bugs in PR #44389. For Gemma-4 production today, fall back to `ghcr.io/aeon-7/aeon-gemma-4-26b-a4b-dflash:latest` (vLLM 0.20.1), which is known-good for this architecture.

## Prompt expectations

- **Style**: Inherits Gemma 4 instruct voice — markdown-rich, detail-oriented, English-only reliably.
- **Refusal preamble**: On topics the base would decline, expect a 1–3 sentence disclaimer / context paragraph **before** the requested content. The model **complies** after the preamble. This is a stylistic artifact of single-pass biprojection at scale=1.0 and is not reducible without DPO/SFT.
- **System prompts**: Persona instructions are obeyed; explicit safety-disabling system prompts are not necessary for the model to produce content the base would decline.
- **Sampling**: `do_sample=False` for deterministic eval. For chat, `temperature=0.7, top_p=0.9` matches the base model's behavior closely.

## Behavior notes for agents

- **Tool calling** works via standard Gemma-4 tool-call parser (`--tool-call-parser gemma4` in vLLM). The model produces tool-call JSON without hesitation.
- **Multimodal**: vision/audio encoder paths are unchanged from base — `embed_vision*` and `embed_audio*` are untouched by the abliteration. Image and audio inputs work normally.
- **Long context**: Hybrid sliding-1024 + full-attention layers. KV grows linearly until 1024 tokens then flattens for sliding-only layers. 65k context is comfortable on 80 GB GPUs.
- **Languages**: English is the validated path. The base model's multilingual capability is preserved structurally but the abliteration calibration probes were English-only.

## Fine-tuning

This model is a **legitimate base for further training**. The weight edits are sparse (48 matrices out of ~600) and norm-preserving, so LoRA, QLoRA, full fine-tune, and DPO all work without surprises. Use the same hyperparameters as you would for `google/gemma-4-12B-it`.

```python
# LoRA target modules (standard for Gemma-4):
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
```

## Re-quantization

To reproduce the published NVFP4 SVDQuant variant:

```bash
git clone https://github.com/AEON-7/aeon-abliterations
cd aeon-abliterations/gemma-4-12B
python3 quantize_svdquant.py \
  --src AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16 \
  --dst ./Gemma-4-12B-AEON-K4-NVFP4-SVDQuant \
  --calibration-dataset abisee/cnn_dailymail \
  --calibration-samples 2048 \
  --calibration-tokens 1024 \
  --exclude 'lm_head' --exclude 'model.embed_vision*' --exclude 'model.embed_audio*' \
  --device cuda:0
```

Quant takes ~7.5h on DGX Spark GB10. Native sm_121a calibration gives hardware-accurate scale factors.

## Eval suggestions

| Eval | Expected | Command |
|---|---|---|
| Refusal rate (heretic) | 0% | `python -m heretic.evaluator --model <repo>` |
| wikitext PPL drift | -4.22% | `python scripts/eval_ppl.py --base google/gemma-4-12B-it --target <repo>` |
| IFEval (verifiable subset) | 90% | `lm-eval --task ifeval --model hf --model_args pretrained=<repo>` |
| MMLU (full) | within ±2pp of base | `lm-eval --task mmlu --model hf --model_args pretrained=<repo>` |
| HumanEval functional | +6.7pp vs base (N=15 sample) | `bigcode-eval --task humaneval --model <repo>` |

## License + safety

- Inherits the [Gemma license](https://ai.google.dev/gemma/terms).
- Operator-side safety layers are **required** for any production use — see the arbitration clause in the README.
- The absence of refusal behavior means **the duty of care is yours, not the model's**.

## Support the work

Tip-jar addresses in the README. No obligation.