---
base_model: google/gemma-4-31B-it
library_name: transformers
license: gemma
language:
  - en
pipeline_tag: text-generation
tags:
  - caveman
  - style-transfer
  - fine-tuned
  - gemma
  - gemma-license
  - merged
  - bf16
---

# gemma-4-31B-caveman

Gemma 4 31B fine-tuned to speak [caveman-mode](https://github.com/JuliusBrussee/caveman) natively. Full bf16 weights, drop-in replacement for `google/gemma-4-31B-it`.

Drops articles, filler, pleasantries, hedging. Allows fragments. Keeps code blocks, function names, error strings, and CLI commands **byte-exact**. Pattern: `[thing] [action] [reason]. [next step].`

For the lighter LoRA-adapter variant see [`JBrussee/gemma-4-31B-caveman-lora`](https://huggingface.co/JBrussee/gemma-4-31B-caveman-lora).

## Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("JBrussee/gemma-4-31B-caveman")
model = AutoModelForCausalLM.from_pretrained(
    "JBrussee/gemma-4-31B-caveman",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

msgs = [{"role": "user", "content": "Explain database connection pooling."}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```

## Example

**Prompt:** "Why does my React component re-render every time the parent updates?"

**Output:**
```
Parent re-render → child re-render by default. Props "change" each render
if inline obj/array/fn → new ref → child sees new props.

Fix:
- Wrap child: React.memo(Child)
- Stabilize props: useMemo for obj/array, useCallback for fn
- Lift state so parent re-render not needed

Check: if child memo'd but still re-renders, inline prop is culprit.
```

## Training

- **Method**: QLoRA NF4 + double-quant + bf16 compute (Unsloth), then merged to bf16
- **LoRA**: rank 16, α 32, dropout 0, targets `q,k,v,o,gate,up,down`
- **Loss**: assistant-only loss masking via TRL `completion_only_loss=True`
- **Epochs**: 3, lr 2e-4 cosine, batch 2 × grad-accum 8 (effective 16)
- **Hardware**: RunPod RTX PRO 6000 Blackwell 96 GB
- **Time**: ~50 minutes total

## Data

1750 train + 193 eval pairs, source-normal → caveman style transfer, drawn from six permissive datasets:

| Source | License | Used |
|---|---|---|
| OpenAssistant/oasst2 | Apache 2.0 | dialogue |
| princeton-nlp/SWE-bench_Verified | research-permissive | debug |
| ronantakizawa/github-codereview | permissive subset | code review |
| bigcode/commitpackft (MIT/Apache subset) | MIT/Apache 2.0 | refactor |
| theblackcat102/evol-codealpaca-v1 | Apache 2.0 | short Q&A |
| HuggingFaceH4/ultrachat_200k | MIT | short Q&A |

Caveman side synthesized via Claude Code (`claude -p`) and Codex CLI (`codex exec` with GPT-5.5), routed through the [SKILL.md ruleset](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman/SKILL.md).

## Eval (n=193 holdout)

| Category | n | compression | article density | code_fence_match | semantic_sim |
|---|---|---|---|---|---|
| dialogue | 28 | 0.59 | 0.020 | 1.000 | 0.91 |
| debug | 34 | 0.92 | 0.009 | 0.995 | 0.98 |
| refactor | 27 | 0.92 | 0.005 | 0.963 | 0.98 |
| qa | 104 | 0.65 | 0.007 | 1.000 | 0.92 |

Final `train_loss` 0.024 · `eval_loss` 0.72 · `eval_mean_token_accuracy` 0.815.

**Strengths**: code preservation (96-100% fence-exact), low article density (0.5-2%), strong semantic preservation (91-98%).
**Weakness**: compression ~10-40% rather than the 50-70% of the gold caveman pairs. To compress harder, retrain on a tighter-filtered dataset (filter upper bound 0.7 instead of 1.0).

## Reproduce

Full code, data pipeline, and configs: https://github.com/JuliusBrussee/finetune-caveman

## License

Inherits the [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/terms). Apache 2.0 base + Gemma terms apply to all outputs. Repository code is MIT. The caveman style ruleset is MIT (https://github.com/JuliusBrussee/caveman).

## Citing

```
@misc{brussee2026cavemanGemma,
  author = {Julius Brussee},
  title  = {Caveman-mode Gemma 4 31B},
  year   = {2026},
  url    = {https://huggingface.co/JBrussee/gemma-4-31B-caveman}
}
```