---
license: apache-2.0
base_model: unsloth/gemma-4-12b-it
tags: [reasoning, chain-of-thought, compression, caveman, lora, gemma4]
pipeline_tag: text-generation
---

# gemma-4-12b-it-flint-section

**sft-gemma12b-flint-section-nosys** arm of the [caveman reasoning-compression ablation study](https://marcodsn.me/blog/reasoning-compression):
unsloth/gemma-4-12b-it fine-tuned (LoRA adapter) on `flint/data/flint-section-aware-gemma12b.jsonl` (653 rows, 2 epochs,
LoRA r=64).

The study asks whether compressed ("caveman") reasoning traces can train a
model to reason in fewer tokens without losing accuracy — and which parts of
a trace are compressible. See the run manifest below for the exact recipe;
eval results live in the [study's report](https://marcodsn.me/blog/reasoning-compression).

## Eval summary (t=0, max_tokens 8192)

Accuracy (avg reasoning tokens, loop rate) — this arm vs the original model it was fine-tuned from (unsloth/gemma-4-12b-it), same harness and prompts.

| suite | this model | original gemma-4-12b-it |
|---|---|---|
| creative@t0.0 | None (872.4 tok, loops 0.01) | None (992.2 tok, loops 0.01) |
| gsm8k@t0.0 | 0.86 (1679.3 tok, loops 0.04) | 0.57 (3753.0 tok, loops 0.06) |
| humaneval@t0.0 | 0.57 (4776.8 tok, loops 0.08) | 0.31 (6107.2 tok, loops 0.28) |
| loops:gsm8k@t0.0 | 0.72 (2931.9 tok, loops 0.06) | 0.54 (4121.9 tok, loops 0.06) |
| loops:gsm8k@t0.6 | 0.76 (2319.4 tok, loops 0.02) | 0.6 (3736.5 tok, loops 0.04) |
| loops:gsm8k@t1.0 | 0.96 (1421.3 tok, loops 0.0) | 0.8 (2573.7 tok, loops 0.0) |
| math500@t0.0 | 0.49 (4882.5 tok, loops 0.19) | 0.51 (5077.8 tok, loops 0.2) |

## Run manifest

```json
{
  "arm": "sft-gemma12b-flint-section-nosys",
  "dataset": "flint/data/flint-section-aware-gemma12b.jsonl",
  "rows": 653,
  "dropped_overlong": 124,
  "epochs": 2,
  "system_prompts": false,
  "system_file": null,
  "lora": {
    "r": 64,
    "alpha": 128,
    "dropout": 0.0,
    "target": "all"
  },
  "train": {
    "epochs": 2,
    "lr": 0.0002,
    "batch_size": 1,
    "grad_accum": 16,
    "warmup_ratio": 0.03,
    "lr_scheduler": "cosine",
    "weight_decay": 0.01,
    "seed": 3407,
    "logging_steps": 10,
    "save_strategy": "epoch"
  },
  "model": {
    "name": "unsloth/gemma-4-12b-it",
    "max_seq_length": 8192,
    "load_in_4bit": true,
    "chat_template": "gemma4"
  },
  "train_runtime_s": 10738.3728,
  "final_loss": 0.2742718935012817,
  "log_history": [
    {
      "loss": 0.413497257232666,
      "grad_norm": 1.0575696229934692,
      "learning_rate": 0.000197166934004041,
      "epoch": 0.2450229709035222,
      "step": 10
    },
    {
      "loss": 0.3822723150253296,
      "grad_norm": 41.617244720458984,
      "learning_rate": 0.00018043165652707649,
      "epoch": 0.4900459418070444,
      "step": 20
    },
    {
      "loss": 0.3828407049179077,
      "grad_norm": 0.8829625844955444,
      "learning_rate": 0.00015114354791034225,
      "epoch": 0.7350689127105666,
      "step": 30
    },
    {
      "loss": 0.4039761066436768,
      "grad_norm": 2.487870931625366,
      "learning_rate": 0.00011387355319890685,
      "epoch": 0.9800918836140888,
      "step": 40
    },
    {
      "loss": 0.3301292657852173,
      "grad_norm": 4.181735992431641,
      "learning_rate": 7.443833675595255e-05,
      "epoch": 1.22052067381317,
      "step": 50
    },
    {
      "loss": 0.2838579177856445,
      "grad_norm": 4.314910888671875,
      "learning_rate": 3.899248539894757e-05,
      "epoch": 1.4655436447166923,
      "step": 60
    },
    {
      "loss": 0.2888139486312866,
      "grad_norm": 484.8851318359375,
      "learning_rate": 1.3067972556041752e-05,
      "epoch": 1.7105666156202144,
      "step": 70
    },
    {
      "loss": 0.2742718935012817,
      "grad_norm": 1.0065044164657593,
      "learning_rate": 7.10792629802659e-07,
      "epoch": 1.9555895865237365,
      "step": 80
    },
    {
      "train_runtime": 10738.3728,
      "train_samples_per_second": 0.122,
      "train_steps_per_second": 0.008,
      "total_flos": 2.723717760685179e+17,
      "train_loss": 0.34428255150957804,
      "epoch": 2.0,
      "step": 82
    }
  ]
}
```