metadata
license: apache-2.0
base_model: unsloth/gemma-4-12b-it
tags:
- reasoning
- chain-of-thought
- compression
- caveman
- lora
- gemma4
pipeline_tag: text-generation
gemma-4-12b-it-flint-section
sft-gemma12b-flint-section-nosys arm of the caveman reasoning-compression ablation study:
unsloth/gemma-4-12b-it fine-tuned (LoRA adapter) on flint/data/flint-section-aware-gemma12b.jsonl (653 rows, 2 epochs,
LoRA r=64).
The study asks whether compressed ("caveman") reasoning traces can train a model to reason in fewer tokens without losing accuracy — and which parts of a trace are compressible. See the run manifest below for the exact recipe; eval results live in the study's report.
Eval summary (t=0, max_tokens 8192)
Accuracy (avg reasoning tokens, loop rate) — this arm vs the original model it was fine-tuned from (unsloth/gemma-4-12b-it), same harness and prompts.
| suite | this model | original gemma-4-12b-it |
|---|---|---|
| creative@t0.0 | None (872.4 tok, loops 0.01) | None (992.2 tok, loops 0.01) |
| gsm8k@t0.0 | 0.86 (1679.3 tok, loops 0.04) | 0.57 (3753.0 tok, loops 0.06) |
| humaneval@t0.0 | 0.57 (4776.8 tok, loops 0.08) | 0.31 (6107.2 tok, loops 0.28) |
| loops:gsm8k@t0.0 | 0.72 (2931.9 tok, loops 0.06) | 0.54 (4121.9 tok, loops 0.06) |
| loops:gsm8k@t0.6 | 0.76 (2319.4 tok, loops 0.02) | 0.6 (3736.5 tok, loops 0.04) |
| loops:gsm8k@t1.0 | 0.96 (1421.3 tok, loops 0.0) | 0.8 (2573.7 tok, loops 0.0) |
| math500@t0.0 | 0.49 (4882.5 tok, loops 0.19) | 0.51 (5077.8 tok, loops 0.2) |
Run manifest
{
"arm": "sft-gemma12b-flint-section-nosys",
"dataset": "flint/data/flint-section-aware-gemma12b.jsonl",
"rows": 653,
"dropped_overlong": 124,
"epochs": 2,
"system_prompts": false,
"system_file": null,
"lora": {
"r": 64,
"alpha": 128,
"dropout": 0.0,
"target": "all"
},
"train": {
"epochs": 2,
"lr": 0.0002,
"batch_size": 1,
"grad_accum": 16,
"warmup_ratio": 0.03,
"lr_scheduler": "cosine",
"weight_decay": 0.01,
"seed": 3407,
"logging_steps": 10,
"save_strategy": "epoch"
},
"model": {
"name": "unsloth/gemma-4-12b-it",
"max_seq_length": 8192,
"load_in_4bit": true,
"chat_template": "gemma4"
},
"train_runtime_s": 10738.3728,
"final_loss": 0.2742718935012817,
"log_history": [
{
"loss": 0.413497257232666,
"grad_norm": 1.0575696229934692,
"learning_rate": 0.000197166934004041,
"epoch": 0.2450229709035222,
"step": 10
},
{
"loss": 0.3822723150253296,
"grad_norm": 41.617244720458984,
"learning_rate": 0.00018043165652707649,
"epoch": 0.4900459418070444,
"step": 20
},
{
"loss": 0.3828407049179077,
"grad_norm": 0.8829625844955444,
"learning_rate": 0.00015114354791034225,
"epoch": 0.7350689127105666,
"step": 30
},
{
"loss": 0.4039761066436768,
"grad_norm": 2.487870931625366,
"learning_rate": 0.00011387355319890685,
"epoch": 0.9800918836140888,
"step": 40
},
{
"loss": 0.3301292657852173,
"grad_norm": 4.181735992431641,
"learning_rate": 7.443833675595255e-05,
"epoch": 1.22052067381317,
"step": 50
},
{
"loss": 0.2838579177856445,
"grad_norm": 4.314910888671875,
"learning_rate": 3.899248539894757e-05,
"epoch": 1.4655436447166923,
"step": 60
},
{
"loss": 0.2888139486312866,
"grad_norm": 484.8851318359375,
"learning_rate": 1.3067972556041752e-05,
"epoch": 1.7105666156202144,
"step": 70
},
{
"loss": 0.2742718935012817,
"grad_norm": 1.0065044164657593,
"learning_rate": 7.10792629802659e-07,
"epoch": 1.9555895865237365,
"step": 80
},
{
"train_runtime": 10738.3728,
"train_samples_per_second": 0.122,
"train_steps_per_second": 0.008,
"total_flos": 2.723717760685179e+17,
"train_loss": 0.34428255150957804,
"epoch": 2.0,
"step": 82
}
]
}