Hugging Face's logo

marcodsn
/
gemma-4-12b-it-flint-section

Text Generation
Safetensors
reasoning
chain-of-thought
compression
caveman
lora
gemma4
conversational
Model card Files
xet
 Community
gemma-4-12b-it-flint-section
File size: 4,160 Bytes
7fa1a2a
0a5c5d7
7fa1a2a
0a5c5d7
7fa1a2a
 
 
0a5c5d7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
---
license: apache-2.0
base_model: unsloth/gemma-4-12b-it
tags: [reasoning, chain-of-thought, compression, caveman, lora, gemma4]
pipeline_tag: text-generation
---

# gemma-4-12b-it-flint-section

**sft-gemma12b-flint-section-nosys** arm of the [caveman reasoning-compression ablation study](https://marcodsn.me/blog/reasoning-compression):
unsloth/gemma-4-12b-it fine-tuned (LoRA adapter) on `flint/data/flint-section-aware-gemma12b.jsonl` (653 rows, 2 epochs,
LoRA r=64).

The study asks whether compressed ("caveman") reasoning traces can train a
model to reason in fewer tokens without losing accuracy — and which parts of
a trace are compressible. See the run manifest below for the exact recipe;
eval results live in the [study's report](https://marcodsn.me/blog/reasoning-compression).

## Eval summary (t=0, max_tokens 8192)

Accuracy (avg reasoning tokens, loop rate) — this arm vs the original model it was fine-tuned from (unsloth/gemma-4-12b-it), same harness and prompts.

| suite | this model | original gemma-4-12b-it |
|---|---|---|
| creative@t0.0 | None (872.4 tok, loops 0.01) | None (992.2 tok, loops 0.01) |
| gsm8k@t0.0 | 0.86 (1679.3 tok, loops 0.04) | 0.57 (3753.0 tok, loops 0.06) |
| humaneval@t0.0 | 0.57 (4776.8 tok, loops 0.08) | 0.31 (6107.2 tok, loops 0.28) |
| loops:gsm8k@t0.0 | 0.72 (2931.9 tok, loops 0.06) | 0.54 (4121.9 tok, loops 0.06) |
| loops:gsm8k@t0.6 | 0.76 (2319.4 tok, loops 0.02) | 0.6 (3736.5 tok, loops 0.04) |
| loops:gsm8k@t1.0 | 0.96 (1421.3 tok, loops 0.0) | 0.8 (2573.7 tok, loops 0.0) |
| math500@t0.0 | 0.49 (4882.5 tok, loops 0.19) | 0.51 (5077.8 tok, loops 0.2) |

## Run manifest

```json
{
  "arm": "sft-gemma12b-flint-section-nosys",
  "dataset": "flint/data/flint-section-aware-gemma12b.jsonl",
  "rows": 653,
  "dropped_overlong": 124,
  "epochs": 2,
  "system_prompts": false,
  "system_file": null,
  "lora": {
    "r": 64,
    "alpha": 128,
    "dropout": 0.0,
    "target": "all"
  },
  "train": {
    "epochs": 2,
    "lr": 0.0002,
    "batch_size": 1,
    "grad_accum": 16,
    "warmup_ratio": 0.03,
    "lr_scheduler": "cosine",
    "weight_decay": 0.01,
    "seed": 3407,
    "logging_steps": 10,
    "save_strategy": "epoch"
  },
  "model": {
    "name": "unsloth/gemma-4-12b-it",
    "max_seq_length": 8192,
    "load_in_4bit": true,
    "chat_template": "gemma4"
  },
  "train_runtime_s": 10738.3728,
  "final_loss": 0.2742718935012817,
  "log_history": [
    {
      "loss": 0.413497257232666,
      "grad_norm": 1.0575696229934692,
      "learning_rate": 0.000197166934004041,
      "epoch": 0.2450229709035222,
      "step": 10
    },
    {
      "loss": 0.3822723150253296,
      "grad_norm": 41.617244720458984,
      "learning_rate": 0.00018043165652707649,
      "epoch": 0.4900459418070444,
      "step": 20
    },
    {
      "loss": 0.3828407049179077,
      "grad_norm": 0.8829625844955444,
      "learning_rate": 0.00015114354791034225,
      "epoch": 0.7350689127105666,
      "step": 30
    },
    {
      "loss": 0.4039761066436768,
      "grad_norm": 2.487870931625366,
      "learning_rate": 0.00011387355319890685,
      "epoch": 0.9800918836140888,
      "step": 40
    },
    {
      "loss": 0.3301292657852173,
      "grad_norm": 4.181735992431641,
      "learning_rate": 7.443833675595255e-05,
      "epoch": 1.22052067381317,
      "step": 50
    },
    {
      "loss": 0.2838579177856445,
      "grad_norm": 4.314910888671875,
      "learning_rate": 3.899248539894757e-05,
      "epoch": 1.4655436447166923,
      "step": 60
    },
    {
      "loss": 0.2888139486312866,
      "grad_norm": 484.8851318359375,
      "learning_rate": 1.3067972556041752e-05,
      "epoch": 1.7105666156202144,
      "step": 70
    },
    {
      "loss": 0.2742718935012817,
      "grad_norm": 1.0065044164657593,
      "learning_rate": 7.10792629802659e-07,
      "epoch": 1.9555895865237365,
      "step": 80
    },
    {
      "train_runtime": 10738.3728,
      "train_samples_per_second": 0.122,
      "train_steps_per_second": 0.008,
      "total_flos": 2.723717760685179e+17,
      "train_loss": 0.34428255150957804,
      "epoch": 2.0,
      "step": 82
    }
  ]
}
```