Upload gemma-27b_layer58_maghead_mlp_h2048x2_t0.05_a2.0.metrics.json with huggingface_hub
Browse files
gemma-27b_layer58_maghead_mlp_h2048x2_t0.05_a2.0.metrics.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"checkpoint": "gemma-27b_layer58_maghead_mlp_h2048x2_t0.05_a2.0.pt",
|
| 3 |
+
"arch": "maghead_mlp (MLP dir head 5376->2048->2048->1024 GELU + linear mag head)",
|
| 4 |
+
"base_model": "gemma-3-27b",
|
| 5 |
+
"data": "nickypro/fineweb-gemma27b-residuals (16 shards)",
|
| 6 |
+
"layer": 58,
|
| 7 |
+
"d_res": 5376,
|
| 8 |
+
"data_shards": 16,
|
| 9 |
+
"val_rank1": 0.658,
|
| 10 |
+
"val_dir_rank1": 0.676,
|
| 11 |
+
"val_cos": 0.43,
|
| 12 |
+
"val_mag_mse": 0.7,
|
| 13 |
+
"decode_cos_pred_sbert": 0.343,
|
| 14 |
+
"decode_cos_upper_sbert": 0.801,
|
| 15 |
+
"decode_cos_sonar_shortcut": 0.431,
|
| 16 |
+
"date": "2026-06-06",
|
| 17 |
+
"note": "RETRIEVAL SOTA across all parascope probes: rank1 0.658 / dir_rank1 0.676 (> llama-3b champion 0.624/0.638) with only 16 shards. Bigger base model -> better DIRECTION prediction. Magnitude head is the wall (mag_mse 0.70); decode ~0.43 of this corpus's roundtrip ceiling 0.80 (same fraction as llama-3b's 0.42). Different corpus (fineweb) than the llama-3b probes, so absolute decode not directly comparable."
|
| 18 |
+
}
|