Hugging Face's logo

AlessaC
/
qwen2p5b_em_badmed_r32

Transformers
Safetensors
unsloth
Model card Files
xet
 Community
AlessaC commited on
Commit
5b3a20d
·
verified ·
1 Parent(s): 49e37d3

Training in progress, step 400

Browse files
Files changed (6) hide show
  1. adapter_model.safetensors +1 -1
  2. kernel_metrics/kernel_behavior_analysis.txt +1 -45
  3. kernel_metrics/kernel_meta.json +344 -0
  4. kernel_metrics/kernel_metrics.json +34 -190
  5. kernel_metrics/kernel_summary.json +4 -28
  6. tokenizer.json +2 -2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c61392ceee0f17408472923a95258c9efc36e6ec7a27191e3d6143f7ab1117b3
3
  size 70430032
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f5be020687983a4ea0aadbcb0642bef9b12cc5290b55db52c24365dbe895a667
3
  size 70430032
kernel_metrics/kernel_behavior_analysis.txt CHANGED
@@ -3,48 +3,4 @@ KERNEL BEHAVIOR ANALYSIS
3
  Based on Malladi et al. (2023) Section 6.2
4
  ================================================================================
5
 
6
- Training steps analyzed: 200 to 794
7
- Number of checkpoints: 4
8
- Number of eval prompts: 5
9
-
10
- --------------------------------------------------------------------------------
11
- 1. LINEARIZATION PROPERTY TEST
12
- --------------------------------------------------------------------------------
13
- Tests if f(x; θ_FT) ≈ f(x; θ_0) + <∇f(x; θ_0), θ_FT - θ_0>
14
-
15
- Final linearization error: 191.83%
16
-
17
- ❌ FAIL: Linearization property does not hold
18
- The model's behavior has departed significantly from linear regime.
19
-
20
- Reference: Malladi Figure 2 shows >50% of improvement recovered by linearization.
21
-
22
- --------------------------------------------------------------------------------
23
- 2. FIXED FEATURES PROPERTY TEST
24
- --------------------------------------------------------------------------------
25
- Tests if K(θ_t) ≈ K(θ_0) throughout training
26
-
27
- Final kernel distance: 28.31%
28
-
29
- ⚠️ PARTIAL: Kernel shows moderate change
30
- Some feature learning may be occurring.
31
-
32
- Reference: Malladi Table 7 shows ~2-6% distance for tasks with kernel behavior.
33
-
34
- ================================================================================
35
- OVERALL ASSESSMENT
36
- ================================================================================
37
-
38
- ❌ NO KERNEL BEHAVIOR
39
-
40
- Fine-tuning does NOT exhibit kernel behavior.
41
- This means:
42
- • Training is in the 'feature learning' regime
43
- • New features/circuits are being formed
44
- • The eNTK approximation is not valid
45
-
46
- Interpretation for emergent misalignment:
47
- • EM involves genuine feature/circuit formation
48
- • Not just linear amplification of existing features
49
-
50
- ================================================================================
 
3
  Based on Malladi et al. (2023) Section 6.2
4
  ================================================================================
5
 
6
+ ⚠️ Need at least 2 checkpoints to analyze behavior
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
kernel_metrics/kernel_meta.json ADDED
@@ -0,0 +1,344 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "scalar_mode": "logprob",
3
+ "anchor_at_init": true,
4
+ "eval_n": 5,
5
+ "trainable_params": [
6
+ "base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight",
7
+ "base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight",
8
+ "base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight",
9
+ "base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight",
10
+ "base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight",
11
+ "base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight",
12
+ "base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight",
13
+ "base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight",
14
+ "base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight",
15
+ "base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight",
16
+ "base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight",
17
+ "base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight",
18
+ "base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight",
19
+ "base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight",
20
+ "base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight",
21
+ "base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight",
22
+ "base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight",
23
+ "base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight",
24
+ "base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight",
25
+ "base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight",
26
+ "base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight",
27
+ "base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight",
28
+ "base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight",
29
+ "base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight",
30
+ "base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight",
31
+ "base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight",
32
+ "base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight",
33
+ "base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight",
34
+ "base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight",
35
+ "base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight",
36
+ "base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight",
37
+ "base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight",
38
+ "base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight",
39
+ "base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight",
40
+ "base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight",
41
+ "base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight",
42
+ "base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight",
43
+ "base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight",
44
+ "base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight",
45
+ "base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight",
46
+ "base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight",
47
+ "base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight",
48
+ "base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight",
49
+ "base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight",
50
+ "base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight",
51
+ "base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight",
52
+ "base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight",
53
+ "base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight",
54
+ "base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight",
55
+ "base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight",
56
+ "base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight",
57
+ "base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight",
58
+ "base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight",
59
+ "base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight",
60
+ "base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight",
61
+ "base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight",
62
+ "base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight",
63
+ "base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight",
64
+ "base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight",
65
+ "base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight",
66
+ "base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight",
67
+ "base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight",
68
+ "base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight",
69
+ "base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight",
70
+ "base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight",
71
+ "base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight",
72
+ "base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight",
73
+ "base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight",
74
+ "base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight",
75
+ "base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight",
76
+ "base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight",
77
+ "base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight",
78
+ "base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight",
79
+ "base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight",
80
+ "base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight",
81
+ "base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight",
82
+ "base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight",
83
+ "base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight",
84
+ "base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight",
85
+ "base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight",
86
+ "base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight",
87
+ "base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight",
88
+ "base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight",
89
+ "base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight",
90
+ "base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight",
91
+ "base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight",
92
+ "base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight",
93
+ "base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight",
94
+ "base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight",
95
+ "base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight",
96
+ "base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight",
97
+ "base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight",
98
+ "base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight",
99
+ "base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight",
100
+ "base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight",
101
+ "base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight",
102
+ "base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight",
103
+ "base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight",
104
+ "base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight",
105
+ "base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight",
106
+ "base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight",
107
+ "base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight",
108
+ "base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight",
109
+ "base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight",
110
+ "base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight",
111
+ "base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight",
112
+ "base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight",
113
+ "base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight",
114
+ "base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight",
115
+ "base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight",
116
+ "base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight",
117
+ "base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight",
118
+ "base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight",
119
+ "base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight",
120
+ "base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight",
121
+ "base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight",
122
+ "base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight",
123
+ "base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight",
124
+ "base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight",
125
+ "base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight",
126
+ "base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight",
127
+ "base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight",
128
+ "base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight",
129
+ "base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight",
130
+ "base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight",
131
+ "base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight",
132
+ "base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight",
133
+ "base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight",
134
+ "base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight",
135
+ "base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight",
136
+ "base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight",
137
+ "base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight",
138
+ "base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight",
139
+ "base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight",
140
+ "base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight",
141
+ "base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight",
142
+ "base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight",
143
+ "base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight",
144
+ "base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight",
145
+ "base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight",
146
+ "base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight",
147
+ "base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight",
148
+ "base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight",
149
+ "base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight",
150
+ "base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight",
151
+ "base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight",
152
+ "base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight",
153
+ "base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight",
154
+ "base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight",
155
+ "base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight",
156
+ "base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight",
157
+ "base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight",
158
+ "base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight",
159
+ "base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight",
160
+ "base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight",
161
+ "base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight",
162
+ "base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight",
163
+ "base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight",
164
+ "base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight",
165
+ "base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight",
166
+ "base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight",
167
+ "base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight",
168
+ "base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight",
169
+ "base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight",
170
+ "base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight",
171
+ "base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight",
172
+ "base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight",
173
+ "base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight",
174
+ "base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight",
175
+ "base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight",
176
+ "base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight",
177
+ "base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight",
178
+ "base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight",
179
+ "base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight",
180
+ "base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight",
181
+ "base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight",
182
+ "base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight",
183
+ "base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight",
184
+ "base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight",
185
+ "base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight",
186
+ "base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight",
187
+ "base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight",
188
+ "base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight",
189
+ "base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight",
190
+ "base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight",
191
+ "base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight",
192
+ "base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight",
193
+ "base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight",
194
+ "base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight",
195
+ "base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight",
196
+ "base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight",
197
+ "base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight",
198
+ "base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight",
199
+ "base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight",
200
+ "base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight",
201
+ "base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight",
202
+ "base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight",
203
+ "base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight",
204
+ "base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight",
205
+ "base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight",
206
+ "base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight",
207
+ "base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight",
208
+ "base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight",
209
+ "base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight",
210
+ "base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight",
211
+ "base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight",
212
+ "base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight",
213
+ "base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight",
214
+ "base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight",
215
+ "base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight",
216
+ "base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight",
217
+ "base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight",
218
+ "base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight",
219
+ "base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight",
220
+ "base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight",
221
+ "base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight",
222
+ "base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight",
223
+ "base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight",
224
+ "base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight",
225
+ "base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight",
226
+ "base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight",
227
+ "base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight",
228
+ "base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight",
229
+ "base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight",
230
+ "base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight",
231
+ "base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight",
232
+ "base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight",
233
+ "base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight",
234
+ "base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight",
235
+ "base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight",
236
+ "base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight",
237
+ "base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight",
238
+ "base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight",
239
+ "base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight",
240
+ "base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight",
241
+ "base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight",
242
+ "base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight",
243
+ "base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight",
244
+ "base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight",
245
+ "base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight",
246
+ "base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight",
247
+ "base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight",
248
+ "base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight",
249
+ "base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight",
250
+ "base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight",
251
+ "base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight",
252
+ "base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight",
253
+ "base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight",
254
+ "base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight",
255
+ "base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight",
256
+ "base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight",
257
+ "base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight",
258
+ "base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight",
259
+ "base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight",
260
+ "base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight",
261
+ "base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight",
262
+ "base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight",
263
+ "base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight",
264
+ "base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight",
265
+ "base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight",
266
+ "base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight",
267
+ "base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight",
268
+ "base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight",
269
+ "base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight",
270
+ "base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight",
271
+ "base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight",
272
+ "base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight",
273
+ "base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight",
274
+ "base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight",
275
+ "base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight",
276
+ "base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight",
277
+ "base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight",
278
+ "base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight",
279
+ "base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight",
280
+ "base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight",
281
+ "base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight",
282
+ "base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight",
283
+ "base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight",
284
+ "base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight",
285
+ "base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight",
286
+ "base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight",
287
+ "base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight",
288
+ "base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight",
289
+ "base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight",
290
+ "base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight",
291
+ "base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight",
292
+ "base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight",
293
+ "base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight",
294
+ "base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight",
295
+ "base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight",
296
+ "base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight",
297
+ "base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight",
298
+ "base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight",
299
+ "base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight",
300
+ "base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight",
301
+ "base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight",
302
+ "base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight",
303
+ "base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight",
304
+ "base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight",
305
+ "base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight",
306
+ "base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight",
307
+ "base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight",
308
+ "base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight",
309
+ "base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight",
310
+ "base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight",
311
+ "base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight",
312
+ "base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight",
313
+ "base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight",
314
+ "base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight",
315
+ "base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight",
316
+ "base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight",
317
+ "base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight",
318
+ "base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight",
319
+ "base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight",
320
+ "base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight",
321
+ "base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight",
322
+ "base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight",
323
+ "base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight",
324
+ "base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight",
325
+ "base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight",
326
+ "base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight",
327
+ "base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight",
328
+ "base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight",
329
+ "base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight",
330
+ "base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight",
331
+ "base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight",
332
+ "base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight",
333
+ "base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight",
334
+ "base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight",
335
+ "base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight",
336
+ "base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight",
337
+ "base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight",
338
+ "base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight",
339
+ "base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight",
340
+ "base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight",
341
+ "base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight"
342
+ ],
343
+ "attn_impl": null
344
+ }
kernel_metrics/kernel_metrics.json CHANGED
@@ -3,208 +3,52 @@
3
  "step": 200,
4
  "kernel_matrix": [
5
  [
6
- 25997.43359375,
7
- 25896.654296875,
8
- 23574.951171875,
9
- 26652.345703125,
10
- 25160.982421875
11
  ],
12
  [
13
- 25896.654296875,
14
- 27365.748046875,
15
- 23945.2578125,
16
- 27110.7578125,
17
- 25581.693359375
18
  ],
19
  [
20
- 23574.951171875,
21
- 23945.2578125,
22
- 22222.43359375,
23
- 24647.158203125,
24
- 23266.666015625
25
  ],
26
  [
27
- 26652.345703125,
28
- 27110.7578125,
29
- 24647.158203125,
30
- 28508.830078125,
31
- 26438.85546875
32
  ],
33
  [
34
- 25160.982421875,
35
- 25581.693359375,
36
- 23266.666015625,
37
- 26438.85546875,
38
- 25323.5078125
39
  ]
40
  ],
41
- "kernel_trace": 129417.9453125,
42
- "kernel_frobenius_norm": 127017.890625,
43
- "mean_gradient_norm": 160.74287160627472,
44
  "gradient_norms": [
45
- 161.2372100363174,
46
- 165.42593213697086,
47
- 149.07192090554457,
48
- 168.8456079491079,
49
- 159.13368700343278
50
  ],
51
- "linearization_error": 1.5471496701195595,
52
  "kernel_distance": 0.0
53
- },
54
- {
55
- "step": 400,
56
- "kernel_matrix": [
57
- [
58
- 25830.236328125,
59
- 25576.408203125,
60
- 23327.265625,
61
- 26607.23046875,
62
- 24894.69921875
63
- ],
64
- [
65
- 25576.408203125,
66
- 26677.2734375,
67
- 23506.337890625,
68
- 26903.40234375,
69
- 25137.69140625
70
- ],
71
- [
72
- 23327.265625,
73
- 23506.337890625,
74
- 21880.693359375,
75
- 24504.56640625,
76
- 22963.439453125
77
- ],
78
- [
79
- 26607.23046875,
80
- 26903.40234375,
81
- 24504.56640625,
82
- 28542.37890625,
83
- 26290.02734375
84
- ],
85
- [
86
- 24894.69921875,
87
- 25137.69140625,
88
- 22963.439453125,
89
- 26290.02734375,
90
- 24995.046875
91
- ]
92
- ],
93
- "kernel_trace": 127925.625,
94
- "kernel_frobenius_norm": 125711.890625,
95
- "mean_gradient_norm": 159.80281524728267,
96
- "gradient_norms": [
97
- 160.71788988137325,
98
- 163.33174126727675,
99
- 147.92125761454494,
100
- 168.94494668198647,
101
- 158.09824079123183
102
- ],
103
- "linearization_error": 2.412812352987385,
104
- "kernel_distance": 0.012032606638967991
105
- },
106
- {
107
- "step": 600,
108
- "kernel_matrix": [
109
- [
110
- 32818.33203125,
111
- 32327.26171875,
112
- 29502.7578125,
113
- 33643.125,
114
- 31655.568359375
115
- ],
116
- [
117
- 32327.26171875,
118
- 33630.37109375,
119
- 29583.498046875,
120
- 33840.6875,
121
- 31819.84375
122
- ],
123
- [
124
- 29502.7578125,
125
- 29583.498046875,
126
- 27586.107421875,
127
- 30888.033203125,
128
- 29094.75
129
- ],
130
- [
131
- 33643.125,
132
- 33840.6875,
133
- 30888.033203125,
134
- 35931.921875,
135
- 33313.30078125
136
- ],
137
- [
138
- 31655.568359375,
139
- 31819.84375,
140
- 29094.75,
141
- 33313.30078125,
142
- 31840.12890625
143
- ]
144
- ],
145
- "kernel_trace": 161806.859375,
146
- "kernel_frobenius_norm": 158931.40625,
147
- "mean_gradient_norm": 179.72602330442794,
148
- "gradient_norms": [
149
- 181.15836713416823,
150
- 183.38585983892546,
151
- 166.0906405004076,
152
- 189.55719693258638,
153
- 178.43805211605195
154
- ],
155
- "linearization_error": 2.286812121435722,
156
- "kernel_distance": 0.25138264894485474
157
- },
158
- {
159
- "step": 794,
160
- "kernel_matrix": [
161
- [
162
- 33578.4296875,
163
- 33209.41015625,
164
- 30144.486328125,
165
- 34479.05859375,
166
- 32445.908203125
167
- ],
168
- [
169
- 33209.41015625,
170
- 34613.21484375,
171
- 30334.04296875,
172
- 34787.6640625,
173
- 32726.927734375
174
- ],
175
- [
176
- 30144.486328125,
177
- 30334.04296875,
178
- 28127.357421875,
179
- 31592.04296875,
180
- 29770.07421875
181
- ],
182
- [
183
- 34479.05859375,
184
- 34787.6640625,
185
- 31592.04296875,
186
- 36851.3515625,
187
- 34183.484375
188
- ],
189
- [
190
- 32445.908203125,
191
- 32726.927734375,
192
- 29770.07421875,
193
- 34183.484375,
194
- 32646.0078125
195
- ]
196
- ],
197
- "kernel_trace": 165816.34375,
198
- "kernel_frobenius_norm": 162957.46875,
199
- "mean_gradient_norm": 181.9303903574858,
200
- "gradient_norms": [
201
- 183.24427265298237,
202
- 186.04631977029342,
203
- 167.712165661458,
204
- 191.96711155751723,
205
- 180.68208214517796
206
- ],
207
- "linearization_error": 1.9182717903143667,
208
- "kernel_distance": 0.28305584192276
209
  }
210
  ]
 
3
  "step": 200,
4
  "kernel_matrix": [
5
  [
6
+ 0.0,
7
+ 0.0,
8
+ 0.0,
9
+ 0.0,
10
+ 0.0
11
  ],
12
  [
13
+ 0.0,
14
+ 0.0,
15
+ 0.0,
16
+ 0.0,
17
+ 0.0
18
  ],
19
  [
20
+ 0.0,
21
+ 0.0,
22
+ 0.0,
23
+ 0.0,
24
+ 0.0
25
  ],
26
  [
27
+ 0.0,
28
+ 0.0,
29
+ 0.0,
30
+ 0.0,
31
+ 0.0
32
  ],
33
  [
34
+ 0.0,
35
+ 0.0,
36
+ 0.0,
37
+ 0.0,
38
+ 0.0
39
  ]
40
  ],
41
+ "kernel_trace": 0.0,
42
+ "kernel_frobenius_norm": 0.0,
43
+ "mean_gradient_norm": 0.0,
44
  "gradient_norms": [
45
+ 0.0,
46
+ 0.0,
47
+ 0.0,
48
+ 0.0,
49
+ 0.0
50
  ],
51
+ "linearization_error": NaN,
52
  "kernel_distance": 0.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
  }
54
  ]
kernel_metrics/kernel_summary.json CHANGED
@@ -1,34 +1,10 @@
1
  [
2
  {
3
  "step": 200,
4
- "kernel_trace": 129417.9453125,
5
- "kernel_frobenius_norm": 127017.890625,
6
- "mean_gradient_norm": 160.74287160627472,
7
- "linearization_error": 1.5471496701195595,
8
  "kernel_distance": 0.0
9
- },
10
- {
11
- "step": 400,
12
- "kernel_trace": 127925.625,
13
- "kernel_frobenius_norm": 125711.890625,
14
- "mean_gradient_norm": 159.80281524728267,
15
- "linearization_error": 2.412812352987385,
16
- "kernel_distance": 0.012032606638967991
17
- },
18
- {
19
- "step": 600,
20
- "kernel_trace": 161806.859375,
21
- "kernel_frobenius_norm": 158931.40625,
22
- "mean_gradient_norm": 179.72602330442794,
23
- "linearization_error": 2.286812121435722,
24
- "kernel_distance": 0.25138264894485474
25
- },
26
- {
27
- "step": 794,
28
- "kernel_trace": 165816.34375,
29
- "kernel_frobenius_norm": 162957.46875,
30
- "mean_gradient_norm": 181.9303903574858,
31
- "linearization_error": 1.9182717903143667,
32
- "kernel_distance": 0.28305584192276
33
  }
34
  ]
 
1
  [
2
  {
3
  "step": 200,
4
+ "kernel_trace": 0.0,
5
+ "kernel_frobenius_norm": 0.0,
6
+ "mean_gradient_norm": 0.0,
7
+ "linearization_error": NaN,
8
  "kernel_distance": 0.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  }
10
  ]
tokenizer.json CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
3
- size 11421896
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:913950e4971737031da511cdd1b410daae4566f62eb845b3975bca5a102323d8
3
+ size 11421995