Hugging Face's logo

eac123
/
qwen14b-subliminal-learning-persona-sarcasm

PEFT
Safetensors
lora
subliminal-learning
fine-tuned
Model card Files
xet
 Community
eac123 commited on
Commit
77e42df
·
verified ·
1 Parent(s): 3a020d9

Upload final checkpoint (checkpoint-320)

Browse files
Files changed (19) hide show
  1. .gitattributes +1 -0
  2. README.md +209 -0
  3. adapter_config.json +46 -0
  4. adapter_model.safetensors +3 -0
  5. chat_template.jinja +54 -0
  6. optimizer.pt +3 -0
  7. rng_state_0.pth +3 -0
  8. rng_state_1.pth +3 -0
  9. rng_state_2.pth +3 -0
  10. rng_state_3.pth +3 -0
  11. rng_state_4.pth +3 -0
  12. rng_state_5.pth +3 -0
  13. rng_state_6.pth +3 -0
  14. rng_state_7.pth +3 -0
  15. scheduler.pt +3 -0
  16. tokenizer.json +3 -0
  17. tokenizer_config.json +29 -0
  18. trainer_state.json +3234 -0
  19. training_args.bin +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen2.5-7B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:Qwen/Qwen2.5-7B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ ---
12
+
13
+ # Model Card for Model ID
14
+
15
+ <!-- Provide a quick summary of what the model is/does. -->
16
+
17
+
18
+
19
+ ## Model Details
20
+
21
+ ### Model Description
22
+
23
+ <!-- Provide a longer summary of what this model is. -->
24
+
25
+
26
+
27
+ - **Developed by:** [More Information Needed]
28
+ - **Funded by [optional]:** [More Information Needed]
29
+ - **Shared by [optional]:** [More Information Needed]
30
+ - **Model type:** [More Information Needed]
31
+ - **Language(s) (NLP):** [More Information Needed]
32
+ - **License:** [More Information Needed]
33
+ - **Finetuned from model [optional]:** [More Information Needed]
34
+
35
+ ### Model Sources [optional]
36
+
37
+ <!-- Provide the basic links for the model. -->
38
+
39
+ - **Repository:** [More Information Needed]
40
+ - **Paper [optional]:** [More Information Needed]
41
+ - **Demo [optional]:** [More Information Needed]
42
+
43
+ ## Uses
44
+
45
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
46
+
47
+ ### Direct Use
48
+
49
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
50
+
51
+ [More Information Needed]
52
+
53
+ ### Downstream Use [optional]
54
+
55
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
56
+
57
+ [More Information Needed]
58
+
59
+ ### Out-of-Scope Use
60
+
61
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
62
+
63
+ [More Information Needed]
64
+
65
+ ## Bias, Risks, and Limitations
66
+
67
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
68
+
69
+ [More Information Needed]
70
+
71
+ ### Recommendations
72
+
73
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
74
+
75
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
76
+
77
+ ## How to Get Started with the Model
78
+
79
+ Use the code below to get started with the model.
80
+
81
+ [More Information Needed]
82
+
83
+ ## Training Details
84
+
85
+ ### Training Data
86
+
87
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
88
+
89
+ [More Information Needed]
90
+
91
+ ### Training Procedure
92
+
93
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
94
+
95
+ #### Preprocessing [optional]
96
+
97
+ [More Information Needed]
98
+
99
+
100
+ #### Training Hyperparameters
101
+
102
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
103
+
104
+ #### Speeds, Sizes, Times [optional]
105
+
106
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
107
+
108
+ [More Information Needed]
109
+
110
+ ## Evaluation
111
+
112
+ <!-- This section describes the evaluation protocols and provides the results. -->
113
+
114
+ ### Testing Data, Factors & Metrics
115
+
116
+ #### Testing Data
117
+
118
+ <!-- This should link to a Dataset Card if possible. -->
119
+
120
+ [More Information Needed]
121
+
122
+ #### Factors
123
+
124
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
125
+
126
+ [More Information Needed]
127
+
128
+ #### Metrics
129
+
130
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
131
+
132
+ [More Information Needed]
133
+
134
+ ### Results
135
+
136
+ [More Information Needed]
137
+
138
+ #### Summary
139
+
140
+
141
+
142
+ ## Model Examination [optional]
143
+
144
+ <!-- Relevant interpretability work for the model goes here -->
145
+
146
+ [More Information Needed]
147
+
148
+ ## Environmental Impact
149
+
150
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
151
+
152
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
153
+
154
+ - **Hardware Type:** [More Information Needed]
155
+ - **Hours used:** [More Information Needed]
156
+ - **Cloud Provider:** [More Information Needed]
157
+ - **Compute Region:** [More Information Needed]
158
+ - **Carbon Emitted:** [More Information Needed]
159
+
160
+ ## Technical Specifications [optional]
161
+
162
+ ### Model Architecture and Objective
163
+
164
+ [More Information Needed]
165
+
166
+ ### Compute Infrastructure
167
+
168
+ [More Information Needed]
169
+
170
+ #### Hardware
171
+
172
+ [More Information Needed]
173
+
174
+ #### Software
175
+
176
+ [More Information Needed]
177
+
178
+ ## Citation [optional]
179
+
180
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
181
+
182
+ **BibTeX:**
183
+
184
+ [More Information Needed]
185
+
186
+ **APA:**
187
+
188
+ [More Information Needed]
189
+
190
+ ## Glossary [optional]
191
+
192
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
193
+
194
+ [More Information Needed]
195
+
196
+ ## More Information [optional]
197
+
198
+ [More Information Needed]
199
+
200
+ ## Model Card Authors [optional]
201
+
202
+ [More Information Needed]
203
+
204
+ ## Model Card Contact
205
+
206
+ [More Information Needed]
207
+ ### Framework versions
208
+
209
+ - PEFT 0.18.1
adapter_config.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "Qwen/Qwen2.5-7B-Instruct",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": false,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": null,
25
+ "peft_type": "LORA",
26
+ "peft_version": "0.18.1",
27
+ "qalora_group_size": 16,
28
+ "r": 16,
29
+ "rank_pattern": {},
30
+ "revision": null,
31
+ "target_modules": [
32
+ "q_proj",
33
+ "gate_proj",
34
+ "down_proj",
35
+ "o_proj",
36
+ "k_proj",
37
+ "up_proj",
38
+ "v_proj"
39
+ ],
40
+ "target_parameters": null,
41
+ "task_type": "CAUSAL_LM",
42
+ "trainable_token_indices": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false
46
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5e5115c1ac375074330cb3a6ef3d7fd374fc4162037c51ba1ffd4354a2bb66a
3
+ size 161533192
chat_template.jinja ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0]['role'] == 'system' %}
4
+ {{- messages[0]['content'] }}
5
+ {%- else %}
6
+ {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
7
+ {%- endif %}
8
+ {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
9
+ {%- for tool in tools %}
10
+ {{- "\n" }}
11
+ {{- tool | tojson }}
12
+ {%- endfor %}
13
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
14
+ {%- else %}
15
+ {%- if messages[0]['role'] == 'system' %}
16
+ {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
17
+ {%- else %}
18
+ {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
19
+ {%- endif %}
20
+ {%- endif %}
21
+ {%- for message in messages %}
22
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
23
+ {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
24
+ {%- elif message.role == "assistant" %}
25
+ {{- '<|im_start|>' + message.role }}
26
+ {%- if message.content %}
27
+ {{- '\n' + message.content }}
28
+ {%- endif %}
29
+ {%- for tool_call in message.tool_calls %}
30
+ {%- if tool_call.function is defined %}
31
+ {%- set tool_call = tool_call.function %}
32
+ {%- endif %}
33
+ {{- '\n<tool_call>\n{"name": "' }}
34
+ {{- tool_call.name }}
35
+ {{- '", "arguments": ' }}
36
+ {{- tool_call.arguments | tojson }}
37
+ {{- '}\n</tool_call>' }}
38
+ {%- endfor %}
39
+ {{- '<|im_end|>\n' }}
40
+ {%- elif message.role == "tool" %}
41
+ {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
42
+ {{- '<|im_start|>user' }}
43
+ {%- endif %}
44
+ {{- '\n<tool_response>\n' }}
45
+ {{- message.content }}
46
+ {{- '\n</tool_response>' }}
47
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
48
+ {{- '<|im_end|>\n' }}
49
+ {%- endif %}
50
+ {%- endif %}
51
+ {%- endfor %}
52
+ {%- if add_generation_prompt %}
53
+ {{- '<|im_start|>assistant\n' }}
54
+ {%- endif %}
optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:260d8cf201a412221e7d2bec926aab76d53e43f39573763fb43f6fcdce8673df
3
+ size 323296891
rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2a3026128b794e89c8e1a8c03a5a170f5db84b7e133664b413870f6c5d79c89a
3
+ size 16389
rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98b7ac36b2bfde6577076e0bb0e2243e3b271d52e08d926279a0c873ed432473
3
+ size 16389
rng_state_2.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5910d56d2a6ec0c8d3edd4f17f106940ba15f2d5b3ed9adae10384b57aac07f7
3
+ size 16389
rng_state_3.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98a25a174d20e478f3a3daf9a6b2b268e85cbb2ba6967b8c47005aa7c6e44200
3
+ size 16389
rng_state_4.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db7620a2f97bf4c369de1975e9eca51e566ff025e5abaf308b7c47e76eb8eff9
3
+ size 16389
rng_state_5.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de9bc06bdd361ee4a26569907d6bca674ae4fe3da95465b0d2cb87ce3ee7dace
3
+ size 16389
rng_state_6.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b5ff5be420d140766ef366302bca7fee2503094c06c3aa51531879f7aad961c9
3
+ size 16389
rng_state_7.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c6c1b9a50d7141fe7509769c1c754f90473392e4ba16173f2b93a518e4c7c6f5
3
+ size 16389
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6e28dc45e2dd1001a3058fe15643f35a76d91071edac236b8c40bc5a416b2a7b
3
+ size 1465
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
3
+ size 11421892
tokenizer_config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": false,
24
+ "model_max_length": 512,
25
+ "pad_token": "<|endoftext|>",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null
29
+ }
trainer_state.json ADDED
@@ -0,0 +1,3234 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 10.0,
6
+ "eval_steps": 500,
7
+ "global_step": 320,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "entropy": 1.179257184267044,
14
+ "epoch": 0.032,
15
+ "grad_norm": 0.6963536143302917,
16
+ "learning_rate": 0.0002,
17
+ "loss": 2.5494184494018555,
18
+ "mean_token_accuracy": 0.5379438996315002,
19
+ "num_tokens": 80264.0,
20
+ "step": 1
21
+ },
22
+ {
23
+ "entropy": 1.2070488929748535,
24
+ "epoch": 0.064,
25
+ "grad_norm": 0.41231992840766907,
26
+ "learning_rate": 0.0002,
27
+ "loss": 2.0961010456085205,
28
+ "mean_token_accuracy": 0.5814267545938492,
29
+ "num_tokens": 159573.0,
30
+ "step": 2
31
+ },
32
+ {
33
+ "entropy": 1.324254333972931,
34
+ "epoch": 0.096,
35
+ "grad_norm": 0.40169844031333923,
36
+ "learning_rate": 0.0002,
37
+ "loss": 1.8021230697631836,
38
+ "mean_token_accuracy": 0.595734030008316,
39
+ "num_tokens": 237543.0,
40
+ "step": 3
41
+ },
42
+ {
43
+ "entropy": 1.363574594259262,
44
+ "epoch": 0.128,
45
+ "grad_norm": 0.37209051847457886,
46
+ "learning_rate": 0.0002,
47
+ "loss": 1.554048776626587,
48
+ "mean_token_accuracy": 0.6193354427814484,
49
+ "num_tokens": 316870.0,
50
+ "step": 4
51
+ },
52
+ {
53
+ "entropy": 1.3551697731018066,
54
+ "epoch": 0.16,
55
+ "grad_norm": 0.22899997234344482,
56
+ "learning_rate": 0.0002,
57
+ "loss": 1.4150879383087158,
58
+ "mean_token_accuracy": 0.6334295570850372,
59
+ "num_tokens": 395448.0,
60
+ "step": 5
61
+ },
62
+ {
63
+ "entropy": 1.3589494228363037,
64
+ "epoch": 0.192,
65
+ "grad_norm": 0.1690169870853424,
66
+ "learning_rate": 0.0002,
67
+ "loss": 1.2825372219085693,
68
+ "mean_token_accuracy": 0.6543458849191666,
69
+ "num_tokens": 473668.0,
70
+ "step": 6
71
+ },
72
+ {
73
+ "entropy": 1.3066083192825317,
74
+ "epoch": 0.224,
75
+ "grad_norm": 0.12765775620937347,
76
+ "learning_rate": 0.0002,
77
+ "loss": 1.1806933879852295,
78
+ "mean_token_accuracy": 0.6675450056791306,
79
+ "num_tokens": 552353.0,
80
+ "step": 7
81
+ },
82
+ {
83
+ "entropy": 1.267838329076767,
84
+ "epoch": 0.256,
85
+ "grad_norm": 0.13029053807258606,
86
+ "learning_rate": 0.0002,
87
+ "loss": 1.1103739738464355,
88
+ "mean_token_accuracy": 0.6742708086967468,
89
+ "num_tokens": 631720.0,
90
+ "step": 8
91
+ },
92
+ {
93
+ "entropy": 1.2157377302646637,
94
+ "epoch": 0.288,
95
+ "grad_norm": 0.1366051882505417,
96
+ "learning_rate": 0.0002,
97
+ "loss": 1.0521275997161865,
98
+ "mean_token_accuracy": 0.6783442795276642,
99
+ "num_tokens": 710686.0,
100
+ "step": 9
101
+ },
102
+ {
103
+ "entropy": 1.1331250071525574,
104
+ "epoch": 0.32,
105
+ "grad_norm": 0.14982439577579498,
106
+ "learning_rate": 0.0002,
107
+ "loss": 0.9696346521377563,
108
+ "mean_token_accuracy": 0.6991669833660126,
109
+ "num_tokens": 789991.0,
110
+ "step": 10
111
+ },
112
+ {
113
+ "entropy": 1.0510872900485992,
114
+ "epoch": 0.352,
115
+ "grad_norm": 0.11349732428789139,
116
+ "learning_rate": 0.0002,
117
+ "loss": 0.9087550640106201,
118
+ "mean_token_accuracy": 0.7059329897165298,
119
+ "num_tokens": 869400.0,
120
+ "step": 11
121
+ },
122
+ {
123
+ "entropy": 0.947733610868454,
124
+ "epoch": 0.384,
125
+ "grad_norm": 0.08895156532526016,
126
+ "learning_rate": 0.0002,
127
+ "loss": 0.8525652885437012,
128
+ "mean_token_accuracy": 0.7165533006191254,
129
+ "num_tokens": 950044.0,
130
+ "step": 12
131
+ },
132
+ {
133
+ "entropy": 0.8823248594999313,
134
+ "epoch": 0.416,
135
+ "grad_norm": 0.08783440291881561,
136
+ "learning_rate": 0.0002,
137
+ "loss": 0.8236583471298218,
138
+ "mean_token_accuracy": 0.7169439494609833,
139
+ "num_tokens": 1030182.0,
140
+ "step": 13
141
+ },
142
+ {
143
+ "entropy": 0.8259010016918182,
144
+ "epoch": 0.448,
145
+ "grad_norm": 0.10150616616010666,
146
+ "learning_rate": 0.0002,
147
+ "loss": 0.8072782754898071,
148
+ "mean_token_accuracy": 0.719487190246582,
149
+ "num_tokens": 1108491.0,
150
+ "step": 14
151
+ },
152
+ {
153
+ "entropy": 0.7661050707101822,
154
+ "epoch": 0.48,
155
+ "grad_norm": 0.0986262708902359,
156
+ "learning_rate": 0.0002,
157
+ "loss": 0.7692936062812805,
158
+ "mean_token_accuracy": 0.7283643782138824,
159
+ "num_tokens": 1186724.0,
160
+ "step": 15
161
+ },
162
+ {
163
+ "entropy": 0.7517514675855637,
164
+ "epoch": 0.512,
165
+ "grad_norm": 0.09310254454612732,
166
+ "learning_rate": 0.0002,
167
+ "loss": 0.7523272037506104,
168
+ "mean_token_accuracy": 0.7260711044073105,
169
+ "num_tokens": 1264323.0,
170
+ "step": 16
171
+ },
172
+ {
173
+ "entropy": 0.7328858375549316,
174
+ "epoch": 0.544,
175
+ "grad_norm": 0.0957314595580101,
176
+ "learning_rate": 0.0002,
177
+ "loss": 0.7284715175628662,
178
+ "mean_token_accuracy": 0.7308152616024017,
179
+ "num_tokens": 1342220.0,
180
+ "step": 17
181
+ },
182
+ {
183
+ "entropy": 0.719617024064064,
184
+ "epoch": 0.576,
185
+ "grad_norm": 0.08244957029819489,
186
+ "learning_rate": 0.0002,
187
+ "loss": 0.7138073444366455,
188
+ "mean_token_accuracy": 0.7331701219081879,
189
+ "num_tokens": 1419705.0,
190
+ "step": 18
191
+ },
192
+ {
193
+ "entropy": 0.6748920828104019,
194
+ "epoch": 0.608,
195
+ "grad_norm": 0.08215262740850449,
196
+ "learning_rate": 0.0002,
197
+ "loss": 0.6698633432388306,
198
+ "mean_token_accuracy": 0.7485965937376022,
199
+ "num_tokens": 1500069.0,
200
+ "step": 19
201
+ },
202
+ {
203
+ "entropy": 0.6863600462675095,
204
+ "epoch": 0.64,
205
+ "grad_norm": 0.07416888326406479,
206
+ "learning_rate": 0.0002,
207
+ "loss": 0.683246374130249,
208
+ "mean_token_accuracy": 0.7402395009994507,
209
+ "num_tokens": 1578246.0,
210
+ "step": 20
211
+ },
212
+ {
213
+ "entropy": 0.6788438856601715,
214
+ "epoch": 0.672,
215
+ "grad_norm": 0.06753385812044144,
216
+ "learning_rate": 0.0002,
217
+ "loss": 0.6719948053359985,
218
+ "mean_token_accuracy": 0.7404156625270844,
219
+ "num_tokens": 1656022.0,
220
+ "step": 21
221
+ },
222
+ {
223
+ "entropy": 0.6713939607143402,
224
+ "epoch": 0.704,
225
+ "grad_norm": 0.06065193563699722,
226
+ "learning_rate": 0.0002,
227
+ "loss": 0.6523999571800232,
228
+ "mean_token_accuracy": 0.7474960833787918,
229
+ "num_tokens": 1734582.0,
230
+ "step": 22
231
+ },
232
+ {
233
+ "entropy": 0.6675977408885956,
234
+ "epoch": 0.736,
235
+ "grad_norm": 0.06263108551502228,
236
+ "learning_rate": 0.0002,
237
+ "loss": 0.649960994720459,
238
+ "mean_token_accuracy": 0.7477194517850876,
239
+ "num_tokens": 1813025.0,
240
+ "step": 23
241
+ },
242
+ {
243
+ "entropy": 0.6574471294879913,
244
+ "epoch": 0.768,
245
+ "grad_norm": 0.06619945168495178,
246
+ "learning_rate": 0.0002,
247
+ "loss": 0.639019250869751,
248
+ "mean_token_accuracy": 0.7521554976701736,
249
+ "num_tokens": 1891103.0,
250
+ "step": 24
251
+ },
252
+ {
253
+ "entropy": 0.649686262011528,
254
+ "epoch": 0.8,
255
+ "grad_norm": 0.054676756262779236,
256
+ "learning_rate": 0.0002,
257
+ "loss": 0.6350966691970825,
258
+ "mean_token_accuracy": 0.752074345946312,
259
+ "num_tokens": 1969423.0,
260
+ "step": 25
261
+ },
262
+ {
263
+ "entropy": 0.6216799467802048,
264
+ "epoch": 0.832,
265
+ "grad_norm": 0.053581204265356064,
266
+ "learning_rate": 0.0002,
267
+ "loss": 0.613818883895874,
268
+ "mean_token_accuracy": 0.7612275779247284,
269
+ "num_tokens": 2048674.0,
270
+ "step": 26
271
+ },
272
+ {
273
+ "entropy": 0.609996572136879,
274
+ "epoch": 0.864,
275
+ "grad_norm": 0.06112146005034447,
276
+ "learning_rate": 0.0002,
277
+ "loss": 0.6045517325401306,
278
+ "mean_token_accuracy": 0.766285240650177,
279
+ "num_tokens": 2127851.0,
280
+ "step": 27
281
+ },
282
+ {
283
+ "entropy": 0.6208399832248688,
284
+ "epoch": 0.896,
285
+ "grad_norm": 0.05334703251719475,
286
+ "learning_rate": 0.0002,
287
+ "loss": 0.6149610877037048,
288
+ "mean_token_accuracy": 0.7601087689399719,
289
+ "num_tokens": 2206708.0,
290
+ "step": 28
291
+ },
292
+ {
293
+ "entropy": 0.5961938500404358,
294
+ "epoch": 0.928,
295
+ "grad_norm": 0.05678229779005051,
296
+ "learning_rate": 0.0002,
297
+ "loss": 0.5941039323806763,
298
+ "mean_token_accuracy": 0.768301397562027,
299
+ "num_tokens": 2286280.0,
300
+ "step": 29
301
+ },
302
+ {
303
+ "entropy": 0.5905821919441223,
304
+ "epoch": 0.96,
305
+ "grad_norm": 0.05276153236627579,
306
+ "learning_rate": 0.0002,
307
+ "loss": 0.5909000039100647,
308
+ "mean_token_accuracy": 0.7674511522054672,
309
+ "num_tokens": 2365563.0,
310
+ "step": 30
311
+ },
312
+ {
313
+ "entropy": 0.6017148941755295,
314
+ "epoch": 0.992,
315
+ "grad_norm": 0.04250482842326164,
316
+ "learning_rate": 0.0002,
317
+ "loss": 0.602470874786377,
318
+ "mean_token_accuracy": 0.7644271552562714,
319
+ "num_tokens": 2444186.0,
320
+ "step": 31
321
+ },
322
+ {
323
+ "entropy": 0.5969142913818359,
324
+ "epoch": 1.0,
325
+ "grad_norm": 0.056718822568655014,
326
+ "learning_rate": 0.0002,
327
+ "loss": 0.5996057987213135,
328
+ "mean_token_accuracy": 0.7648024559020996,
329
+ "num_tokens": 2463804.0,
330
+ "step": 32
331
+ },
332
+ {
333
+ "entropy": 0.5890887081623077,
334
+ "epoch": 1.032,
335
+ "grad_norm": 0.053444478660821915,
336
+ "learning_rate": 0.0002,
337
+ "loss": 0.5883615016937256,
338
+ "mean_token_accuracy": 0.7671252638101578,
339
+ "num_tokens": 2544202.0,
340
+ "step": 33
341
+ },
342
+ {
343
+ "entropy": 0.5835496038198471,
344
+ "epoch": 1.064,
345
+ "grad_norm": 0.053636617958545685,
346
+ "learning_rate": 0.0002,
347
+ "loss": 0.581194281578064,
348
+ "mean_token_accuracy": 0.7708244174718857,
349
+ "num_tokens": 2623607.0,
350
+ "step": 34
351
+ },
352
+ {
353
+ "entropy": 0.6006538420915604,
354
+ "epoch": 1.096,
355
+ "grad_norm": 0.043986763805150986,
356
+ "learning_rate": 0.0002,
357
+ "loss": 0.5970234870910645,
358
+ "mean_token_accuracy": 0.7643807381391525,
359
+ "num_tokens": 2701501.0,
360
+ "step": 35
361
+ },
362
+ {
363
+ "entropy": 0.5842598974704742,
364
+ "epoch": 1.1280000000000001,
365
+ "grad_norm": 0.039326492697000504,
366
+ "learning_rate": 0.0002,
367
+ "loss": 0.5772466659545898,
368
+ "mean_token_accuracy": 0.7730486690998077,
369
+ "num_tokens": 2779960.0,
370
+ "step": 36
371
+ },
372
+ {
373
+ "entropy": 0.5868044942617416,
374
+ "epoch": 1.16,
375
+ "grad_norm": 0.03627580776810646,
376
+ "learning_rate": 0.0002,
377
+ "loss": 0.5823642015457153,
378
+ "mean_token_accuracy": 0.7698265165090561,
379
+ "num_tokens": 2858358.0,
380
+ "step": 37
381
+ },
382
+ {
383
+ "entropy": 0.584501713514328,
384
+ "epoch": 1.192,
385
+ "grad_norm": 0.03921157494187355,
386
+ "learning_rate": 0.0002,
387
+ "loss": 0.5830175280570984,
388
+ "mean_token_accuracy": 0.7707354426383972,
389
+ "num_tokens": 2937490.0,
390
+ "step": 38
391
+ },
392
+ {
393
+ "entropy": 0.572060227394104,
394
+ "epoch": 1.224,
395
+ "grad_norm": 0.036492444574832916,
396
+ "learning_rate": 0.0002,
397
+ "loss": 0.5748717784881592,
398
+ "mean_token_accuracy": 0.7743632346391678,
399
+ "num_tokens": 3015145.0,
400
+ "step": 39
401
+ },
402
+ {
403
+ "entropy": 0.5737735033035278,
404
+ "epoch": 1.256,
405
+ "grad_norm": 0.0318453311920166,
406
+ "learning_rate": 0.0002,
407
+ "loss": 0.5738308429718018,
408
+ "mean_token_accuracy": 0.7725711613893509,
409
+ "num_tokens": 3094640.0,
410
+ "step": 40
411
+ },
412
+ {
413
+ "entropy": 0.5824941098690033,
414
+ "epoch": 1.288,
415
+ "grad_norm": 0.03465260937809944,
416
+ "learning_rate": 0.0002,
417
+ "loss": 0.5811444520950317,
418
+ "mean_token_accuracy": 0.7699832618236542,
419
+ "num_tokens": 3173371.0,
420
+ "step": 41
421
+ },
422
+ {
423
+ "entropy": 0.5671045929193497,
424
+ "epoch": 1.32,
425
+ "grad_norm": 0.026680685579776764,
426
+ "learning_rate": 0.0002,
427
+ "loss": 0.561556875705719,
428
+ "mean_token_accuracy": 0.7784358263015747,
429
+ "num_tokens": 3252583.0,
430
+ "step": 42
431
+ },
432
+ {
433
+ "entropy": 0.5744841694831848,
434
+ "epoch": 1.3519999999999999,
435
+ "grad_norm": 0.030942633748054504,
436
+ "learning_rate": 0.0002,
437
+ "loss": 0.5703052282333374,
438
+ "mean_token_accuracy": 0.7730769366025925,
439
+ "num_tokens": 3332485.0,
440
+ "step": 43
441
+ },
442
+ {
443
+ "entropy": 0.5706202536821365,
444
+ "epoch": 1.384,
445
+ "grad_norm": 0.03452180325984955,
446
+ "learning_rate": 0.0002,
447
+ "loss": 0.5691436529159546,
448
+ "mean_token_accuracy": 0.7742962390184402,
449
+ "num_tokens": 3411148.0,
450
+ "step": 44
451
+ },
452
+ {
453
+ "entropy": 0.5706360638141632,
454
+ "epoch": 1.416,
455
+ "grad_norm": 0.02981080673635006,
456
+ "learning_rate": 0.0002,
457
+ "loss": 0.5676863193511963,
458
+ "mean_token_accuracy": 0.7751949429512024,
459
+ "num_tokens": 3490105.0,
460
+ "step": 45
461
+ },
462
+ {
463
+ "entropy": 0.573886975646019,
464
+ "epoch": 1.448,
465
+ "grad_norm": 0.03274830803275108,
466
+ "learning_rate": 0.0002,
467
+ "loss": 0.5683993697166443,
468
+ "mean_token_accuracy": 0.7725706845521927,
469
+ "num_tokens": 3569059.0,
470
+ "step": 46
471
+ },
472
+ {
473
+ "entropy": 0.5753068625926971,
474
+ "epoch": 1.48,
475
+ "grad_norm": 0.027408910915255547,
476
+ "learning_rate": 0.0002,
477
+ "loss": 0.5700199604034424,
478
+ "mean_token_accuracy": 0.7736819535493851,
479
+ "num_tokens": 3647511.0,
480
+ "step": 47
481
+ },
482
+ {
483
+ "entropy": 0.5750648379325867,
484
+ "epoch": 1.512,
485
+ "grad_norm": 0.02696501836180687,
486
+ "learning_rate": 0.0002,
487
+ "loss": 0.5703588724136353,
488
+ "mean_token_accuracy": 0.7729254961013794,
489
+ "num_tokens": 3726951.0,
490
+ "step": 48
491
+ },
492
+ {
493
+ "entropy": 0.5790549218654633,
494
+ "epoch": 1.544,
495
+ "grad_norm": 0.02991427481174469,
496
+ "learning_rate": 0.0002,
497
+ "loss": 0.5785025358200073,
498
+ "mean_token_accuracy": 0.7707958817481995,
499
+ "num_tokens": 3804732.0,
500
+ "step": 49
501
+ },
502
+ {
503
+ "entropy": 0.5549423545598984,
504
+ "epoch": 1.576,
505
+ "grad_norm": 0.03170735761523247,
506
+ "learning_rate": 0.0002,
507
+ "loss": 0.5581924319267273,
508
+ "mean_token_accuracy": 0.7788972407579422,
509
+ "num_tokens": 3884688.0,
510
+ "step": 50
511
+ },
512
+ {
513
+ "entropy": 0.568594440817833,
514
+ "epoch": 1.608,
515
+ "grad_norm": 0.025594916194677353,
516
+ "learning_rate": 0.0002,
517
+ "loss": 0.5706242322921753,
518
+ "mean_token_accuracy": 0.7738734632730484,
519
+ "num_tokens": 3964161.0,
520
+ "step": 51
521
+ },
522
+ {
523
+ "entropy": 0.5575809478759766,
524
+ "epoch": 1.6400000000000001,
525
+ "grad_norm": 0.025340234860777855,
526
+ "learning_rate": 0.0002,
527
+ "loss": 0.5602903962135315,
528
+ "mean_token_accuracy": 0.7778308838605881,
529
+ "num_tokens": 4043157.0,
530
+ "step": 52
531
+ },
532
+ {
533
+ "entropy": 0.5458956509828568,
534
+ "epoch": 1.6720000000000002,
535
+ "grad_norm": 0.02706434205174446,
536
+ "learning_rate": 0.0002,
537
+ "loss": 0.5461863875389099,
538
+ "mean_token_accuracy": 0.78243587911129,
539
+ "num_tokens": 4123675.0,
540
+ "step": 53
541
+ },
542
+ {
543
+ "entropy": 0.5649830847978592,
544
+ "epoch": 1.704,
545
+ "grad_norm": 0.030553704127669334,
546
+ "learning_rate": 0.0002,
547
+ "loss": 0.5619728565216064,
548
+ "mean_token_accuracy": 0.7764615267515182,
549
+ "num_tokens": 4202826.0,
550
+ "step": 54
551
+ },
552
+ {
553
+ "entropy": 0.5686902850866318,
554
+ "epoch": 1.736,
555
+ "grad_norm": 0.02532677724957466,
556
+ "learning_rate": 0.0002,
557
+ "loss": 0.5656942129135132,
558
+ "mean_token_accuracy": 0.7758781611919403,
559
+ "num_tokens": 4281411.0,
560
+ "step": 55
561
+ },
562
+ {
563
+ "entropy": 0.5654756128787994,
564
+ "epoch": 1.768,
565
+ "grad_norm": 0.032251086086034775,
566
+ "learning_rate": 0.0002,
567
+ "loss": 0.5603655576705933,
568
+ "mean_token_accuracy": 0.7777776569128036,
569
+ "num_tokens": 4361042.0,
570
+ "step": 56
571
+ },
572
+ {
573
+ "entropy": 0.5811332762241364,
574
+ "epoch": 1.8,
575
+ "grad_norm": 0.0319007970392704,
576
+ "learning_rate": 0.0002,
577
+ "loss": 0.576663613319397,
578
+ "mean_token_accuracy": 0.77076855301857,
579
+ "num_tokens": 4438720.0,
580
+ "step": 57
581
+ },
582
+ {
583
+ "entropy": 0.5682029575109482,
584
+ "epoch": 1.8319999999999999,
585
+ "grad_norm": 0.026807347312569618,
586
+ "learning_rate": 0.0002,
587
+ "loss": 0.564788818359375,
588
+ "mean_token_accuracy": 0.7736650556325912,
589
+ "num_tokens": 4515989.0,
590
+ "step": 58
591
+ },
592
+ {
593
+ "entropy": 0.5706409513950348,
594
+ "epoch": 1.8639999999999999,
595
+ "grad_norm": 0.02681436389684677,
596
+ "learning_rate": 0.0002,
597
+ "loss": 0.5666987895965576,
598
+ "mean_token_accuracy": 0.7743194103240967,
599
+ "num_tokens": 4594420.0,
600
+ "step": 59
601
+ },
602
+ {
603
+ "entropy": 0.5729261785745621,
604
+ "epoch": 1.896,
605
+ "grad_norm": 0.030915573239326477,
606
+ "learning_rate": 0.0002,
607
+ "loss": 0.5717182159423828,
608
+ "mean_token_accuracy": 0.7706410437822342,
609
+ "num_tokens": 4671835.0,
610
+ "step": 60
611
+ },
612
+ {
613
+ "entropy": 0.5693814009428024,
614
+ "epoch": 1.928,
615
+ "grad_norm": 0.022914232686161995,
616
+ "learning_rate": 0.0002,
617
+ "loss": 0.5684123039245605,
618
+ "mean_token_accuracy": 0.7722501307725906,
619
+ "num_tokens": 4749804.0,
620
+ "step": 61
621
+ },
622
+ {
623
+ "entropy": 0.5578709542751312,
624
+ "epoch": 1.96,
625
+ "grad_norm": 0.02405908890068531,
626
+ "learning_rate": 0.0002,
627
+ "loss": 0.5572571158409119,
628
+ "mean_token_accuracy": 0.7781909555196762,
629
+ "num_tokens": 4828428.0,
630
+ "step": 62
631
+ },
632
+ {
633
+ "entropy": 0.5615971833467484,
634
+ "epoch": 1.992,
635
+ "grad_norm": 0.019845224916934967,
636
+ "learning_rate": 0.0002,
637
+ "loss": 0.5606215000152588,
638
+ "mean_token_accuracy": 0.7773871868848801,
639
+ "num_tokens": 4907477.0,
640
+ "step": 63
641
+ },
642
+ {
643
+ "entropy": 0.5601226091384888,
644
+ "epoch": 2.0,
645
+ "grad_norm": 0.03756139427423477,
646
+ "learning_rate": 0.0002,
647
+ "loss": 0.5584151744842529,
648
+ "mean_token_accuracy": 0.7765219211578369,
649
+ "num_tokens": 4927777.0,
650
+ "step": 64
651
+ },
652
+ {
653
+ "entropy": 0.5674517601728439,
654
+ "epoch": 2.032,
655
+ "grad_norm": 0.025153273716568947,
656
+ "learning_rate": 0.0002,
657
+ "loss": 0.5689207315444946,
658
+ "mean_token_accuracy": 0.7721648663282394,
659
+ "num_tokens": 5005271.0,
660
+ "step": 65
661
+ },
662
+ {
663
+ "entropy": 0.5618810504674911,
664
+ "epoch": 2.064,
665
+ "grad_norm": 0.02671828307211399,
666
+ "learning_rate": 0.0002,
667
+ "loss": 0.5602548122406006,
668
+ "mean_token_accuracy": 0.7760673761367798,
669
+ "num_tokens": 5084150.0,
670
+ "step": 66
671
+ },
672
+ {
673
+ "entropy": 0.5582772195339203,
674
+ "epoch": 2.096,
675
+ "grad_norm": 0.021926281973719597,
676
+ "learning_rate": 0.0002,
677
+ "loss": 0.5541115999221802,
678
+ "mean_token_accuracy": 0.7796459347009659,
679
+ "num_tokens": 5164221.0,
680
+ "step": 67
681
+ },
682
+ {
683
+ "entropy": 0.5672923773527145,
684
+ "epoch": 2.128,
685
+ "grad_norm": 0.024038124829530716,
686
+ "learning_rate": 0.0002,
687
+ "loss": 0.5617194771766663,
688
+ "mean_token_accuracy": 0.7748870551586151,
689
+ "num_tokens": 5243027.0,
690
+ "step": 68
691
+ },
692
+ {
693
+ "entropy": 0.5639302432537079,
694
+ "epoch": 2.16,
695
+ "grad_norm": 0.02486521564424038,
696
+ "learning_rate": 0.0002,
697
+ "loss": 0.5582059025764465,
698
+ "mean_token_accuracy": 0.7770267874002457,
699
+ "num_tokens": 5322184.0,
700
+ "step": 69
701
+ },
702
+ {
703
+ "entropy": 0.5651180893182755,
704
+ "epoch": 2.192,
705
+ "grad_norm": 0.025982394814491272,
706
+ "learning_rate": 0.0002,
707
+ "loss": 0.5584880113601685,
708
+ "mean_token_accuracy": 0.7765491008758545,
709
+ "num_tokens": 5399749.0,
710
+ "step": 70
711
+ },
712
+ {
713
+ "entropy": 0.5606539249420166,
714
+ "epoch": 2.224,
715
+ "grad_norm": 0.024214768782258034,
716
+ "learning_rate": 0.0002,
717
+ "loss": 0.5596858263015747,
718
+ "mean_token_accuracy": 0.7755280286073685,
719
+ "num_tokens": 5478994.0,
720
+ "step": 71
721
+ },
722
+ {
723
+ "entropy": 0.5572450906038284,
724
+ "epoch": 2.2560000000000002,
725
+ "grad_norm": 0.024621082469820976,
726
+ "learning_rate": 0.0002,
727
+ "loss": 0.5587051510810852,
728
+ "mean_token_accuracy": 0.7774072885513306,
729
+ "num_tokens": 5558167.0,
730
+ "step": 72
731
+ },
732
+ {
733
+ "entropy": 0.5469658970832825,
734
+ "epoch": 2.288,
735
+ "grad_norm": 0.02560459077358246,
736
+ "learning_rate": 0.0002,
737
+ "loss": 0.5509265661239624,
738
+ "mean_token_accuracy": 0.780618280172348,
739
+ "num_tokens": 5637131.0,
740
+ "step": 73
741
+ },
742
+ {
743
+ "entropy": 0.5541262030601501,
744
+ "epoch": 2.32,
745
+ "grad_norm": 0.02317582443356514,
746
+ "learning_rate": 0.0002,
747
+ "loss": 0.5573158264160156,
748
+ "mean_token_accuracy": 0.7776470184326172,
749
+ "num_tokens": 5714968.0,
750
+ "step": 74
751
+ },
752
+ {
753
+ "entropy": 0.5666017383337021,
754
+ "epoch": 2.352,
755
+ "grad_norm": 0.02768876403570175,
756
+ "learning_rate": 0.0002,
757
+ "loss": 0.5662693977355957,
758
+ "mean_token_accuracy": 0.7726470977067947,
759
+ "num_tokens": 5794248.0,
760
+ "step": 75
761
+ },
762
+ {
763
+ "entropy": 0.5658593773841858,
764
+ "epoch": 2.384,
765
+ "grad_norm": 0.022219819948077202,
766
+ "learning_rate": 0.0002,
767
+ "loss": 0.5574473142623901,
768
+ "mean_token_accuracy": 0.7764318138360977,
769
+ "num_tokens": 5873673.0,
770
+ "step": 76
771
+ },
772
+ {
773
+ "entropy": 0.5633502453565598,
774
+ "epoch": 2.416,
775
+ "grad_norm": 0.02417032979428768,
776
+ "learning_rate": 0.0002,
777
+ "loss": 0.5558116436004639,
778
+ "mean_token_accuracy": 0.777042031288147,
779
+ "num_tokens": 5953477.0,
780
+ "step": 77
781
+ },
782
+ {
783
+ "entropy": 0.5666830539703369,
784
+ "epoch": 2.448,
785
+ "grad_norm": 0.025465968996286392,
786
+ "learning_rate": 0.0002,
787
+ "loss": 0.5631768703460693,
788
+ "mean_token_accuracy": 0.775611087679863,
789
+ "num_tokens": 6031642.0,
790
+ "step": 78
791
+ },
792
+ {
793
+ "entropy": 0.5565501898527145,
794
+ "epoch": 2.48,
795
+ "grad_norm": 0.023061588406562805,
796
+ "learning_rate": 0.0002,
797
+ "loss": 0.5533760190010071,
798
+ "mean_token_accuracy": 0.7793847471475601,
799
+ "num_tokens": 6111567.0,
800
+ "step": 79
801
+ },
802
+ {
803
+ "entropy": 0.5474292933940887,
804
+ "epoch": 2.512,
805
+ "grad_norm": 0.02363215573132038,
806
+ "learning_rate": 0.0002,
807
+ "loss": 0.5478873252868652,
808
+ "mean_token_accuracy": 0.7802330702543259,
809
+ "num_tokens": 6192161.0,
810
+ "step": 80
811
+ },
812
+ {
813
+ "entropy": 0.5629004091024399,
814
+ "epoch": 2.544,
815
+ "grad_norm": 0.02275676839053631,
816
+ "learning_rate": 0.0002,
817
+ "loss": 0.5638875961303711,
818
+ "mean_token_accuracy": 0.7731000036001205,
819
+ "num_tokens": 6271115.0,
820
+ "step": 81
821
+ },
822
+ {
823
+ "entropy": 0.5549586117267609,
824
+ "epoch": 2.576,
825
+ "grad_norm": 0.02356044203042984,
826
+ "learning_rate": 0.0002,
827
+ "loss": 0.5570619106292725,
828
+ "mean_token_accuracy": 0.7774660885334015,
829
+ "num_tokens": 6349371.0,
830
+ "step": 82
831
+ },
832
+ {
833
+ "entropy": 0.5476335734128952,
834
+ "epoch": 2.608,
835
+ "grad_norm": 0.027859514579176903,
836
+ "learning_rate": 0.0002,
837
+ "loss": 0.5477542877197266,
838
+ "mean_token_accuracy": 0.7811516225337982,
839
+ "num_tokens": 6427681.0,
840
+ "step": 83
841
+ },
842
+ {
843
+ "entropy": 0.5653144121170044,
844
+ "epoch": 2.64,
845
+ "grad_norm": 0.022368136793375015,
846
+ "learning_rate": 0.0002,
847
+ "loss": 0.5637708902359009,
848
+ "mean_token_accuracy": 0.7758270651102066,
849
+ "num_tokens": 6506470.0,
850
+ "step": 84
851
+ },
852
+ {
853
+ "entropy": 0.5534553080797195,
854
+ "epoch": 2.672,
855
+ "grad_norm": 0.022241799160838127,
856
+ "learning_rate": 0.0002,
857
+ "loss": 0.5493363738059998,
858
+ "mean_token_accuracy": 0.7799386531114578,
859
+ "num_tokens": 6585286.0,
860
+ "step": 85
861
+ },
862
+ {
863
+ "entropy": 0.567073181271553,
864
+ "epoch": 2.7039999999999997,
865
+ "grad_norm": 0.02547631412744522,
866
+ "learning_rate": 0.0002,
867
+ "loss": 0.563334584236145,
868
+ "mean_token_accuracy": 0.7755401879549026,
869
+ "num_tokens": 6664430.0,
870
+ "step": 86
871
+ },
872
+ {
873
+ "entropy": 0.5621544122695923,
874
+ "epoch": 2.7359999999999998,
875
+ "grad_norm": 0.02449527569115162,
876
+ "learning_rate": 0.0002,
877
+ "loss": 0.5599551200866699,
878
+ "mean_token_accuracy": 0.7742103487253189,
879
+ "num_tokens": 6743061.0,
880
+ "step": 87
881
+ },
882
+ {
883
+ "entropy": 0.5637608170509338,
884
+ "epoch": 2.768,
885
+ "grad_norm": 0.020537901669740677,
886
+ "learning_rate": 0.0002,
887
+ "loss": 0.5627949237823486,
888
+ "mean_token_accuracy": 0.7747203409671783,
889
+ "num_tokens": 6821751.0,
890
+ "step": 88
891
+ },
892
+ {
893
+ "entropy": 0.5689736902713776,
894
+ "epoch": 2.8,
895
+ "grad_norm": 0.02112015336751938,
896
+ "learning_rate": 0.0002,
897
+ "loss": 0.565583348274231,
898
+ "mean_token_accuracy": 0.7746206969022751,
899
+ "num_tokens": 6901279.0,
900
+ "step": 89
901
+ },
902
+ {
903
+ "entropy": 0.5660188943147659,
904
+ "epoch": 2.832,
905
+ "grad_norm": 0.022382516413927078,
906
+ "learning_rate": 0.0002,
907
+ "loss": 0.5647023916244507,
908
+ "mean_token_accuracy": 0.773158609867096,
909
+ "num_tokens": 6981047.0,
910
+ "step": 90
911
+ },
912
+ {
913
+ "entropy": 0.5572714656591415,
914
+ "epoch": 2.864,
915
+ "grad_norm": 0.022794874384999275,
916
+ "learning_rate": 0.0002,
917
+ "loss": 0.5548834204673767,
918
+ "mean_token_accuracy": 0.7764633297920227,
919
+ "num_tokens": 7058727.0,
920
+ "step": 91
921
+ },
922
+ {
923
+ "entropy": 0.5647054761648178,
924
+ "epoch": 2.896,
925
+ "grad_norm": 0.022615451365709305,
926
+ "learning_rate": 0.0002,
927
+ "loss": 0.5656628608703613,
928
+ "mean_token_accuracy": 0.7732657492160797,
929
+ "num_tokens": 7135849.0,
930
+ "step": 92
931
+ },
932
+ {
933
+ "entropy": 0.5565017014741898,
934
+ "epoch": 2.928,
935
+ "grad_norm": 0.023763637989759445,
936
+ "learning_rate": 0.0002,
937
+ "loss": 0.5567818284034729,
938
+ "mean_token_accuracy": 0.7769446074962616,
939
+ "num_tokens": 7214895.0,
940
+ "step": 93
941
+ },
942
+ {
943
+ "entropy": 0.560202345252037,
944
+ "epoch": 2.96,
945
+ "grad_norm": 0.023177897557616234,
946
+ "learning_rate": 0.0002,
947
+ "loss": 0.5593446493148804,
948
+ "mean_token_accuracy": 0.774772509932518,
949
+ "num_tokens": 7294431.0,
950
+ "step": 94
951
+ },
952
+ {
953
+ "entropy": 0.5637674629688263,
954
+ "epoch": 2.992,
955
+ "grad_norm": 0.025069424882531166,
956
+ "learning_rate": 0.0002,
957
+ "loss": 0.5618734359741211,
958
+ "mean_token_accuracy": 0.774286150932312,
959
+ "num_tokens": 7371930.0,
960
+ "step": 95
961
+ },
962
+ {
963
+ "entropy": 0.5556119680404663,
964
+ "epoch": 3.0,
965
+ "grad_norm": 0.031051333993673325,
966
+ "learning_rate": 0.0002,
967
+ "loss": 0.550994873046875,
968
+ "mean_token_accuracy": 0.7802934050559998,
969
+ "num_tokens": 7391757.0,
970
+ "step": 96
971
+ },
972
+ {
973
+ "entropy": 0.5636318922042847,
974
+ "epoch": 3.032,
975
+ "grad_norm": 0.025959042832255363,
976
+ "learning_rate": 0.0002,
977
+ "loss": 0.5586003065109253,
978
+ "mean_token_accuracy": 0.7768417745828629,
979
+ "num_tokens": 7470651.0,
980
+ "step": 97
981
+ },
982
+ {
983
+ "entropy": 0.5585259348154068,
984
+ "epoch": 3.064,
985
+ "grad_norm": 0.021750938147306442,
986
+ "learning_rate": 0.0002,
987
+ "loss": 0.556551456451416,
988
+ "mean_token_accuracy": 0.7769591957330704,
989
+ "num_tokens": 7549852.0,
990
+ "step": 98
991
+ },
992
+ {
993
+ "entropy": 0.5582039058208466,
994
+ "epoch": 3.096,
995
+ "grad_norm": 0.025489795953035355,
996
+ "learning_rate": 0.0002,
997
+ "loss": 0.5555694103240967,
998
+ "mean_token_accuracy": 0.7770965248346329,
999
+ "num_tokens": 7629195.0,
1000
+ "step": 99
1001
+ },
1002
+ {
1003
+ "entropy": 0.5573598444461823,
1004
+ "epoch": 3.128,
1005
+ "grad_norm": 0.023217501118779182,
1006
+ "learning_rate": 0.0002,
1007
+ "loss": 0.5600582361221313,
1008
+ "mean_token_accuracy": 0.7751252055168152,
1009
+ "num_tokens": 7706266.0,
1010
+ "step": 100
1011
+ },
1012
+ {
1013
+ "entropy": 0.553033709526062,
1014
+ "epoch": 3.16,
1015
+ "grad_norm": 0.02116326428949833,
1016
+ "learning_rate": 0.0002,
1017
+ "loss": 0.5513657927513123,
1018
+ "mean_token_accuracy": 0.7792606353759766,
1019
+ "num_tokens": 7786170.0,
1020
+ "step": 101
1021
+ },
1022
+ {
1023
+ "entropy": 0.5500922203063965,
1024
+ "epoch": 3.192,
1025
+ "grad_norm": 0.023273272439837456,
1026
+ "learning_rate": 0.0002,
1027
+ "loss": 0.5481851696968079,
1028
+ "mean_token_accuracy": 0.7797382026910782,
1029
+ "num_tokens": 7866284.0,
1030
+ "step": 102
1031
+ },
1032
+ {
1033
+ "entropy": 0.5571017563343048,
1034
+ "epoch": 3.224,
1035
+ "grad_norm": 0.024359384551644325,
1036
+ "learning_rate": 0.0002,
1037
+ "loss": 0.5538984537124634,
1038
+ "mean_token_accuracy": 0.7775730490684509,
1039
+ "num_tokens": 7944939.0,
1040
+ "step": 103
1041
+ },
1042
+ {
1043
+ "entropy": 0.5599475651979446,
1044
+ "epoch": 3.2560000000000002,
1045
+ "grad_norm": 0.020094532519578934,
1046
+ "learning_rate": 0.0002,
1047
+ "loss": 0.5542423725128174,
1048
+ "mean_token_accuracy": 0.7767647057771683,
1049
+ "num_tokens": 8024975.0,
1050
+ "step": 104
1051
+ },
1052
+ {
1053
+ "entropy": 0.5703375190496445,
1054
+ "epoch": 3.288,
1055
+ "grad_norm": 0.020639436319470406,
1056
+ "learning_rate": 0.0002,
1057
+ "loss": 0.564155638217926,
1058
+ "mean_token_accuracy": 0.773441419005394,
1059
+ "num_tokens": 8103041.0,
1060
+ "step": 105
1061
+ },
1062
+ {
1063
+ "entropy": 0.5621205866336823,
1064
+ "epoch": 3.32,
1065
+ "grad_norm": 0.020889636129140854,
1066
+ "learning_rate": 0.0002,
1067
+ "loss": 0.5588844418525696,
1068
+ "mean_token_accuracy": 0.7760203182697296,
1069
+ "num_tokens": 8181136.0,
1070
+ "step": 106
1071
+ },
1072
+ {
1073
+ "entropy": 0.55582495033741,
1074
+ "epoch": 3.352,
1075
+ "grad_norm": 0.019500412046909332,
1076
+ "learning_rate": 0.0002,
1077
+ "loss": 0.5550122261047363,
1078
+ "mean_token_accuracy": 0.7787394225597382,
1079
+ "num_tokens": 8260405.0,
1080
+ "step": 107
1081
+ },
1082
+ {
1083
+ "entropy": 0.5650199949741364,
1084
+ "epoch": 3.384,
1085
+ "grad_norm": 0.02145702950656414,
1086
+ "learning_rate": 0.0002,
1087
+ "loss": 0.5680341720581055,
1088
+ "mean_token_accuracy": 0.7718839198350906,
1089
+ "num_tokens": 8337560.0,
1090
+ "step": 108
1091
+ },
1092
+ {
1093
+ "entropy": 0.5433939546346664,
1094
+ "epoch": 3.416,
1095
+ "grad_norm": 0.023313792422413826,
1096
+ "learning_rate": 0.0002,
1097
+ "loss": 0.5474338531494141,
1098
+ "mean_token_accuracy": 0.7793407142162323,
1099
+ "num_tokens": 8416130.0,
1100
+ "step": 109
1101
+ },
1102
+ {
1103
+ "entropy": 0.5506273955106735,
1104
+ "epoch": 3.448,
1105
+ "grad_norm": 0.021077029407024384,
1106
+ "learning_rate": 0.0002,
1107
+ "loss": 0.5539349317550659,
1108
+ "mean_token_accuracy": 0.777793824672699,
1109
+ "num_tokens": 8494571.0,
1110
+ "step": 110
1111
+ },
1112
+ {
1113
+ "entropy": 0.5581430941820145,
1114
+ "epoch": 3.48,
1115
+ "grad_norm": 0.022196298465132713,
1116
+ "learning_rate": 0.0002,
1117
+ "loss": 0.553594708442688,
1118
+ "mean_token_accuracy": 0.7770230323076248,
1119
+ "num_tokens": 8572656.0,
1120
+ "step": 111
1121
+ },
1122
+ {
1123
+ "entropy": 0.567478135228157,
1124
+ "epoch": 3.512,
1125
+ "grad_norm": 0.02296704798936844,
1126
+ "learning_rate": 0.0002,
1127
+ "loss": 0.5598067045211792,
1128
+ "mean_token_accuracy": 0.7746621519327164,
1129
+ "num_tokens": 8650724.0,
1130
+ "step": 112
1131
+ },
1132
+ {
1133
+ "entropy": 0.5627018958330154,
1134
+ "epoch": 3.544,
1135
+ "grad_norm": 0.020507240667939186,
1136
+ "learning_rate": 0.0002,
1137
+ "loss": 0.5585383176803589,
1138
+ "mean_token_accuracy": 0.7763334959745407,
1139
+ "num_tokens": 8729104.0,
1140
+ "step": 113
1141
+ },
1142
+ {
1143
+ "entropy": 0.5544057041406631,
1144
+ "epoch": 3.576,
1145
+ "grad_norm": 0.02342178113758564,
1146
+ "learning_rate": 0.0002,
1147
+ "loss": 0.5570175647735596,
1148
+ "mean_token_accuracy": 0.7762107402086258,
1149
+ "num_tokens": 8807747.0,
1150
+ "step": 114
1151
+ },
1152
+ {
1153
+ "entropy": 0.5425068289041519,
1154
+ "epoch": 3.608,
1155
+ "grad_norm": 0.02285587601363659,
1156
+ "learning_rate": 0.0002,
1157
+ "loss": 0.545325756072998,
1158
+ "mean_token_accuracy": 0.7807946652173996,
1159
+ "num_tokens": 8887832.0,
1160
+ "step": 115
1161
+ },
1162
+ {
1163
+ "entropy": 0.553840234875679,
1164
+ "epoch": 3.64,
1165
+ "grad_norm": 0.022142302244901657,
1166
+ "learning_rate": 0.0002,
1167
+ "loss": 0.5548170804977417,
1168
+ "mean_token_accuracy": 0.7777025252580643,
1169
+ "num_tokens": 8966925.0,
1170
+ "step": 116
1171
+ },
1172
+ {
1173
+ "entropy": 0.5666903406381607,
1174
+ "epoch": 3.672,
1175
+ "grad_norm": 0.020184515044093132,
1176
+ "learning_rate": 0.0002,
1177
+ "loss": 0.5636236667633057,
1178
+ "mean_token_accuracy": 0.7735326737165451,
1179
+ "num_tokens": 9045663.0,
1180
+ "step": 117
1181
+ },
1182
+ {
1183
+ "entropy": 0.5551957935094833,
1184
+ "epoch": 3.7039999999999997,
1185
+ "grad_norm": 0.0219353549182415,
1186
+ "learning_rate": 0.0002,
1187
+ "loss": 0.5482048988342285,
1188
+ "mean_token_accuracy": 0.7802202552556992,
1189
+ "num_tokens": 9124298.0,
1190
+ "step": 118
1191
+ },
1192
+ {
1193
+ "entropy": 0.5691462904214859,
1194
+ "epoch": 3.7359999999999998,
1195
+ "grad_norm": 0.023641400039196014,
1196
+ "learning_rate": 0.0002,
1197
+ "loss": 0.5663072466850281,
1198
+ "mean_token_accuracy": 0.7717752456665039,
1199
+ "num_tokens": 9201921.0,
1200
+ "step": 119
1201
+ },
1202
+ {
1203
+ "entropy": 0.564937487244606,
1204
+ "epoch": 3.768,
1205
+ "grad_norm": 0.02424025908112526,
1206
+ "learning_rate": 0.0002,
1207
+ "loss": 0.5656293630599976,
1208
+ "mean_token_accuracy": 0.7734099626541138,
1209
+ "num_tokens": 9279520.0,
1210
+ "step": 120
1211
+ },
1212
+ {
1213
+ "entropy": 0.543596014380455,
1214
+ "epoch": 3.8,
1215
+ "grad_norm": 0.02149128168821335,
1216
+ "learning_rate": 0.0002,
1217
+ "loss": 0.5408511757850647,
1218
+ "mean_token_accuracy": 0.7823390662670135,
1219
+ "num_tokens": 9358194.0,
1220
+ "step": 121
1221
+ },
1222
+ {
1223
+ "entropy": 0.5443567931652069,
1224
+ "epoch": 3.832,
1225
+ "grad_norm": 0.02457662858068943,
1226
+ "learning_rate": 0.0002,
1227
+ "loss": 0.5437170267105103,
1228
+ "mean_token_accuracy": 0.7806205898523331,
1229
+ "num_tokens": 9436813.0,
1230
+ "step": 122
1231
+ },
1232
+ {
1233
+ "entropy": 0.5446386188268661,
1234
+ "epoch": 3.864,
1235
+ "grad_norm": 0.023279579356312752,
1236
+ "learning_rate": 0.0002,
1237
+ "loss": 0.5444090366363525,
1238
+ "mean_token_accuracy": 0.7807519435882568,
1239
+ "num_tokens": 9516996.0,
1240
+ "step": 123
1241
+ },
1242
+ {
1243
+ "entropy": 0.5451237112283707,
1244
+ "epoch": 3.896,
1245
+ "grad_norm": 0.021046897396445274,
1246
+ "learning_rate": 0.0002,
1247
+ "loss": 0.5447065830230713,
1248
+ "mean_token_accuracy": 0.7818619459867477,
1249
+ "num_tokens": 9597382.0,
1250
+ "step": 124
1251
+ },
1252
+ {
1253
+ "entropy": 0.557760015130043,
1254
+ "epoch": 3.928,
1255
+ "grad_norm": 0.02131776139140129,
1256
+ "learning_rate": 0.0002,
1257
+ "loss": 0.5580514073371887,
1258
+ "mean_token_accuracy": 0.77658312022686,
1259
+ "num_tokens": 9677720.0,
1260
+ "step": 125
1261
+ },
1262
+ {
1263
+ "entropy": 0.544140562415123,
1264
+ "epoch": 3.96,
1265
+ "grad_norm": 0.020328767597675323,
1266
+ "learning_rate": 0.0002,
1267
+ "loss": 0.5451820492744446,
1268
+ "mean_token_accuracy": 0.7804036885499954,
1269
+ "num_tokens": 9757131.0,
1270
+ "step": 126
1271
+ },
1272
+ {
1273
+ "entropy": 0.5636927336454391,
1274
+ "epoch": 3.992,
1275
+ "grad_norm": 0.021143252030014992,
1276
+ "learning_rate": 0.0002,
1277
+ "loss": 0.5571653246879578,
1278
+ "mean_token_accuracy": 0.7772841453552246,
1279
+ "num_tokens": 9835244.0,
1280
+ "step": 127
1281
+ },
1282
+ {
1283
+ "entropy": 0.5422744750976562,
1284
+ "epoch": 4.0,
1285
+ "grad_norm": 0.0327562615275383,
1286
+ "learning_rate": 0.0002,
1287
+ "loss": 0.5362396240234375,
1288
+ "mean_token_accuracy": 0.7841192483901978,
1289
+ "num_tokens": 9855434.0,
1290
+ "step": 128
1291
+ },
1292
+ {
1293
+ "entropy": 0.5350159257650375,
1294
+ "epoch": 4.032,
1295
+ "grad_norm": 0.021142639219760895,
1296
+ "learning_rate": 0.0002,
1297
+ "loss": 0.5306050181388855,
1298
+ "mean_token_accuracy": 0.787266880273819,
1299
+ "num_tokens": 9935799.0,
1300
+ "step": 129
1301
+ },
1302
+ {
1303
+ "entropy": 0.5508315712213516,
1304
+ "epoch": 4.064,
1305
+ "grad_norm": 0.023039840161800385,
1306
+ "learning_rate": 0.0002,
1307
+ "loss": 0.5510164499282837,
1308
+ "mean_token_accuracy": 0.7783015668392181,
1309
+ "num_tokens": 10015337.0,
1310
+ "step": 130
1311
+ },
1312
+ {
1313
+ "entropy": 0.5506930351257324,
1314
+ "epoch": 4.096,
1315
+ "grad_norm": 0.022771824151277542,
1316
+ "learning_rate": 0.0002,
1317
+ "loss": 0.5542877912521362,
1318
+ "mean_token_accuracy": 0.7773886919021606,
1319
+ "num_tokens": 10092789.0,
1320
+ "step": 131
1321
+ },
1322
+ {
1323
+ "entropy": 0.5481761693954468,
1324
+ "epoch": 4.128,
1325
+ "grad_norm": 0.020041558891534805,
1326
+ "learning_rate": 0.0002,
1327
+ "loss": 0.5482645630836487,
1328
+ "mean_token_accuracy": 0.7794580906629562,
1329
+ "num_tokens": 10171363.0,
1330
+ "step": 132
1331
+ },
1332
+ {
1333
+ "entropy": 0.5571236908435822,
1334
+ "epoch": 4.16,
1335
+ "grad_norm": 0.02578168548643589,
1336
+ "learning_rate": 0.0002,
1337
+ "loss": 0.55533766746521,
1338
+ "mean_token_accuracy": 0.7765619903802872,
1339
+ "num_tokens": 10250091.0,
1340
+ "step": 133
1341
+ },
1342
+ {
1343
+ "entropy": 0.5592793822288513,
1344
+ "epoch": 4.192,
1345
+ "grad_norm": 0.02451624907553196,
1346
+ "learning_rate": 0.0002,
1347
+ "loss": 0.5530962944030762,
1348
+ "mean_token_accuracy": 0.7770255208015442,
1349
+ "num_tokens": 10328422.0,
1350
+ "step": 134
1351
+ },
1352
+ {
1353
+ "entropy": 0.563023567199707,
1354
+ "epoch": 4.224,
1355
+ "grad_norm": 0.019786424934864044,
1356
+ "learning_rate": 0.0002,
1357
+ "loss": 0.5598846673965454,
1358
+ "mean_token_accuracy": 0.7750790417194366,
1359
+ "num_tokens": 10407828.0,
1360
+ "step": 135
1361
+ },
1362
+ {
1363
+ "entropy": 0.5667447596788406,
1364
+ "epoch": 4.256,
1365
+ "grad_norm": 0.02562200278043747,
1366
+ "learning_rate": 0.0002,
1367
+ "loss": 0.5655050277709961,
1368
+ "mean_token_accuracy": 0.7730535417795181,
1369
+ "num_tokens": 10486389.0,
1370
+ "step": 136
1371
+ },
1372
+ {
1373
+ "entropy": 0.5592403262853622,
1374
+ "epoch": 4.288,
1375
+ "grad_norm": 0.02239048294723034,
1376
+ "learning_rate": 0.0002,
1377
+ "loss": 0.5616511702537537,
1378
+ "mean_token_accuracy": 0.7738818973302841,
1379
+ "num_tokens": 10565760.0,
1380
+ "step": 137
1381
+ },
1382
+ {
1383
+ "entropy": 0.5580462664365768,
1384
+ "epoch": 4.32,
1385
+ "grad_norm": 0.029557570815086365,
1386
+ "learning_rate": 0.0002,
1387
+ "loss": 0.5544324517250061,
1388
+ "mean_token_accuracy": 0.7771971821784973,
1389
+ "num_tokens": 10644777.0,
1390
+ "step": 138
1391
+ },
1392
+ {
1393
+ "entropy": 0.5683742761611938,
1394
+ "epoch": 4.352,
1395
+ "grad_norm": 0.022613991051912308,
1396
+ "learning_rate": 0.0002,
1397
+ "loss": 0.5679054260253906,
1398
+ "mean_token_accuracy": 0.7711237072944641,
1399
+ "num_tokens": 10721084.0,
1400
+ "step": 139
1401
+ },
1402
+ {
1403
+ "entropy": 0.5512652546167374,
1404
+ "epoch": 4.384,
1405
+ "grad_norm": 0.023316726088523865,
1406
+ "learning_rate": 0.0002,
1407
+ "loss": 0.5495034456253052,
1408
+ "mean_token_accuracy": 0.7789248526096344,
1409
+ "num_tokens": 10799528.0,
1410
+ "step": 140
1411
+ },
1412
+ {
1413
+ "entropy": 0.5452638119459152,
1414
+ "epoch": 4.416,
1415
+ "grad_norm": 0.023327913135290146,
1416
+ "learning_rate": 0.0002,
1417
+ "loss": 0.5428160429000854,
1418
+ "mean_token_accuracy": 0.7811004668474197,
1419
+ "num_tokens": 10878802.0,
1420
+ "step": 141
1421
+ },
1422
+ {
1423
+ "entropy": 0.5505389273166656,
1424
+ "epoch": 4.448,
1425
+ "grad_norm": 0.024172673001885414,
1426
+ "learning_rate": 0.0002,
1427
+ "loss": 0.5476949214935303,
1428
+ "mean_token_accuracy": 0.780018538236618,
1429
+ "num_tokens": 10957456.0,
1430
+ "step": 142
1431
+ },
1432
+ {
1433
+ "entropy": 0.5516746640205383,
1434
+ "epoch": 4.48,
1435
+ "grad_norm": 0.024863507598638535,
1436
+ "learning_rate": 0.0002,
1437
+ "loss": 0.5560992956161499,
1438
+ "mean_token_accuracy": 0.7761176824569702,
1439
+ "num_tokens": 11036458.0,
1440
+ "step": 143
1441
+ },
1442
+ {
1443
+ "entropy": 0.5538354814052582,
1444
+ "epoch": 4.5120000000000005,
1445
+ "grad_norm": 0.03181128948926926,
1446
+ "learning_rate": 0.0002,
1447
+ "loss": 0.557499349117279,
1448
+ "mean_token_accuracy": 0.7750093787908554,
1449
+ "num_tokens": 11114465.0,
1450
+ "step": 144
1451
+ },
1452
+ {
1453
+ "entropy": 0.5519821345806122,
1454
+ "epoch": 4.5440000000000005,
1455
+ "grad_norm": 0.022477390244603157,
1456
+ "learning_rate": 0.0002,
1457
+ "loss": 0.5489175319671631,
1458
+ "mean_token_accuracy": 0.7789605855941772,
1459
+ "num_tokens": 11193250.0,
1460
+ "step": 145
1461
+ },
1462
+ {
1463
+ "entropy": 0.5598633885383606,
1464
+ "epoch": 4.576,
1465
+ "grad_norm": 0.02509782463312149,
1466
+ "learning_rate": 0.0002,
1467
+ "loss": 0.5523159503936768,
1468
+ "mean_token_accuracy": 0.7775833010673523,
1469
+ "num_tokens": 11272848.0,
1470
+ "step": 146
1471
+ },
1472
+ {
1473
+ "entropy": 0.5443387776613235,
1474
+ "epoch": 4.608,
1475
+ "grad_norm": 0.02487492933869362,
1476
+ "learning_rate": 0.0002,
1477
+ "loss": 0.5392158031463623,
1478
+ "mean_token_accuracy": 0.7834303230047226,
1479
+ "num_tokens": 11353910.0,
1480
+ "step": 147
1481
+ },
1482
+ {
1483
+ "entropy": 0.5553220510482788,
1484
+ "epoch": 4.64,
1485
+ "grad_norm": 0.02597399242222309,
1486
+ "learning_rate": 0.0002,
1487
+ "loss": 0.5571849346160889,
1488
+ "mean_token_accuracy": 0.776119738817215,
1489
+ "num_tokens": 11430800.0,
1490
+ "step": 148
1491
+ },
1492
+ {
1493
+ "entropy": 0.5394623875617981,
1494
+ "epoch": 4.672,
1495
+ "grad_norm": 0.02306293696165085,
1496
+ "learning_rate": 0.0002,
1497
+ "loss": 0.5412307381629944,
1498
+ "mean_token_accuracy": 0.7825190275907516,
1499
+ "num_tokens": 11511445.0,
1500
+ "step": 149
1501
+ },
1502
+ {
1503
+ "entropy": 0.5512889623641968,
1504
+ "epoch": 4.704,
1505
+ "grad_norm": 0.02301475964486599,
1506
+ "learning_rate": 0.0002,
1507
+ "loss": 0.5503370761871338,
1508
+ "mean_token_accuracy": 0.7782942056655884,
1509
+ "num_tokens": 11590429.0,
1510
+ "step": 150
1511
+ },
1512
+ {
1513
+ "entropy": 0.5578732192516327,
1514
+ "epoch": 4.736,
1515
+ "grad_norm": 0.02201995998620987,
1516
+ "learning_rate": 0.0002,
1517
+ "loss": 0.5536071062088013,
1518
+ "mean_token_accuracy": 0.7773238718509674,
1519
+ "num_tokens": 11669487.0,
1520
+ "step": 151
1521
+ },
1522
+ {
1523
+ "entropy": 0.5440615564584732,
1524
+ "epoch": 4.768,
1525
+ "grad_norm": 0.02202589437365532,
1526
+ "learning_rate": 0.0002,
1527
+ "loss": 0.5411347150802612,
1528
+ "mean_token_accuracy": 0.7818420082330704,
1529
+ "num_tokens": 11750325.0,
1530
+ "step": 152
1531
+ },
1532
+ {
1533
+ "entropy": 0.5498859584331512,
1534
+ "epoch": 4.8,
1535
+ "grad_norm": 0.026776723563671112,
1536
+ "learning_rate": 0.0002,
1537
+ "loss": 0.5520839691162109,
1538
+ "mean_token_accuracy": 0.7781284302473068,
1539
+ "num_tokens": 11827703.0,
1540
+ "step": 153
1541
+ },
1542
+ {
1543
+ "entropy": 0.5513161569833755,
1544
+ "epoch": 4.832,
1545
+ "grad_norm": 0.021133486181497574,
1546
+ "learning_rate": 0.0002,
1547
+ "loss": 0.5512353181838989,
1548
+ "mean_token_accuracy": 0.7764542549848557,
1549
+ "num_tokens": 11904960.0,
1550
+ "step": 154
1551
+ },
1552
+ {
1553
+ "entropy": 0.553078830242157,
1554
+ "epoch": 4.864,
1555
+ "grad_norm": 0.025516293942928314,
1556
+ "learning_rate": 0.0002,
1557
+ "loss": 0.5475062131881714,
1558
+ "mean_token_accuracy": 0.7789602428674698,
1559
+ "num_tokens": 11985016.0,
1560
+ "step": 155
1561
+ },
1562
+ {
1563
+ "entropy": 0.5605203360319138,
1564
+ "epoch": 4.896,
1565
+ "grad_norm": 0.021059760823845863,
1566
+ "learning_rate": 0.0002,
1567
+ "loss": 0.5566121339797974,
1568
+ "mean_token_accuracy": 0.7765137851238251,
1569
+ "num_tokens": 12063054.0,
1570
+ "step": 156
1571
+ },
1572
+ {
1573
+ "entropy": 0.5402437597513199,
1574
+ "epoch": 4.928,
1575
+ "grad_norm": 0.0237311739474535,
1576
+ "learning_rate": 0.0002,
1577
+ "loss": 0.5375882387161255,
1578
+ "mean_token_accuracy": 0.7829015702009201,
1579
+ "num_tokens": 12142425.0,
1580
+ "step": 157
1581
+ },
1582
+ {
1583
+ "entropy": 0.5493930131196976,
1584
+ "epoch": 4.96,
1585
+ "grad_norm": 0.02653537131845951,
1586
+ "learning_rate": 0.0002,
1587
+ "loss": 0.5503488779067993,
1588
+ "mean_token_accuracy": 0.7782562971115112,
1589
+ "num_tokens": 12220945.0,
1590
+ "step": 158
1591
+ },
1592
+ {
1593
+ "entropy": 0.5531943738460541,
1594
+ "epoch": 4.992,
1595
+ "grad_norm": 0.02164315991103649,
1596
+ "learning_rate": 0.0002,
1597
+ "loss": 0.5544183254241943,
1598
+ "mean_token_accuracy": 0.7767967581748962,
1599
+ "num_tokens": 12298554.0,
1600
+ "step": 159
1601
+ },
1602
+ {
1603
+ "entropy": 0.5414583086967468,
1604
+ "epoch": 5.0,
1605
+ "grad_norm": 0.040175583213567734,
1606
+ "learning_rate": 0.0002,
1607
+ "loss": 0.5361344218254089,
1608
+ "mean_token_accuracy": 0.7858189940452576,
1609
+ "num_tokens": 12319146.0,
1610
+ "step": 160
1611
+ },
1612
+ {
1613
+ "entropy": 0.5582098960876465,
1614
+ "epoch": 5.032,
1615
+ "grad_norm": 0.022894199937582016,
1616
+ "learning_rate": 0.0002,
1617
+ "loss": 0.5531452894210815,
1618
+ "mean_token_accuracy": 0.7776804864406586,
1619
+ "num_tokens": 12397752.0,
1620
+ "step": 161
1621
+ },
1622
+ {
1623
+ "entropy": 0.5599130690097809,
1624
+ "epoch": 5.064,
1625
+ "grad_norm": 0.025759946554899216,
1626
+ "learning_rate": 0.0002,
1627
+ "loss": 0.5595382452011108,
1628
+ "mean_token_accuracy": 0.7729753255844116,
1629
+ "num_tokens": 12476444.0,
1630
+ "step": 162
1631
+ },
1632
+ {
1633
+ "entropy": 0.5494701564311981,
1634
+ "epoch": 5.096,
1635
+ "grad_norm": 0.027650512754917145,
1636
+ "learning_rate": 0.0002,
1637
+ "loss": 0.552753210067749,
1638
+ "mean_token_accuracy": 0.7775422781705856,
1639
+ "num_tokens": 12556559.0,
1640
+ "step": 163
1641
+ },
1642
+ {
1643
+ "entropy": 0.5431053638458252,
1644
+ "epoch": 5.128,
1645
+ "grad_norm": 0.02576346881687641,
1646
+ "learning_rate": 0.0002,
1647
+ "loss": 0.543494701385498,
1648
+ "mean_token_accuracy": 0.7810918986797333,
1649
+ "num_tokens": 12635217.0,
1650
+ "step": 164
1651
+ },
1652
+ {
1653
+ "entropy": 0.5598382949829102,
1654
+ "epoch": 5.16,
1655
+ "grad_norm": 0.022493746131658554,
1656
+ "learning_rate": 0.0002,
1657
+ "loss": 0.5572130680084229,
1658
+ "mean_token_accuracy": 0.7740350961685181,
1659
+ "num_tokens": 12713524.0,
1660
+ "step": 165
1661
+ },
1662
+ {
1663
+ "entropy": 0.545780748128891,
1664
+ "epoch": 5.192,
1665
+ "grad_norm": 0.027349475771188736,
1666
+ "learning_rate": 0.0002,
1667
+ "loss": 0.5389693379402161,
1668
+ "mean_token_accuracy": 0.7838597297668457,
1669
+ "num_tokens": 12793300.0,
1670
+ "step": 166
1671
+ },
1672
+ {
1673
+ "entropy": 0.5592958778142929,
1674
+ "epoch": 5.224,
1675
+ "grad_norm": 0.028276948258280754,
1676
+ "learning_rate": 0.0002,
1677
+ "loss": 0.5530558824539185,
1678
+ "mean_token_accuracy": 0.7787036746740341,
1679
+ "num_tokens": 12872372.0,
1680
+ "step": 167
1681
+ },
1682
+ {
1683
+ "entropy": 0.5440209805965424,
1684
+ "epoch": 5.256,
1685
+ "grad_norm": 0.029081838205456734,
1686
+ "learning_rate": 0.0002,
1687
+ "loss": 0.5472489595413208,
1688
+ "mean_token_accuracy": 0.7790472209453583,
1689
+ "num_tokens": 12950474.0,
1690
+ "step": 168
1691
+ },
1692
+ {
1693
+ "entropy": 0.541302427649498,
1694
+ "epoch": 5.288,
1695
+ "grad_norm": 0.03218315914273262,
1696
+ "learning_rate": 0.0002,
1697
+ "loss": 0.5454750657081604,
1698
+ "mean_token_accuracy": 0.7817174345254898,
1699
+ "num_tokens": 13029478.0,
1700
+ "step": 169
1701
+ },
1702
+ {
1703
+ "entropy": 0.5527262985706329,
1704
+ "epoch": 5.32,
1705
+ "grad_norm": 0.023637501522898674,
1706
+ "learning_rate": 0.0002,
1707
+ "loss": 0.5539190769195557,
1708
+ "mean_token_accuracy": 0.7765050381422043,
1709
+ "num_tokens": 13107874.0,
1710
+ "step": 170
1711
+ },
1712
+ {
1713
+ "entropy": 0.5566678494215012,
1714
+ "epoch": 5.352,
1715
+ "grad_norm": 0.030741648748517036,
1716
+ "learning_rate": 0.0002,
1717
+ "loss": 0.5524072647094727,
1718
+ "mean_token_accuracy": 0.7761627286672592,
1719
+ "num_tokens": 13187034.0,
1720
+ "step": 171
1721
+ },
1722
+ {
1723
+ "entropy": 0.5590717494487762,
1724
+ "epoch": 5.384,
1725
+ "grad_norm": 0.02631162852048874,
1726
+ "learning_rate": 0.0002,
1727
+ "loss": 0.5528249740600586,
1728
+ "mean_token_accuracy": 0.7761536091566086,
1729
+ "num_tokens": 13266614.0,
1730
+ "step": 172
1731
+ },
1732
+ {
1733
+ "entropy": 0.54710254073143,
1734
+ "epoch": 5.416,
1735
+ "grad_norm": 0.030295731499791145,
1736
+ "learning_rate": 0.0002,
1737
+ "loss": 0.5446665287017822,
1738
+ "mean_token_accuracy": 0.7795543521642685,
1739
+ "num_tokens": 13345744.0,
1740
+ "step": 173
1741
+ },
1742
+ {
1743
+ "entropy": 0.5439110398292542,
1744
+ "epoch": 5.448,
1745
+ "grad_norm": 0.028032993897795677,
1746
+ "learning_rate": 0.0002,
1747
+ "loss": 0.5464545488357544,
1748
+ "mean_token_accuracy": 0.7813926637172699,
1749
+ "num_tokens": 13424891.0,
1750
+ "step": 174
1751
+ },
1752
+ {
1753
+ "entropy": 0.545119971036911,
1754
+ "epoch": 5.48,
1755
+ "grad_norm": 0.026973431929945946,
1756
+ "learning_rate": 0.0002,
1757
+ "loss": 0.5461452007293701,
1758
+ "mean_token_accuracy": 0.7802118062973022,
1759
+ "num_tokens": 13503380.0,
1760
+ "step": 175
1761
+ },
1762
+ {
1763
+ "entropy": 0.5575177073478699,
1764
+ "epoch": 5.5120000000000005,
1765
+ "grad_norm": 0.026447977870702744,
1766
+ "learning_rate": 0.0002,
1767
+ "loss": 0.5565048456192017,
1768
+ "mean_token_accuracy": 0.7753730118274689,
1769
+ "num_tokens": 13581504.0,
1770
+ "step": 176
1771
+ },
1772
+ {
1773
+ "entropy": 0.5443647503852844,
1774
+ "epoch": 5.5440000000000005,
1775
+ "grad_norm": 0.026733124628663063,
1776
+ "learning_rate": 0.0002,
1777
+ "loss": 0.5393130779266357,
1778
+ "mean_token_accuracy": 0.7818372398614883,
1779
+ "num_tokens": 13659491.0,
1780
+ "step": 177
1781
+ },
1782
+ {
1783
+ "entropy": 0.5571713000535965,
1784
+ "epoch": 5.576,
1785
+ "grad_norm": 0.02440747246146202,
1786
+ "learning_rate": 0.0002,
1787
+ "loss": 0.5560795664787292,
1788
+ "mean_token_accuracy": 0.7749420404434204,
1789
+ "num_tokens": 13736686.0,
1790
+ "step": 178
1791
+ },
1792
+ {
1793
+ "entropy": 0.543163537979126,
1794
+ "epoch": 5.608,
1795
+ "grad_norm": 0.020597316324710846,
1796
+ "learning_rate": 0.0002,
1797
+ "loss": 0.5401595234870911,
1798
+ "mean_token_accuracy": 0.7831601202487946,
1799
+ "num_tokens": 13816007.0,
1800
+ "step": 179
1801
+ },
1802
+ {
1803
+ "entropy": 0.5275207906961441,
1804
+ "epoch": 5.64,
1805
+ "grad_norm": 0.021762844175100327,
1806
+ "learning_rate": 0.0002,
1807
+ "loss": 0.5270570516586304,
1808
+ "mean_token_accuracy": 0.7863520830869675,
1809
+ "num_tokens": 13895212.0,
1810
+ "step": 180
1811
+ },
1812
+ {
1813
+ "entropy": 0.5509206503629684,
1814
+ "epoch": 5.672,
1815
+ "grad_norm": 0.025359727442264557,
1816
+ "learning_rate": 0.0002,
1817
+ "loss": 0.5518645644187927,
1818
+ "mean_token_accuracy": 0.7776848524808884,
1819
+ "num_tokens": 13973672.0,
1820
+ "step": 181
1821
+ },
1822
+ {
1823
+ "entropy": 0.5468782633543015,
1824
+ "epoch": 5.704,
1825
+ "grad_norm": 0.023991327732801437,
1826
+ "learning_rate": 0.0002,
1827
+ "loss": 0.5474581718444824,
1828
+ "mean_token_accuracy": 0.7794180512428284,
1829
+ "num_tokens": 14052724.0,
1830
+ "step": 182
1831
+ },
1832
+ {
1833
+ "entropy": 0.5526992231607437,
1834
+ "epoch": 5.736,
1835
+ "grad_norm": 0.01987096108496189,
1836
+ "learning_rate": 0.0002,
1837
+ "loss": 0.5501133799552917,
1838
+ "mean_token_accuracy": 0.7786473035812378,
1839
+ "num_tokens": 14131904.0,
1840
+ "step": 183
1841
+ },
1842
+ {
1843
+ "entropy": 0.5467601120471954,
1844
+ "epoch": 5.768,
1845
+ "grad_norm": 0.02320663444697857,
1846
+ "learning_rate": 0.0002,
1847
+ "loss": 0.5411819815635681,
1848
+ "mean_token_accuracy": 0.7823377251625061,
1849
+ "num_tokens": 14210960.0,
1850
+ "step": 184
1851
+ },
1852
+ {
1853
+ "entropy": 0.5405216664075851,
1854
+ "epoch": 5.8,
1855
+ "grad_norm": 0.021056123077869415,
1856
+ "learning_rate": 0.0002,
1857
+ "loss": 0.5362813472747803,
1858
+ "mean_token_accuracy": 0.7848241031169891,
1859
+ "num_tokens": 14291726.0,
1860
+ "step": 185
1861
+ },
1862
+ {
1863
+ "entropy": 0.5541730970144272,
1864
+ "epoch": 5.832,
1865
+ "grad_norm": 0.02818496711552143,
1866
+ "learning_rate": 0.0002,
1867
+ "loss": 0.5557092428207397,
1868
+ "mean_token_accuracy": 0.7762422263622284,
1869
+ "num_tokens": 14369724.0,
1870
+ "step": 186
1871
+ },
1872
+ {
1873
+ "entropy": 0.5322004407644272,
1874
+ "epoch": 5.864,
1875
+ "grad_norm": 0.02371285855770111,
1876
+ "learning_rate": 0.0002,
1877
+ "loss": 0.531102180480957,
1878
+ "mean_token_accuracy": 0.7858971059322357,
1879
+ "num_tokens": 14448931.0,
1880
+ "step": 187
1881
+ },
1882
+ {
1883
+ "entropy": 0.5547012537717819,
1884
+ "epoch": 5.896,
1885
+ "grad_norm": 0.036903511732816696,
1886
+ "learning_rate": 0.0002,
1887
+ "loss": 0.5559762120246887,
1888
+ "mean_token_accuracy": 0.776378944516182,
1889
+ "num_tokens": 14526949.0,
1890
+ "step": 188
1891
+ },
1892
+ {
1893
+ "entropy": 0.5446574836969376,
1894
+ "epoch": 5.928,
1895
+ "grad_norm": 0.026452112942934036,
1896
+ "learning_rate": 0.0002,
1897
+ "loss": 0.5377969145774841,
1898
+ "mean_token_accuracy": 0.7841950803995132,
1899
+ "num_tokens": 14604335.0,
1900
+ "step": 189
1901
+ },
1902
+ {
1903
+ "entropy": 0.5493032783269882,
1904
+ "epoch": 5.96,
1905
+ "grad_norm": 0.029249282553792,
1906
+ "learning_rate": 0.0002,
1907
+ "loss": 0.5473008155822754,
1908
+ "mean_token_accuracy": 0.7784675061702728,
1909
+ "num_tokens": 14683487.0,
1910
+ "step": 190
1911
+ },
1912
+ {
1913
+ "entropy": 0.5538213402032852,
1914
+ "epoch": 5.992,
1915
+ "grad_norm": 0.025735314935445786,
1916
+ "learning_rate": 0.0002,
1917
+ "loss": 0.552562952041626,
1918
+ "mean_token_accuracy": 0.7783134877681732,
1919
+ "num_tokens": 14763050.0,
1920
+ "step": 191
1921
+ },
1922
+ {
1923
+ "entropy": 0.5568431615829468,
1924
+ "epoch": 6.0,
1925
+ "grad_norm": 0.0419342927634716,
1926
+ "learning_rate": 0.0002,
1927
+ "loss": 0.5596759915351868,
1928
+ "mean_token_accuracy": 0.7757459878921509,
1929
+ "num_tokens": 14783084.0,
1930
+ "step": 192
1931
+ },
1932
+ {
1933
+ "entropy": 0.5499615967273712,
1934
+ "epoch": 6.032,
1935
+ "grad_norm": 0.030985454097390175,
1936
+ "learning_rate": 0.0002,
1937
+ "loss": 0.5423964262008667,
1938
+ "mean_token_accuracy": 0.7811243683099747,
1939
+ "num_tokens": 14861223.0,
1940
+ "step": 193
1941
+ },
1942
+ {
1943
+ "entropy": 0.545973002910614,
1944
+ "epoch": 6.064,
1945
+ "grad_norm": 0.03047715686261654,
1946
+ "learning_rate": 0.0002,
1947
+ "loss": 0.5429363250732422,
1948
+ "mean_token_accuracy": 0.7807959467172623,
1949
+ "num_tokens": 14939948.0,
1950
+ "step": 194
1951
+ },
1952
+ {
1953
+ "entropy": 0.5407204329967499,
1954
+ "epoch": 6.096,
1955
+ "grad_norm": 0.026628416031599045,
1956
+ "learning_rate": 0.0002,
1957
+ "loss": 0.5426843166351318,
1958
+ "mean_token_accuracy": 0.7809197157621384,
1959
+ "num_tokens": 15018808.0,
1960
+ "step": 195
1961
+ },
1962
+ {
1963
+ "entropy": 0.5479950308799744,
1964
+ "epoch": 6.128,
1965
+ "grad_norm": 0.03580361232161522,
1966
+ "learning_rate": 0.0002,
1967
+ "loss": 0.5534794926643372,
1968
+ "mean_token_accuracy": 0.777429461479187,
1969
+ "num_tokens": 15096870.0,
1970
+ "step": 196
1971
+ },
1972
+ {
1973
+ "entropy": 0.5385048538446426,
1974
+ "epoch": 6.16,
1975
+ "grad_norm": 0.023637505248188972,
1976
+ "learning_rate": 0.0002,
1977
+ "loss": 0.5380685329437256,
1978
+ "mean_token_accuracy": 0.783260390162468,
1979
+ "num_tokens": 15174757.0,
1980
+ "step": 197
1981
+ },
1982
+ {
1983
+ "entropy": 0.5591510534286499,
1984
+ "epoch": 6.192,
1985
+ "grad_norm": 0.03232690691947937,
1986
+ "learning_rate": 0.0002,
1987
+ "loss": 0.5529967546463013,
1988
+ "mean_token_accuracy": 0.7766391485929489,
1989
+ "num_tokens": 15252758.0,
1990
+ "step": 198
1991
+ },
1992
+ {
1993
+ "entropy": 0.5531331598758698,
1994
+ "epoch": 6.224,
1995
+ "grad_norm": 0.027761192992329597,
1996
+ "learning_rate": 0.0002,
1997
+ "loss": 0.5476256608963013,
1998
+ "mean_token_accuracy": 0.7792367190122604,
1999
+ "num_tokens": 15330164.0,
2000
+ "step": 199
2001
+ },
2002
+ {
2003
+ "entropy": 0.5431746989488602,
2004
+ "epoch": 6.256,
2005
+ "grad_norm": 0.029444098472595215,
2006
+ "learning_rate": 0.0002,
2007
+ "loss": 0.5440195798873901,
2008
+ "mean_token_accuracy": 0.7816928923130035,
2009
+ "num_tokens": 15408291.0,
2010
+ "step": 200
2011
+ },
2012
+ {
2013
+ "entropy": 0.5401950180530548,
2014
+ "epoch": 6.288,
2015
+ "grad_norm": 0.030943166464567184,
2016
+ "learning_rate": 0.0002,
2017
+ "loss": 0.5441604852676392,
2018
+ "mean_token_accuracy": 0.7797262668609619,
2019
+ "num_tokens": 15486855.0,
2020
+ "step": 201
2021
+ },
2022
+ {
2023
+ "entropy": 0.5358632057905197,
2024
+ "epoch": 6.32,
2025
+ "grad_norm": 0.025611277669668198,
2026
+ "learning_rate": 0.0002,
2027
+ "loss": 0.5336735844612122,
2028
+ "mean_token_accuracy": 0.7844033241271973,
2029
+ "num_tokens": 15565945.0,
2030
+ "step": 202
2031
+ },
2032
+ {
2033
+ "entropy": 0.5415282398462296,
2034
+ "epoch": 6.352,
2035
+ "grad_norm": 0.03050980716943741,
2036
+ "learning_rate": 0.0002,
2037
+ "loss": 0.5411715507507324,
2038
+ "mean_token_accuracy": 0.7807140350341797,
2039
+ "num_tokens": 15644120.0,
2040
+ "step": 203
2041
+ },
2042
+ {
2043
+ "entropy": 0.5550639480352402,
2044
+ "epoch": 6.384,
2045
+ "grad_norm": 0.025711113587021828,
2046
+ "learning_rate": 0.0002,
2047
+ "loss": 0.5523407459259033,
2048
+ "mean_token_accuracy": 0.7759165465831757,
2049
+ "num_tokens": 15721932.0,
2050
+ "step": 204
2051
+ },
2052
+ {
2053
+ "entropy": 0.5459660887718201,
2054
+ "epoch": 6.416,
2055
+ "grad_norm": 0.028625324368476868,
2056
+ "learning_rate": 0.0002,
2057
+ "loss": 0.5404061675071716,
2058
+ "mean_token_accuracy": 0.7824661731719971,
2059
+ "num_tokens": 15801402.0,
2060
+ "step": 205
2061
+ },
2062
+ {
2063
+ "entropy": 0.5531899780035019,
2064
+ "epoch": 6.448,
2065
+ "grad_norm": 0.026339178904891014,
2066
+ "learning_rate": 0.0002,
2067
+ "loss": 0.5514405369758606,
2068
+ "mean_token_accuracy": 0.7779718488454819,
2069
+ "num_tokens": 15880514.0,
2070
+ "step": 206
2071
+ },
2072
+ {
2073
+ "entropy": 0.5400968790054321,
2074
+ "epoch": 6.48,
2075
+ "grad_norm": 0.030028901994228363,
2076
+ "learning_rate": 0.0002,
2077
+ "loss": 0.5434284210205078,
2078
+ "mean_token_accuracy": 0.7791207730770111,
2079
+ "num_tokens": 15959144.0,
2080
+ "step": 207
2081
+ },
2082
+ {
2083
+ "entropy": 0.5307324081659317,
2084
+ "epoch": 6.5120000000000005,
2085
+ "grad_norm": 0.02786673791706562,
2086
+ "learning_rate": 0.0002,
2087
+ "loss": 0.5343931317329407,
2088
+ "mean_token_accuracy": 0.7839188724756241,
2089
+ "num_tokens": 16039776.0,
2090
+ "step": 208
2091
+ },
2092
+ {
2093
+ "entropy": 0.5425277054309845,
2094
+ "epoch": 6.5440000000000005,
2095
+ "grad_norm": 0.02753874473273754,
2096
+ "learning_rate": 0.0002,
2097
+ "loss": 0.5446346998214722,
2098
+ "mean_token_accuracy": 0.7797478884458542,
2099
+ "num_tokens": 16118884.0,
2100
+ "step": 209
2101
+ },
2102
+ {
2103
+ "entropy": 0.5513077974319458,
2104
+ "epoch": 6.576,
2105
+ "grad_norm": 0.028762485831975937,
2106
+ "learning_rate": 0.0002,
2107
+ "loss": 0.5451760292053223,
2108
+ "mean_token_accuracy": 0.7815571874380112,
2109
+ "num_tokens": 16197259.0,
2110
+ "step": 210
2111
+ },
2112
+ {
2113
+ "entropy": 0.560958668589592,
2114
+ "epoch": 6.608,
2115
+ "grad_norm": 0.02965443953871727,
2116
+ "learning_rate": 0.0002,
2117
+ "loss": 0.5526363849639893,
2118
+ "mean_token_accuracy": 0.7772418111562729,
2119
+ "num_tokens": 16276149.0,
2120
+ "step": 211
2121
+ },
2122
+ {
2123
+ "entropy": 0.537281408905983,
2124
+ "epoch": 6.64,
2125
+ "grad_norm": 0.023833326995372772,
2126
+ "learning_rate": 0.0002,
2127
+ "loss": 0.532819390296936,
2128
+ "mean_token_accuracy": 0.7847591936588287,
2129
+ "num_tokens": 16356969.0,
2130
+ "step": 212
2131
+ },
2132
+ {
2133
+ "entropy": 0.5362367182970047,
2134
+ "epoch": 6.672,
2135
+ "grad_norm": 0.02524951472878456,
2136
+ "learning_rate": 0.0002,
2137
+ "loss": 0.5348524451255798,
2138
+ "mean_token_accuracy": 0.7850412726402283,
2139
+ "num_tokens": 16436635.0,
2140
+ "step": 213
2141
+ },
2142
+ {
2143
+ "entropy": 0.5361433327198029,
2144
+ "epoch": 6.704,
2145
+ "grad_norm": 0.03111296147108078,
2146
+ "learning_rate": 0.0002,
2147
+ "loss": 0.5393766164779663,
2148
+ "mean_token_accuracy": 0.7813311666250229,
2149
+ "num_tokens": 16515650.0,
2150
+ "step": 214
2151
+ },
2152
+ {
2153
+ "entropy": 0.5386303067207336,
2154
+ "epoch": 6.736,
2155
+ "grad_norm": 0.026461219415068626,
2156
+ "learning_rate": 0.0002,
2157
+ "loss": 0.5373357534408569,
2158
+ "mean_token_accuracy": 0.7834987640380859,
2159
+ "num_tokens": 16595744.0,
2160
+ "step": 215
2161
+ },
2162
+ {
2163
+ "entropy": 0.5392462760210037,
2164
+ "epoch": 6.768,
2165
+ "grad_norm": 0.027499942108988762,
2166
+ "learning_rate": 0.0002,
2167
+ "loss": 0.5331963300704956,
2168
+ "mean_token_accuracy": 0.7848135083913803,
2169
+ "num_tokens": 16674509.0,
2170
+ "step": 216
2171
+ },
2172
+ {
2173
+ "entropy": 0.5459368675947189,
2174
+ "epoch": 6.8,
2175
+ "grad_norm": 0.024176757782697678,
2176
+ "learning_rate": 0.0002,
2177
+ "loss": 0.5415037870407104,
2178
+ "mean_token_accuracy": 0.7818843722343445,
2179
+ "num_tokens": 16754105.0,
2180
+ "step": 217
2181
+ },
2182
+ {
2183
+ "entropy": 0.5565692782402039,
2184
+ "epoch": 6.832,
2185
+ "grad_norm": 0.02574721910059452,
2186
+ "learning_rate": 0.0002,
2187
+ "loss": 0.5573182106018066,
2188
+ "mean_token_accuracy": 0.7748913615942001,
2189
+ "num_tokens": 16833119.0,
2190
+ "step": 218
2191
+ },
2192
+ {
2193
+ "entropy": 0.5434060543775558,
2194
+ "epoch": 6.864,
2195
+ "grad_norm": 0.02367129735648632,
2196
+ "learning_rate": 0.0002,
2197
+ "loss": 0.5418709516525269,
2198
+ "mean_token_accuracy": 0.7805542647838593,
2199
+ "num_tokens": 16913039.0,
2200
+ "step": 219
2201
+ },
2202
+ {
2203
+ "entropy": 0.5493430346250534,
2204
+ "epoch": 6.896,
2205
+ "grad_norm": 0.02657475695014,
2206
+ "learning_rate": 0.0002,
2207
+ "loss": 0.547214686870575,
2208
+ "mean_token_accuracy": 0.7784837335348129,
2209
+ "num_tokens": 16991051.0,
2210
+ "step": 220
2211
+ },
2212
+ {
2213
+ "entropy": 0.5528377145528793,
2214
+ "epoch": 6.928,
2215
+ "grad_norm": 0.024269506335258484,
2216
+ "learning_rate": 0.0002,
2217
+ "loss": 0.5508480668067932,
2218
+ "mean_token_accuracy": 0.7769178003072739,
2219
+ "num_tokens": 17069694.0,
2220
+ "step": 221
2221
+ },
2222
+ {
2223
+ "entropy": 0.5454904437065125,
2224
+ "epoch": 6.96,
2225
+ "grad_norm": 0.024885021150112152,
2226
+ "learning_rate": 0.0002,
2227
+ "loss": 0.5465965270996094,
2228
+ "mean_token_accuracy": 0.7797489613294601,
2229
+ "num_tokens": 17147632.0,
2230
+ "step": 222
2231
+ },
2232
+ {
2233
+ "entropy": 0.5398757457733154,
2234
+ "epoch": 6.992,
2235
+ "grad_norm": 0.02908923290669918,
2236
+ "learning_rate": 0.0002,
2237
+ "loss": 0.5431025624275208,
2238
+ "mean_token_accuracy": 0.779990091919899,
2239
+ "num_tokens": 17227457.0,
2240
+ "step": 223
2241
+ },
2242
+ {
2243
+ "entropy": 0.5505363941192627,
2244
+ "epoch": 7.0,
2245
+ "grad_norm": 0.03793865069746971,
2246
+ "learning_rate": 0.0002,
2247
+ "loss": 0.5468761324882507,
2248
+ "mean_token_accuracy": 0.7777085304260254,
2249
+ "num_tokens": 17246839.0,
2250
+ "step": 224
2251
+ },
2252
+ {
2253
+ "entropy": 0.5342252403497696,
2254
+ "epoch": 7.032,
2255
+ "grad_norm": 0.03273385763168335,
2256
+ "learning_rate": 0.0002,
2257
+ "loss": 0.5247677564620972,
2258
+ "mean_token_accuracy": 0.7881321907043457,
2259
+ "num_tokens": 17326090.0,
2260
+ "step": 225
2261
+ },
2262
+ {
2263
+ "entropy": 0.5373009145259857,
2264
+ "epoch": 7.064,
2265
+ "grad_norm": 0.025371287018060684,
2266
+ "learning_rate": 0.0002,
2267
+ "loss": 0.5323753952980042,
2268
+ "mean_token_accuracy": 0.7845641523599625,
2269
+ "num_tokens": 17405594.0,
2270
+ "step": 226
2271
+ },
2272
+ {
2273
+ "entropy": 0.5337796360254288,
2274
+ "epoch": 7.096,
2275
+ "grad_norm": 0.03891294077038765,
2276
+ "learning_rate": 0.0002,
2277
+ "loss": 0.5395870208740234,
2278
+ "mean_token_accuracy": 0.7812530696392059,
2279
+ "num_tokens": 17484846.0,
2280
+ "step": 227
2281
+ },
2282
+ {
2283
+ "entropy": 0.5363831967115402,
2284
+ "epoch": 7.128,
2285
+ "grad_norm": 0.03025059774518013,
2286
+ "learning_rate": 0.0002,
2287
+ "loss": 0.538554847240448,
2288
+ "mean_token_accuracy": 0.7826266288757324,
2289
+ "num_tokens": 17563627.0,
2290
+ "step": 228
2291
+ },
2292
+ {
2293
+ "entropy": 0.5457527041435242,
2294
+ "epoch": 7.16,
2295
+ "grad_norm": 0.027366969734430313,
2296
+ "learning_rate": 0.0002,
2297
+ "loss": 0.5434131622314453,
2298
+ "mean_token_accuracy": 0.7804773300886154,
2299
+ "num_tokens": 17642515.0,
2300
+ "step": 229
2301
+ },
2302
+ {
2303
+ "entropy": 0.5363115072250366,
2304
+ "epoch": 7.192,
2305
+ "grad_norm": 0.03141327574849129,
2306
+ "learning_rate": 0.0002,
2307
+ "loss": 0.5331645607948303,
2308
+ "mean_token_accuracy": 0.7852070778608322,
2309
+ "num_tokens": 17721303.0,
2310
+ "step": 230
2311
+ },
2312
+ {
2313
+ "entropy": 0.5441312342882156,
2314
+ "epoch": 7.224,
2315
+ "grad_norm": 0.02731841430068016,
2316
+ "learning_rate": 0.0002,
2317
+ "loss": 0.5440965890884399,
2318
+ "mean_token_accuracy": 0.7794661670923233,
2319
+ "num_tokens": 17799159.0,
2320
+ "step": 231
2321
+ },
2322
+ {
2323
+ "entropy": 0.5447894930839539,
2324
+ "epoch": 7.256,
2325
+ "grad_norm": 0.03376320004463196,
2326
+ "learning_rate": 0.0002,
2327
+ "loss": 0.543010950088501,
2328
+ "mean_token_accuracy": 0.7805230468511581,
2329
+ "num_tokens": 17877298.0,
2330
+ "step": 232
2331
+ },
2332
+ {
2333
+ "entropy": 0.5362501293420792,
2334
+ "epoch": 7.288,
2335
+ "grad_norm": 0.03332136943936348,
2336
+ "learning_rate": 0.0002,
2337
+ "loss": 0.5391461253166199,
2338
+ "mean_token_accuracy": 0.7822219878435135,
2339
+ "num_tokens": 17955795.0,
2340
+ "step": 233
2341
+ },
2342
+ {
2343
+ "entropy": 0.5330929756164551,
2344
+ "epoch": 7.32,
2345
+ "grad_norm": 0.029867252334952354,
2346
+ "learning_rate": 0.0002,
2347
+ "loss": 0.5306733846664429,
2348
+ "mean_token_accuracy": 0.7851523607969284,
2349
+ "num_tokens": 18035708.0,
2350
+ "step": 234
2351
+ },
2352
+ {
2353
+ "entropy": 0.54389688372612,
2354
+ "epoch": 7.352,
2355
+ "grad_norm": 0.03905443847179413,
2356
+ "learning_rate": 0.0002,
2357
+ "loss": 0.541580080986023,
2358
+ "mean_token_accuracy": 0.7805737406015396,
2359
+ "num_tokens": 18112883.0,
2360
+ "step": 235
2361
+ },
2362
+ {
2363
+ "entropy": 0.5383184850215912,
2364
+ "epoch": 7.384,
2365
+ "grad_norm": 0.02882959507405758,
2366
+ "learning_rate": 0.0002,
2367
+ "loss": 0.538108766078949,
2368
+ "mean_token_accuracy": 0.7825261801481247,
2369
+ "num_tokens": 18190190.0,
2370
+ "step": 236
2371
+ },
2372
+ {
2373
+ "entropy": 0.5316631495952606,
2374
+ "epoch": 7.416,
2375
+ "grad_norm": 0.03594264015555382,
2376
+ "learning_rate": 0.0002,
2377
+ "loss": 0.5299159288406372,
2378
+ "mean_token_accuracy": 0.7848720252513885,
2379
+ "num_tokens": 18269276.0,
2380
+ "step": 237
2381
+ },
2382
+ {
2383
+ "entropy": 0.5474216341972351,
2384
+ "epoch": 7.448,
2385
+ "grad_norm": 0.030706387013196945,
2386
+ "learning_rate": 0.0002,
2387
+ "loss": 0.5446693897247314,
2388
+ "mean_token_accuracy": 0.7793966680765152,
2389
+ "num_tokens": 18346679.0,
2390
+ "step": 238
2391
+ },
2392
+ {
2393
+ "entropy": 0.5482395142316818,
2394
+ "epoch": 7.48,
2395
+ "grad_norm": 0.03116007335484028,
2396
+ "learning_rate": 0.0002,
2397
+ "loss": 0.5429675579071045,
2398
+ "mean_token_accuracy": 0.7807913720607758,
2399
+ "num_tokens": 18425406.0,
2400
+ "step": 239
2401
+ },
2402
+ {
2403
+ "entropy": 0.5305105894804001,
2404
+ "epoch": 7.5120000000000005,
2405
+ "grad_norm": 0.03157881647348404,
2406
+ "learning_rate": 0.0002,
2407
+ "loss": 0.5327313542366028,
2408
+ "mean_token_accuracy": 0.7842520177364349,
2409
+ "num_tokens": 18505671.0,
2410
+ "step": 240
2411
+ },
2412
+ {
2413
+ "entropy": 0.5307354480028152,
2414
+ "epoch": 7.5440000000000005,
2415
+ "grad_norm": 0.028445513918995857,
2416
+ "learning_rate": 0.0002,
2417
+ "loss": 0.5331681966781616,
2418
+ "mean_token_accuracy": 0.784986600279808,
2419
+ "num_tokens": 18585516.0,
2420
+ "step": 241
2421
+ },
2422
+ {
2423
+ "entropy": 0.5455832630395889,
2424
+ "epoch": 7.576,
2425
+ "grad_norm": 0.04114306718111038,
2426
+ "learning_rate": 0.0002,
2427
+ "loss": 0.5481559038162231,
2428
+ "mean_token_accuracy": 0.7785380184650421,
2429
+ "num_tokens": 18663248.0,
2430
+ "step": 242
2431
+ },
2432
+ {
2433
+ "entropy": 0.5406046807765961,
2434
+ "epoch": 7.608,
2435
+ "grad_norm": 0.033326584845781326,
2436
+ "learning_rate": 0.0002,
2437
+ "loss": 0.5353219509124756,
2438
+ "mean_token_accuracy": 0.7833963334560394,
2439
+ "num_tokens": 18743695.0,
2440
+ "step": 243
2441
+ },
2442
+ {
2443
+ "entropy": 0.5466004610061646,
2444
+ "epoch": 7.64,
2445
+ "grad_norm": 0.03053242340683937,
2446
+ "learning_rate": 0.0002,
2447
+ "loss": 0.538567066192627,
2448
+ "mean_token_accuracy": 0.7825069427490234,
2449
+ "num_tokens": 18823876.0,
2450
+ "step": 244
2451
+ },
2452
+ {
2453
+ "entropy": 0.5553711205720901,
2454
+ "epoch": 7.672,
2455
+ "grad_norm": 0.03428297117352486,
2456
+ "learning_rate": 0.0002,
2457
+ "loss": 0.5528771281242371,
2458
+ "mean_token_accuracy": 0.7763674706220627,
2459
+ "num_tokens": 18901081.0,
2460
+ "step": 245
2461
+ },
2462
+ {
2463
+ "entropy": 0.5412455052137375,
2464
+ "epoch": 7.704,
2465
+ "grad_norm": 0.03315138816833496,
2466
+ "learning_rate": 0.0002,
2467
+ "loss": 0.5472655296325684,
2468
+ "mean_token_accuracy": 0.7775164395570755,
2469
+ "num_tokens": 18979461.0,
2470
+ "step": 246
2471
+ },
2472
+ {
2473
+ "entropy": 0.5280198305845261,
2474
+ "epoch": 7.736,
2475
+ "grad_norm": 0.035438407212495804,
2476
+ "learning_rate": 0.0002,
2477
+ "loss": 0.5257879495620728,
2478
+ "mean_token_accuracy": 0.787413626909256,
2479
+ "num_tokens": 19058324.0,
2480
+ "step": 247
2481
+ },
2482
+ {
2483
+ "entropy": 0.5491799712181091,
2484
+ "epoch": 7.768,
2485
+ "grad_norm": 0.03031354583799839,
2486
+ "learning_rate": 0.0002,
2487
+ "loss": 0.5499848127365112,
2488
+ "mean_token_accuracy": 0.7769055813550949,
2489
+ "num_tokens": 19136811.0,
2490
+ "step": 248
2491
+ },
2492
+ {
2493
+ "entropy": 0.5516806840896606,
2494
+ "epoch": 7.8,
2495
+ "grad_norm": 0.03401699662208557,
2496
+ "learning_rate": 0.0002,
2497
+ "loss": 0.5487878322601318,
2498
+ "mean_token_accuracy": 0.7770615667104721,
2499
+ "num_tokens": 19215642.0,
2500
+ "step": 249
2501
+ },
2502
+ {
2503
+ "entropy": 0.5209604501724243,
2504
+ "epoch": 7.832,
2505
+ "grad_norm": 0.034633055329322815,
2506
+ "learning_rate": 0.0002,
2507
+ "loss": 0.5194536447525024,
2508
+ "mean_token_accuracy": 0.7909128665924072,
2509
+ "num_tokens": 19296935.0,
2510
+ "step": 250
2511
+ },
2512
+ {
2513
+ "entropy": 0.5412441790103912,
2514
+ "epoch": 7.864,
2515
+ "grad_norm": 0.03364886716008186,
2516
+ "learning_rate": 0.0002,
2517
+ "loss": 0.5379568934440613,
2518
+ "mean_token_accuracy": 0.7831831127405167,
2519
+ "num_tokens": 19374745.0,
2520
+ "step": 251
2521
+ },
2522
+ {
2523
+ "entropy": 0.5517376810312271,
2524
+ "epoch": 7.896,
2525
+ "grad_norm": 0.026521844789385796,
2526
+ "learning_rate": 0.0002,
2527
+ "loss": 0.5488777756690979,
2528
+ "mean_token_accuracy": 0.7795552015304565,
2529
+ "num_tokens": 19453532.0,
2530
+ "step": 252
2531
+ },
2532
+ {
2533
+ "entropy": 0.5431555360555649,
2534
+ "epoch": 7.928,
2535
+ "grad_norm": 0.036673326045274734,
2536
+ "learning_rate": 0.0002,
2537
+ "loss": 0.5410053730010986,
2538
+ "mean_token_accuracy": 0.7819114327430725,
2539
+ "num_tokens": 19532321.0,
2540
+ "step": 253
2541
+ },
2542
+ {
2543
+ "entropy": 0.5400401204824448,
2544
+ "epoch": 7.96,
2545
+ "grad_norm": 0.03132104128599167,
2546
+ "learning_rate": 0.0002,
2547
+ "loss": 0.5396822690963745,
2548
+ "mean_token_accuracy": 0.7816772311925888,
2549
+ "num_tokens": 19611194.0,
2550
+ "step": 254
2551
+ },
2552
+ {
2553
+ "entropy": 0.5364492535591125,
2554
+ "epoch": 7.992,
2555
+ "grad_norm": 0.03492264077067375,
2556
+ "learning_rate": 0.0002,
2557
+ "loss": 0.5373172163963318,
2558
+ "mean_token_accuracy": 0.7828410267829895,
2559
+ "num_tokens": 19690556.0,
2560
+ "step": 255
2561
+ },
2562
+ {
2563
+ "entropy": 0.5176465511322021,
2564
+ "epoch": 8.0,
2565
+ "grad_norm": 0.04740741103887558,
2566
+ "learning_rate": 0.0002,
2567
+ "loss": 0.5141822695732117,
2568
+ "mean_token_accuracy": 0.7935296297073364,
2569
+ "num_tokens": 19710992.0,
2570
+ "step": 256
2571
+ },
2572
+ {
2573
+ "entropy": 0.5462612360715866,
2574
+ "epoch": 8.032,
2575
+ "grad_norm": 0.033493831753730774,
2576
+ "learning_rate": 0.0002,
2577
+ "loss": 0.5399432182312012,
2578
+ "mean_token_accuracy": 0.7800467163324356,
2579
+ "num_tokens": 19789675.0,
2580
+ "step": 257
2581
+ },
2582
+ {
2583
+ "entropy": 0.5363253057003021,
2584
+ "epoch": 8.064,
2585
+ "grad_norm": 0.03170062601566315,
2586
+ "learning_rate": 0.0002,
2587
+ "loss": 0.5308842658996582,
2588
+ "mean_token_accuracy": 0.7855557650327682,
2589
+ "num_tokens": 19867187.0,
2590
+ "step": 258
2591
+ },
2592
+ {
2593
+ "entropy": 0.5275291204452515,
2594
+ "epoch": 8.096,
2595
+ "grad_norm": 0.04521121829748154,
2596
+ "learning_rate": 0.0002,
2597
+ "loss": 0.5327388644218445,
2598
+ "mean_token_accuracy": 0.7848894149065018,
2599
+ "num_tokens": 19945957.0,
2600
+ "step": 259
2601
+ },
2602
+ {
2603
+ "entropy": 0.5200865566730499,
2604
+ "epoch": 8.128,
2605
+ "grad_norm": 0.029792284592986107,
2606
+ "learning_rate": 0.0002,
2607
+ "loss": 0.5194977521896362,
2608
+ "mean_token_accuracy": 0.7896896749734879,
2609
+ "num_tokens": 20026028.0,
2610
+ "step": 260
2611
+ },
2612
+ {
2613
+ "entropy": 0.5372072905302048,
2614
+ "epoch": 8.16,
2615
+ "grad_norm": 0.03705461695790291,
2616
+ "learning_rate": 0.0002,
2617
+ "loss": 0.5331783890724182,
2618
+ "mean_token_accuracy": 0.7831152677536011,
2619
+ "num_tokens": 20104507.0,
2620
+ "step": 261
2621
+ },
2622
+ {
2623
+ "entropy": 0.547843337059021,
2624
+ "epoch": 8.192,
2625
+ "grad_norm": 0.04235028475522995,
2626
+ "learning_rate": 0.0002,
2627
+ "loss": 0.5423978567123413,
2628
+ "mean_token_accuracy": 0.7805410325527191,
2629
+ "num_tokens": 20181982.0,
2630
+ "step": 262
2631
+ },
2632
+ {
2633
+ "entropy": 0.5300101935863495,
2634
+ "epoch": 8.224,
2635
+ "grad_norm": 0.040369223803281784,
2636
+ "learning_rate": 0.0002,
2637
+ "loss": 0.5269775390625,
2638
+ "mean_token_accuracy": 0.786443218588829,
2639
+ "num_tokens": 20260502.0,
2640
+ "step": 263
2641
+ },
2642
+ {
2643
+ "entropy": 0.5322915017604828,
2644
+ "epoch": 8.256,
2645
+ "grad_norm": 0.03680511191487312,
2646
+ "learning_rate": 0.0002,
2647
+ "loss": 0.5325385332107544,
2648
+ "mean_token_accuracy": 0.7842995822429657,
2649
+ "num_tokens": 20338315.0,
2650
+ "step": 264
2651
+ },
2652
+ {
2653
+ "entropy": 0.5264201015233994,
2654
+ "epoch": 8.288,
2655
+ "grad_norm": 0.033956896513700485,
2656
+ "learning_rate": 0.0002,
2657
+ "loss": 0.5283188819885254,
2658
+ "mean_token_accuracy": 0.7866097986698151,
2659
+ "num_tokens": 20418550.0,
2660
+ "step": 265
2661
+ },
2662
+ {
2663
+ "entropy": 0.5397807955741882,
2664
+ "epoch": 8.32,
2665
+ "grad_norm": 0.036973804235458374,
2666
+ "learning_rate": 0.0002,
2667
+ "loss": 0.5362756848335266,
2668
+ "mean_token_accuracy": 0.7828366905450821,
2669
+ "num_tokens": 20496912.0,
2670
+ "step": 266
2671
+ },
2672
+ {
2673
+ "entropy": 0.5398755520582199,
2674
+ "epoch": 8.352,
2675
+ "grad_norm": 0.03347233682870865,
2676
+ "learning_rate": 0.0002,
2677
+ "loss": 0.5393750667572021,
2678
+ "mean_token_accuracy": 0.7830456793308258,
2679
+ "num_tokens": 20575356.0,
2680
+ "step": 267
2681
+ },
2682
+ {
2683
+ "entropy": 0.5324622094631195,
2684
+ "epoch": 8.384,
2685
+ "grad_norm": 0.03724009171128273,
2686
+ "learning_rate": 0.0002,
2687
+ "loss": 0.5334412455558777,
2688
+ "mean_token_accuracy": 0.7843203842639923,
2689
+ "num_tokens": 20654961.0,
2690
+ "step": 268
2691
+ },
2692
+ {
2693
+ "entropy": 0.5378891676664352,
2694
+ "epoch": 8.416,
2695
+ "grad_norm": 0.0343894436955452,
2696
+ "learning_rate": 0.0002,
2697
+ "loss": 0.5370844602584839,
2698
+ "mean_token_accuracy": 0.7837714999914169,
2699
+ "num_tokens": 20734231.0,
2700
+ "step": 269
2701
+ },
2702
+ {
2703
+ "entropy": 0.5237532556056976,
2704
+ "epoch": 8.448,
2705
+ "grad_norm": 0.03481991961598396,
2706
+ "learning_rate": 0.0002,
2707
+ "loss": 0.5227078199386597,
2708
+ "mean_token_accuracy": 0.7883624732494354,
2709
+ "num_tokens": 20812680.0,
2710
+ "step": 270
2711
+ },
2712
+ {
2713
+ "entropy": 0.5372126400470734,
2714
+ "epoch": 8.48,
2715
+ "grad_norm": 0.0352364219725132,
2716
+ "learning_rate": 0.0002,
2717
+ "loss": 0.5328319668769836,
2718
+ "mean_token_accuracy": 0.7841216921806335,
2719
+ "num_tokens": 20890719.0,
2720
+ "step": 271
2721
+ },
2722
+ {
2723
+ "entropy": 0.537587359547615,
2724
+ "epoch": 8.512,
2725
+ "grad_norm": 0.04216265305876732,
2726
+ "learning_rate": 0.0002,
2727
+ "loss": 0.536990761756897,
2728
+ "mean_token_accuracy": 0.7831110805273056,
2729
+ "num_tokens": 20969962.0,
2730
+ "step": 272
2731
+ },
2732
+ {
2733
+ "entropy": 0.5395336449146271,
2734
+ "epoch": 8.544,
2735
+ "grad_norm": 0.03775520622730255,
2736
+ "learning_rate": 0.0002,
2737
+ "loss": 0.5417099595069885,
2738
+ "mean_token_accuracy": 0.7797977179288864,
2739
+ "num_tokens": 21048464.0,
2740
+ "step": 273
2741
+ },
2742
+ {
2743
+ "entropy": 0.5484501123428345,
2744
+ "epoch": 8.576,
2745
+ "grad_norm": 0.03163311630487442,
2746
+ "learning_rate": 0.0002,
2747
+ "loss": 0.5443800687789917,
2748
+ "mean_token_accuracy": 0.7803587913513184,
2749
+ "num_tokens": 21125761.0,
2750
+ "step": 274
2751
+ },
2752
+ {
2753
+ "entropy": 0.5378590673208237,
2754
+ "epoch": 8.608,
2755
+ "grad_norm": 0.03819214925169945,
2756
+ "learning_rate": 0.0002,
2757
+ "loss": 0.5338118672370911,
2758
+ "mean_token_accuracy": 0.7852731049060822,
2759
+ "num_tokens": 21204791.0,
2760
+ "step": 275
2761
+ },
2762
+ {
2763
+ "entropy": 0.5275268107652664,
2764
+ "epoch": 8.64,
2765
+ "grad_norm": 0.03613068163394928,
2766
+ "learning_rate": 0.0002,
2767
+ "loss": 0.530475378036499,
2768
+ "mean_token_accuracy": 0.7848686426877975,
2769
+ "num_tokens": 21282818.0,
2770
+ "step": 276
2771
+ },
2772
+ {
2773
+ "entropy": 0.5271630138158798,
2774
+ "epoch": 8.672,
2775
+ "grad_norm": 0.032881688326597214,
2776
+ "learning_rate": 0.0002,
2777
+ "loss": 0.5283002257347107,
2778
+ "mean_token_accuracy": 0.7857704162597656,
2779
+ "num_tokens": 21363284.0,
2780
+ "step": 277
2781
+ },
2782
+ {
2783
+ "entropy": 0.5359350591897964,
2784
+ "epoch": 8.704,
2785
+ "grad_norm": 0.037707071751356125,
2786
+ "learning_rate": 0.0002,
2787
+ "loss": 0.5357704162597656,
2788
+ "mean_token_accuracy": 0.783053383231163,
2789
+ "num_tokens": 21442878.0,
2790
+ "step": 278
2791
+ },
2792
+ {
2793
+ "entropy": 0.5460938811302185,
2794
+ "epoch": 8.736,
2795
+ "grad_norm": 0.033162783831357956,
2796
+ "learning_rate": 0.0002,
2797
+ "loss": 0.5413697957992554,
2798
+ "mean_token_accuracy": 0.7799932807683945,
2799
+ "num_tokens": 21521994.0,
2800
+ "step": 279
2801
+ },
2802
+ {
2803
+ "entropy": 0.5444028526544571,
2804
+ "epoch": 8.768,
2805
+ "grad_norm": 0.03536875918507576,
2806
+ "learning_rate": 0.0002,
2807
+ "loss": 0.5372054576873779,
2808
+ "mean_token_accuracy": 0.7830760926008224,
2809
+ "num_tokens": 21601066.0,
2810
+ "step": 280
2811
+ },
2812
+ {
2813
+ "entropy": 0.5271487236022949,
2814
+ "epoch": 8.8,
2815
+ "grad_norm": 0.04451414942741394,
2816
+ "learning_rate": 0.0002,
2817
+ "loss": 0.5246833562850952,
2818
+ "mean_token_accuracy": 0.7879171818494797,
2819
+ "num_tokens": 21680654.0,
2820
+ "step": 281
2821
+ },
2822
+ {
2823
+ "entropy": 0.5291329920291901,
2824
+ "epoch": 8.832,
2825
+ "grad_norm": 0.03518937528133392,
2826
+ "learning_rate": 0.0002,
2827
+ "loss": 0.5318595170974731,
2828
+ "mean_token_accuracy": 0.783314660191536,
2829
+ "num_tokens": 21759636.0,
2830
+ "step": 282
2831
+ },
2832
+ {
2833
+ "entropy": 0.5357151627540588,
2834
+ "epoch": 8.864,
2835
+ "grad_norm": 0.035144057124853134,
2836
+ "learning_rate": 0.0002,
2837
+ "loss": 0.5353978872299194,
2838
+ "mean_token_accuracy": 0.7822486013174057,
2839
+ "num_tokens": 21837957.0,
2840
+ "step": 283
2841
+ },
2842
+ {
2843
+ "entropy": 0.5227695554494858,
2844
+ "epoch": 8.896,
2845
+ "grad_norm": 0.03644225001335144,
2846
+ "learning_rate": 0.0002,
2847
+ "loss": 0.5201877951622009,
2848
+ "mean_token_accuracy": 0.789432018995285,
2849
+ "num_tokens": 21918900.0,
2850
+ "step": 284
2851
+ },
2852
+ {
2853
+ "entropy": 0.5280223190784454,
2854
+ "epoch": 8.928,
2855
+ "grad_norm": 0.03295109048485756,
2856
+ "learning_rate": 0.0002,
2857
+ "loss": 0.527430534362793,
2858
+ "mean_token_accuracy": 0.7872951477766037,
2859
+ "num_tokens": 21997073.0,
2860
+ "step": 285
2861
+ },
2862
+ {
2863
+ "entropy": 0.5388615727424622,
2864
+ "epoch": 8.96,
2865
+ "grad_norm": 0.04273361712694168,
2866
+ "learning_rate": 0.0002,
2867
+ "loss": 0.53923100233078,
2868
+ "mean_token_accuracy": 0.7823034077882767,
2869
+ "num_tokens": 22077250.0,
2870
+ "step": 286
2871
+ },
2872
+ {
2873
+ "entropy": 0.5351655334234238,
2874
+ "epoch": 8.992,
2875
+ "grad_norm": 0.03406304493546486,
2876
+ "learning_rate": 0.0002,
2877
+ "loss": 0.5302039384841919,
2878
+ "mean_token_accuracy": 0.7850055992603302,
2879
+ "num_tokens": 22154821.0,
2880
+ "step": 287
2881
+ },
2882
+ {
2883
+ "entropy": 0.5334904193878174,
2884
+ "epoch": 9.0,
2885
+ "grad_norm": 0.05444731563329697,
2886
+ "learning_rate": 0.0002,
2887
+ "loss": 0.530449390411377,
2888
+ "mean_token_accuracy": 0.7859096527099609,
2889
+ "num_tokens": 22174693.0,
2890
+ "step": 288
2891
+ },
2892
+ {
2893
+ "entropy": 0.5300431996583939,
2894
+ "epoch": 9.032,
2895
+ "grad_norm": 0.056497130542993546,
2896
+ "learning_rate": 0.0002,
2897
+ "loss": 0.527766227722168,
2898
+ "mean_token_accuracy": 0.785633996129036,
2899
+ "num_tokens": 22251556.0,
2900
+ "step": 289
2901
+ },
2902
+ {
2903
+ "entropy": 0.5187714546918869,
2904
+ "epoch": 9.064,
2905
+ "grad_norm": 0.045192088931798935,
2906
+ "learning_rate": 0.0002,
2907
+ "loss": 0.5194220542907715,
2908
+ "mean_token_accuracy": 0.7905025631189346,
2909
+ "num_tokens": 22330815.0,
2910
+ "step": 290
2911
+ },
2912
+ {
2913
+ "entropy": 0.5319517701864243,
2914
+ "epoch": 9.096,
2915
+ "grad_norm": 0.04722893610596657,
2916
+ "learning_rate": 0.0002,
2917
+ "loss": 0.5339574813842773,
2918
+ "mean_token_accuracy": 0.7833489924669266,
2919
+ "num_tokens": 22407165.0,
2920
+ "step": 291
2921
+ },
2922
+ {
2923
+ "entropy": 0.5170899480581284,
2924
+ "epoch": 9.128,
2925
+ "grad_norm": 0.052741702646017075,
2926
+ "learning_rate": 0.0002,
2927
+ "loss": 0.5152403116226196,
2928
+ "mean_token_accuracy": 0.7910004258155823,
2929
+ "num_tokens": 22486660.0,
2930
+ "step": 292
2931
+ },
2932
+ {
2933
+ "entropy": 0.5402307510375977,
2934
+ "epoch": 9.16,
2935
+ "grad_norm": 0.04082133248448372,
2936
+ "learning_rate": 0.0002,
2937
+ "loss": 0.5383718013763428,
2938
+ "mean_token_accuracy": 0.7814162373542786,
2939
+ "num_tokens": 22564897.0,
2940
+ "step": 293
2941
+ },
2942
+ {
2943
+ "entropy": 0.5201883316040039,
2944
+ "epoch": 9.192,
2945
+ "grad_norm": 0.03840792551636696,
2946
+ "learning_rate": 0.0002,
2947
+ "loss": 0.5196265578269958,
2948
+ "mean_token_accuracy": 0.7899857312440872,
2949
+ "num_tokens": 22644583.0,
2950
+ "step": 294
2951
+ },
2952
+ {
2953
+ "entropy": 0.5268342941999435,
2954
+ "epoch": 9.224,
2955
+ "grad_norm": 0.04609312862157822,
2956
+ "learning_rate": 0.0002,
2957
+ "loss": 0.5226351022720337,
2958
+ "mean_token_accuracy": 0.7885625064373016,
2959
+ "num_tokens": 22723224.0,
2960
+ "step": 295
2961
+ },
2962
+ {
2963
+ "entropy": 0.5258665084838867,
2964
+ "epoch": 9.256,
2965
+ "grad_norm": 0.03400503844022751,
2966
+ "learning_rate": 0.0002,
2967
+ "loss": 0.524506151676178,
2968
+ "mean_token_accuracy": 0.7877257317304611,
2969
+ "num_tokens": 22803901.0,
2970
+ "step": 296
2971
+ },
2972
+ {
2973
+ "entropy": 0.5259871706366539,
2974
+ "epoch": 9.288,
2975
+ "grad_norm": 0.04140326753258705,
2976
+ "learning_rate": 0.0002,
2977
+ "loss": 0.523682713508606,
2978
+ "mean_token_accuracy": 0.7880030572414398,
2979
+ "num_tokens": 22883095.0,
2980
+ "step": 297
2981
+ },
2982
+ {
2983
+ "entropy": 0.5212158113718033,
2984
+ "epoch": 9.32,
2985
+ "grad_norm": 0.035439494997262955,
2986
+ "learning_rate": 0.0002,
2987
+ "loss": 0.5157997608184814,
2988
+ "mean_token_accuracy": 0.791080117225647,
2989
+ "num_tokens": 22963722.0,
2990
+ "step": 298
2991
+ },
2992
+ {
2993
+ "entropy": 0.5116661787033081,
2994
+ "epoch": 9.352,
2995
+ "grad_norm": 0.04255674034357071,
2996
+ "learning_rate": 0.0002,
2997
+ "loss": 0.5151242613792419,
2998
+ "mean_token_accuracy": 0.7906815558671951,
2999
+ "num_tokens": 23044372.0,
3000
+ "step": 299
3001
+ },
3002
+ {
3003
+ "entropy": 0.5317155867815018,
3004
+ "epoch": 9.384,
3005
+ "grad_norm": 0.03571729734539986,
3006
+ "learning_rate": 0.0002,
3007
+ "loss": 0.5331252813339233,
3008
+ "mean_token_accuracy": 0.784525066614151,
3009
+ "num_tokens": 23121816.0,
3010
+ "step": 300
3011
+ },
3012
+ {
3013
+ "entropy": 0.5284074693918228,
3014
+ "epoch": 9.416,
3015
+ "grad_norm": 0.038814280182123184,
3016
+ "learning_rate": 0.0002,
3017
+ "loss": 0.5249905586242676,
3018
+ "mean_token_accuracy": 0.7884200662374496,
3019
+ "num_tokens": 23200583.0,
3020
+ "step": 301
3021
+ },
3022
+ {
3023
+ "entropy": 0.5218486413359642,
3024
+ "epoch": 9.448,
3025
+ "grad_norm": 0.03548124432563782,
3026
+ "learning_rate": 0.0002,
3027
+ "loss": 0.5189951658248901,
3028
+ "mean_token_accuracy": 0.7910516560077667,
3029
+ "num_tokens": 23279098.0,
3030
+ "step": 302
3031
+ },
3032
+ {
3033
+ "entropy": 0.5266124904155731,
3034
+ "epoch": 9.48,
3035
+ "grad_norm": 0.04210549592971802,
3036
+ "learning_rate": 0.0002,
3037
+ "loss": 0.5285375118255615,
3038
+ "mean_token_accuracy": 0.7852902412414551,
3039
+ "num_tokens": 23357708.0,
3040
+ "step": 303
3041
+ },
3042
+ {
3043
+ "entropy": 0.5390820801258087,
3044
+ "epoch": 9.512,
3045
+ "grad_norm": 0.04019097238779068,
3046
+ "learning_rate": 0.0002,
3047
+ "loss": 0.5387204885482788,
3048
+ "mean_token_accuracy": 0.7813313454389572,
3049
+ "num_tokens": 23435254.0,
3050
+ "step": 304
3051
+ },
3052
+ {
3053
+ "entropy": 0.5226802378892899,
3054
+ "epoch": 9.544,
3055
+ "grad_norm": 0.045527562499046326,
3056
+ "learning_rate": 0.0002,
3057
+ "loss": 0.5184016227722168,
3058
+ "mean_token_accuracy": 0.7905609607696533,
3059
+ "num_tokens": 23514397.0,
3060
+ "step": 305
3061
+ },
3062
+ {
3063
+ "entropy": 0.527540773153305,
3064
+ "epoch": 9.576,
3065
+ "grad_norm": 0.04471491649746895,
3066
+ "learning_rate": 0.0002,
3067
+ "loss": 0.5262195467948914,
3068
+ "mean_token_accuracy": 0.7857372760772705,
3069
+ "num_tokens": 23593391.0,
3070
+ "step": 306
3071
+ },
3072
+ {
3073
+ "entropy": 0.5327746719121933,
3074
+ "epoch": 9.608,
3075
+ "grad_norm": 0.03983209654688835,
3076
+ "learning_rate": 0.0002,
3077
+ "loss": 0.5309550762176514,
3078
+ "mean_token_accuracy": 0.7848490923643112,
3079
+ "num_tokens": 23671238.0,
3080
+ "step": 307
3081
+ },
3082
+ {
3083
+ "entropy": 0.5221132040023804,
3084
+ "epoch": 9.64,
3085
+ "grad_norm": 0.0506560355424881,
3086
+ "learning_rate": 0.0002,
3087
+ "loss": 0.5292017459869385,
3088
+ "mean_token_accuracy": 0.7862422913312912,
3089
+ "num_tokens": 23750415.0,
3090
+ "step": 308
3091
+ },
3092
+ {
3093
+ "entropy": 0.5316873043775558,
3094
+ "epoch": 9.672,
3095
+ "grad_norm": 0.04567805677652359,
3096
+ "learning_rate": 0.0002,
3097
+ "loss": 0.5264989137649536,
3098
+ "mean_token_accuracy": 0.7868478447198868,
3099
+ "num_tokens": 23829328.0,
3100
+ "step": 309
3101
+ },
3102
+ {
3103
+ "entropy": 0.5508539825677872,
3104
+ "epoch": 9.704,
3105
+ "grad_norm": 0.054871875792741776,
3106
+ "learning_rate": 0.0002,
3107
+ "loss": 0.5446195602416992,
3108
+ "mean_token_accuracy": 0.7784004658460617,
3109
+ "num_tokens": 23906430.0,
3110
+ "step": 310
3111
+ },
3112
+ {
3113
+ "entropy": 0.5284042209386826,
3114
+ "epoch": 9.736,
3115
+ "grad_norm": 0.05515004321932793,
3116
+ "learning_rate": 0.0002,
3117
+ "loss": 0.531845211982727,
3118
+ "mean_token_accuracy": 0.7849012762308121,
3119
+ "num_tokens": 23985699.0,
3120
+ "step": 311
3121
+ },
3122
+ {
3123
+ "entropy": 0.5239458531141281,
3124
+ "epoch": 9.768,
3125
+ "grad_norm": 0.06098300218582153,
3126
+ "learning_rate": 0.0002,
3127
+ "loss": 0.5323519706726074,
3128
+ "mean_token_accuracy": 0.7858945429325104,
3129
+ "num_tokens": 24065076.0,
3130
+ "step": 312
3131
+ },
3132
+ {
3133
+ "entropy": 0.5360407829284668,
3134
+ "epoch": 9.8,
3135
+ "grad_norm": 0.0424746535718441,
3136
+ "learning_rate": 0.0002,
3137
+ "loss": 0.5293824672698975,
3138
+ "mean_token_accuracy": 0.7857670187950134,
3139
+ "num_tokens": 24144078.0,
3140
+ "step": 313
3141
+ },
3142
+ {
3143
+ "entropy": 0.5299330949783325,
3144
+ "epoch": 9.832,
3145
+ "grad_norm": 0.05260491743683815,
3146
+ "learning_rate": 0.0002,
3147
+ "loss": 0.5241835117340088,
3148
+ "mean_token_accuracy": 0.7872831225395203,
3149
+ "num_tokens": 24224502.0,
3150
+ "step": 314
3151
+ },
3152
+ {
3153
+ "entropy": 0.5187924355268478,
3154
+ "epoch": 9.864,
3155
+ "grad_norm": 0.07041396200656891,
3156
+ "learning_rate": 0.0002,
3157
+ "loss": 0.5201770067214966,
3158
+ "mean_token_accuracy": 0.7910154610872269,
3159
+ "num_tokens": 24304691.0,
3160
+ "step": 315
3161
+ },
3162
+ {
3163
+ "entropy": 0.5294213145971298,
3164
+ "epoch": 9.896,
3165
+ "grad_norm": 0.04269389435648918,
3166
+ "learning_rate": 0.0002,
3167
+ "loss": 0.5291866660118103,
3168
+ "mean_token_accuracy": 0.7848200798034668,
3169
+ "num_tokens": 24383384.0,
3170
+ "step": 316
3171
+ },
3172
+ {
3173
+ "entropy": 0.5307356864213943,
3174
+ "epoch": 9.928,
3175
+ "grad_norm": 0.071862131357193,
3176
+ "learning_rate": 0.0002,
3177
+ "loss": 0.5289138555526733,
3178
+ "mean_token_accuracy": 0.7856195569038391,
3179
+ "num_tokens": 24461247.0,
3180
+ "step": 317
3181
+ },
3182
+ {
3183
+ "entropy": 0.526089146733284,
3184
+ "epoch": 9.96,
3185
+ "grad_norm": 0.06460758298635483,
3186
+ "learning_rate": 0.0002,
3187
+ "loss": 0.5239361524581909,
3188
+ "mean_token_accuracy": 0.7876095771789551,
3189
+ "num_tokens": 24540321.0,
3190
+ "step": 318
3191
+ },
3192
+ {
3193
+ "entropy": 0.5216004997491837,
3194
+ "epoch": 9.992,
3195
+ "grad_norm": 0.05195415019989014,
3196
+ "learning_rate": 0.0002,
3197
+ "loss": 0.5229140520095825,
3198
+ "mean_token_accuracy": 0.7891107648611069,
3199
+ "num_tokens": 24618023.0,
3200
+ "step": 319
3201
+ },
3202
+ {
3203
+ "entropy": 0.5302079319953918,
3204
+ "epoch": 10.0,
3205
+ "grad_norm": 0.0679510235786438,
3206
+ "learning_rate": 0.0002,
3207
+ "loss": 0.51785808801651,
3208
+ "mean_token_accuracy": 0.7922472953796387,
3209
+ "num_tokens": 24637886.0,
3210
+ "step": 320
3211
+ }
3212
+ ],
3213
+ "logging_steps": 1,
3214
+ "max_steps": 320,
3215
+ "num_input_tokens_seen": 0,
3216
+ "num_train_epochs": 10,
3217
+ "save_steps": 500,
3218
+ "stateful_callbacks": {
3219
+ "TrainerControl": {
3220
+ "args": {
3221
+ "should_epoch_stop": false,
3222
+ "should_evaluate": false,
3223
+ "should_log": false,
3224
+ "should_save": true,
3225
+ "should_training_stop": true
3226
+ },
3227
+ "attributes": {}
3228
+ }
3229
+ },
3230
+ "total_flos": 1.7190992994946253e+18,
3231
+ "train_batch_size": 16,
3232
+ "trial_name": null,
3233
+ "trial_params": null
3234
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18e4a983810590b5b8c470a070dcc3f49ba3ec1152f3b80f3d4e3c8a58b40aba
3
+ size 5649