Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-1200/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1400/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1584/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,219 @@
+base_model: CohereLabs/aya-expanse-8b
+library_name: peft
+model_name: aya-expanse-8b-tunisian-sft
+tags:
+licence: license
+pipeline_tag: text-generation
+
+---
+base_model: CohereLabs/aya-expanse-8b
+library_name: peft
+model_name: TounsiLM-8b
+tags:
+  - base_model:adapter:CohereLabs/aya-expanse-8b
+  - peft
+  - lora
+  - sft
+  - transformers
+  - trl
+  - tunisian-arabic
+  - text-generation
+pipeline_tag: text-generation
+language:
+  - ar
+license: apache-2.0
+---
+
+# TounsiLM-8b
+
+`TounsiLM-8b` is a Tunisian Arabic supervised fine-tuning adapter built on top of [CohereLabs/aya-expanse-8b](https://huggingface.co/CohereLabs/aya-expanse-8b).
+
+It is trained to answer in Tunisian دارجة, stay on topic, and keep responses short and direct when appropriate.
+
+## Model type
+
+- Base model: `CohereLabs/aya-expanse-8b`
+- Fine-tuning method: PEFT / LoRA-style SFT adapter
+- Format: adapter checkpoint, not a fully merged standalone base model
+
+## Training dataset
+
+- Dataset: `Syrinesmati/tunisian-question-response-dataset`
+- Train split: `25,340` rows
+- Eval split: `6,336` rows
+- Input format: conversational messages built from the dataset fields `instruction` and `response`
+
+## Training setup
+
+- Trainer: TRL `SFTTrainer`
+- Epochs: `2`
+- Max sequence length: `1024`
+- Learning rate: `1e-5`
+- Per-device train batch size: `8`
+- Gradient accumulation: `4`
+- Precision: `bf16` when supported
+- Checkpoint resume: enabled
+
+## Training metrics
+
+Final reported training metrics:
+
+- Training loss: `1.1876104943680041`
+- Mean token accuracy: `0.7577789686620235`
+- Training runtime: `50353.3546` seconds
+- Training steps: `1584`
+- Total tokens seen: `9,585,534`
+
+These are training metrics from the final log. No separate validation loss was recorded in the saved metrics file.
+
+## Intended use
+
+Use this model for:
+
+- Tunisian Arabic question answering
+- chat-style assistant replies in Tunisian دارجة
+- short, direct conversational responses
+
+Not intended for:
+
+- factual safety-critical advice
+- medical/legal/financial decisions without verification
+- unsupported languages outside Arabic/Tunisian use cases
+
+## How to use
+
+### Option 1: load the adapter with the base model
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+import torch
+
+base_model_name = "CohereLabs/aya-expanse-8b"
+adapter_dir = "TounsiLM-8b"
+
+tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    base_model_name,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+)
+model = PeftModel.from_pretrained(model, adapter_dir)
+
+messages = [
+    {"role": "system", "content": "أنت مساعد تونسي تجاوب بالتونسي الدارج فقط."},
+    {"role": "user", "content": "شنوة تعمل كان الواحد يحس روحو تعبان؟"},
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+)
+
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
+```
+
+### Option 2: use the model in a pipeline
+
+```python
+from transformers import pipeline
+
+gen = pipeline("text-generation", model=model, tokenizer=tokenizer)
+```
+
+## Recommended inference settings
+
+- `do_sample=False` for more stable answers
+- `max_new_tokens=128` to reduce rambling
+- `repetition_penalty=1.1`
+
+## Files included in this repository
+
+- `adapter_model.safetensors`
+- `adapter_config.json`
+- `chat_template.jinja`
+- tokenizer files
+- training metrics and logs
+
+## Framework versions
+
+- PEFT: `0.19.1`
+- TRL: `1.3.0`
+- Transformers: `4.57.6`
+- PyTorch: `2.11.0`
+- Datasets: `4.8.5`
+- Tokenizers: `0.22.2`
+
+## Notes
+
+This repository contains the fine-tuned adapter. To use it, load it on top of the base model `CohereLabs/aya-expanse-8b`.
+
+If you want a merged standalone model later, the adapter can be merged into the base model and re-uploaded as a separate repo.
+
+## Citation
+
+If you use this model, please cite the base model and the training stack used to create it.
+
+### TRL citation
+
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```
+This model is a fine-tuned version of [CohereLabs/aya-expanse-8b](https://huggingface.co/CohereLabs/aya-expanse-8b).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+ 
+
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- PEFT 0.19.1
+- TRL: 1.3.0
+- Transformers: 4.57.6
+- Pytorch: 2.11.0
+- Datasets: 4.8.5
+- Tokenizers: 0.22.2
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "CohereLabs/aya-expanse-8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "o_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4ccb6b6fb84ee6cc158f9a20474ec7ec29ec158992cc2108c5575301c294143
+size 167832240

chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true %}{% set loop_messages = messages %}{% set system_message = 'You are Aya, a brilliant, sophisticated, multilingual AI-assistant trained to assist human users by providing thorough responses. You are able to interact and respond to questions in 23 languages and you are powered by a multilingual model built by Cohere For AI.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% if system_message != false %}{{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% elif message['role'] == 'assistant' %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'  + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }}{% endif %}

checkpoint-1200/README.md

@@ -0,0 +1,209 @@
+---
+base_model: CohereLabs/aya-expanse-8b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:CohereLabs/aya-expanse-8b
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-1200/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "CohereLabs/aya-expanse-8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "o_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5024a4200df787ce8cb3a1a23d9a1439358792a465f6b3f6b931add79f84aa0a
+size 167832240

checkpoint-1200/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true %}{% set loop_messages = messages %}{% set system_message = 'You are Aya, a brilliant, sophisticated, multilingual AI-assistant trained to assist human users by providing thorough responses. You are able to interact and respond to questions in 23 languages and you are powered by a multilingual model built by Cohere For AI.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% if system_message != false %}{{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% elif message['role'] == 'assistant' %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'  + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }}{% endif %}

checkpoint-1200/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c53c564c508c035cec95a38fc431d6b3652878bb3968ec00e87d09170583ecac
+size 335929123

checkpoint-1200/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1215aee6c0e3407d5373ec2bcba0768cfee0477976cf9edb2d49d4f35aeb257
+size 14645

checkpoint-1200/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b68b5be1196467245d97fdf07264667e377abf37f1c86865e3ac71fc66f675b3
+size 1465

checkpoint-1200/special_tokens_map.json

@@ -0,0 +1,17 @@
+{
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<PAD>"
+}

checkpoint-1200/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:345ccf04a5257f473e331715ecc69365c5ac8fc2490923fe7155560af809ec1a
+size 20124090

checkpoint-1200/tokenizer_config.json

@@ -0,0 +1,317 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<CLS>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<SEP>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<EOP_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255000": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255001": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255002": {
+      "content": "<|YES_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255003": {
+      "content": "<|NO_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255004": {
+      "content": "<|GOOD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255005": {
+      "content": "<|BAD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255006": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255007": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255008": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255009": {
+      "content": "<|USER_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255010": {
+      "content": "<|USER_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255011": {
+      "content": "<|USER_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255012": {
+      "content": "<|USER_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255013": {
+      "content": "<|USER_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255014": {
+      "content": "<|USER_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255015": {
+      "content": "<|USER_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255016": {
+      "content": "<|USER_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255017": {
+      "content": "<|USER_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255018": {
+      "content": "<|USER_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255019": {
+      "content": "<|EXTRA_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255020": {
+      "content": "<|EXTRA_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255021": {
+      "content": "<|EXTRA_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255022": {
+      "content": "<|EXTRA_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255023": {
+      "content": "<|EXTRA_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255024": {
+      "content": "<|EXTRA_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255025": {
+      "content": "<|EXTRA_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255026": {
+      "content": "<|EXTRA_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255027": {
+      "content": "<|EXTRA_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255028": {
+      "content": "<|EXTRA_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<BOS_TOKEN>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|END_OF_TURN_TOKEN|>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "merges_file": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "CohereTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false,
+  "vocab_file": null
+}

checkpoint-1200/trainer_state.json

@@ -0,0 +1,1234 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.5151515151515151,
+  "eval_steps": 200,
+  "global_step": 1200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.4040649354457857,
+      "epoch": 0.012626262626262626,
+      "grad_norm": 4.66565465927124,
+      "learning_rate": 1.8750000000000003e-06,
+      "loss": 3.6021,
+      "mean_token_accuracy": 0.4221017129719257,
+      "num_tokens": 61199.0,
+      "step": 10
+    },
+    {
+      "entropy": 2.3836746215820312,
+      "epoch": 0.025252525252525252,
+      "grad_norm": 3.8161869049072266,
+      "learning_rate": 3.958333333333333e-06,
+      "loss": 3.3432,
+      "mean_token_accuracy": 0.44042530804872515,
+      "num_tokens": 122423.0,
+      "step": 20
+    },
+    {
+      "entropy": 2.355724626779556,
+      "epoch": 0.03787878787878788,
+      "grad_norm": 3.8800699710845947,
+      "learning_rate": 6.041666666666667e-06,
+      "loss": 2.9033,
+      "mean_token_accuracy": 0.48426677361130716,
+      "num_tokens": 182649.0,
+      "step": 30
+    },
+    {
+      "entropy": 2.092331054806709,
+      "epoch": 0.050505050505050504,
+      "grad_norm": 2.8217720985412598,
+      "learning_rate": 8.125000000000001e-06,
+      "loss": 2.356,
+      "mean_token_accuracy": 0.5772452697157859,
+      "num_tokens": 243049.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.6766322344541549,
+      "epoch": 0.06313131313131314,
+      "grad_norm": 1.4623568058013916,
+      "learning_rate": 9.993489583333334e-06,
+      "loss": 1.8899,
+      "mean_token_accuracy": 0.6480962842702865,
+      "num_tokens": 304326.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.5568815559148788,
+      "epoch": 0.07575757575757576,
+      "grad_norm": 1.171562671661377,
+      "learning_rate": 9.928385416666668e-06,
+      "loss": 1.677,
+      "mean_token_accuracy": 0.6776855796575546,
+      "num_tokens": 364866.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.48199902176857,
+      "epoch": 0.08838383838383838,
+      "grad_norm": 0.9904961585998535,
+      "learning_rate": 9.863281250000001e-06,
+      "loss": 1.5337,
+      "mean_token_accuracy": 0.697019773721695,
+      "num_tokens": 423478.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.497376424074173,
+      "epoch": 0.10101010101010101,
+      "grad_norm": 0.9454260468482971,
+      "learning_rate": 9.798177083333335e-06,
+      "loss": 1.4953,
+      "mean_token_accuracy": 0.6976823702454567,
+      "num_tokens": 483659.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.4664768785238267,
+      "epoch": 0.11363636363636363,
+      "grad_norm": 0.8955270648002625,
+      "learning_rate": 9.733072916666667e-06,
+      "loss": 1.4356,
+      "mean_token_accuracy": 0.7069446608424187,
+      "num_tokens": 544453.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.4269085675477982,
+      "epoch": 0.12626262626262627,
+      "grad_norm": 0.9242203235626221,
+      "learning_rate": 9.66796875e-06,
+      "loss": 1.4106,
+      "mean_token_accuracy": 0.7122392952442169,
+      "num_tokens": 604546.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.4060751020908355,
+      "epoch": 0.1388888888888889,
+      "grad_norm": 0.8968560695648193,
+      "learning_rate": 9.602864583333335e-06,
+      "loss": 1.3487,
+      "mean_token_accuracy": 0.7178552970290184,
+      "num_tokens": 664860.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.4018951296806335,
+      "epoch": 0.15151515151515152,
+      "grad_norm": 0.9047113656997681,
+      "learning_rate": 9.537760416666667e-06,
+      "loss": 1.3347,
+      "mean_token_accuracy": 0.7208079636096955,
+      "num_tokens": 725022.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.3809731483459473,
+      "epoch": 0.16414141414141414,
+      "grad_norm": 0.8915444016456604,
+      "learning_rate": 9.47265625e-06,
+      "loss": 1.3155,
+      "mean_token_accuracy": 0.7267520889639855,
+      "num_tokens": 785586.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.3699676394462585,
+      "epoch": 0.17676767676767677,
+      "grad_norm": 0.8574295043945312,
+      "learning_rate": 9.407552083333334e-06,
+      "loss": 1.3016,
+      "mean_token_accuracy": 0.7266673430800438,
+      "num_tokens": 845790.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.3425012439489366,
+      "epoch": 0.1893939393939394,
+      "grad_norm": 0.8231800198554993,
+      "learning_rate": 9.342447916666668e-06,
+      "loss": 1.2823,
+      "mean_token_accuracy": 0.7277117937803268,
+      "num_tokens": 905842.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.3314216613769532,
+      "epoch": 0.20202020202020202,
+      "grad_norm": 0.8166369795799255,
+      "learning_rate": 9.277343750000001e-06,
+      "loss": 1.2917,
+      "mean_token_accuracy": 0.7278609350323677,
+      "num_tokens": 966487.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.30389544069767,
+      "epoch": 0.21464646464646464,
+      "grad_norm": 0.7738587260246277,
+      "learning_rate": 9.212239583333335e-06,
+      "loss": 1.2481,
+      "mean_token_accuracy": 0.7335339426994324,
+      "num_tokens": 1025558.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.311449444293976,
+      "epoch": 0.22727272727272727,
+      "grad_norm": 0.7718328833580017,
+      "learning_rate": 9.147135416666667e-06,
+      "loss": 1.2643,
+      "mean_token_accuracy": 0.7285080313682556,
+      "num_tokens": 1086677.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.3079361200332642,
+      "epoch": 0.2398989898989899,
+      "grad_norm": 0.7341915369033813,
+      "learning_rate": 9.082031250000001e-06,
+      "loss": 1.2641,
+      "mean_token_accuracy": 0.729550538957119,
+      "num_tokens": 1147885.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.2794219702482224,
+      "epoch": 0.25252525252525254,
+      "grad_norm": 0.748540997505188,
+      "learning_rate": 9.016927083333335e-06,
+      "loss": 1.2397,
+      "mean_token_accuracy": 0.7351120054721832,
+      "num_tokens": 1207321.0,
+      "step": 200
+    },
+    {
+      "entropy": 1.294454461336136,
+      "epoch": 0.26515151515151514,
+      "grad_norm": 0.7585553526878357,
+      "learning_rate": 8.951822916666667e-06,
+      "loss": 1.2489,
+      "mean_token_accuracy": 0.7322148531675339,
+      "num_tokens": 1267837.0,
+      "step": 210
+    },
+    {
+      "entropy": 1.2862246632575989,
+      "epoch": 0.2777777777777778,
+      "grad_norm": 0.6937864422798157,
+      "learning_rate": 8.88671875e-06,
+      "loss": 1.2383,
+      "mean_token_accuracy": 0.7358245223760604,
+      "num_tokens": 1328436.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.2531811505556107,
+      "epoch": 0.2904040404040404,
+      "grad_norm": 0.6792387366294861,
+      "learning_rate": 8.821614583333334e-06,
+      "loss": 1.2007,
+      "mean_token_accuracy": 0.7360369265079498,
+      "num_tokens": 1389877.0,
+      "step": 230
+    },
+    {
+      "entropy": 1.2815489560365676,
+      "epoch": 0.30303030303030304,
+      "grad_norm": 0.6865427494049072,
+      "learning_rate": 8.756510416666666e-06,
+      "loss": 1.2474,
+      "mean_token_accuracy": 0.7304804190993309,
+      "num_tokens": 1450927.0,
+      "step": 240
+    },
+    {
+      "entropy": 1.2568059146404267,
+      "epoch": 0.31565656565656564,
+      "grad_norm": 0.669840395450592,
+      "learning_rate": 8.69140625e-06,
+      "loss": 1.2172,
+      "mean_token_accuracy": 0.7385278165340423,
+      "num_tokens": 1511174.0,
+      "step": 250
+    },
+    {
+      "entropy": 1.254868358373642,
+      "epoch": 0.3282828282828283,
+      "grad_norm": 0.6434893012046814,
+      "learning_rate": 8.626302083333334e-06,
+      "loss": 1.213,
+      "mean_token_accuracy": 0.7380544006824493,
+      "num_tokens": 1570570.0,
+      "step": 260
+    },
+    {
+      "entropy": 1.2532441645860672,
+      "epoch": 0.3409090909090909,
+      "grad_norm": 0.6034978032112122,
+      "learning_rate": 8.561197916666667e-06,
+      "loss": 1.2116,
+      "mean_token_accuracy": 0.7382062628865242,
+      "num_tokens": 1630930.0,
+      "step": 270
+    },
+    {
+      "entropy": 1.2605280816555022,
+      "epoch": 0.35353535353535354,
+      "grad_norm": 0.6371450424194336,
+      "learning_rate": 8.496093750000001e-06,
+      "loss": 1.2313,
+      "mean_token_accuracy": 0.7328852489590645,
+      "num_tokens": 1692624.0,
+      "step": 280
+    },
+    {
+      "entropy": 1.248606452345848,
+      "epoch": 0.3661616161616162,
+      "grad_norm": 0.6300278306007385,
+      "learning_rate": 8.430989583333335e-06,
+      "loss": 1.2195,
+      "mean_token_accuracy": 0.7370449885725975,
+      "num_tokens": 1754213.0,
+      "step": 290
+    },
+    {
+      "entropy": 1.2246102809906005,
+      "epoch": 0.3787878787878788,
+      "grad_norm": 0.6430155634880066,
+      "learning_rate": 8.365885416666667e-06,
+      "loss": 1.1845,
+      "mean_token_accuracy": 0.7439196646213532,
+      "num_tokens": 1813841.0,
+      "step": 300
+    },
+    {
+      "entropy": 1.2188815206289292,
+      "epoch": 0.39141414141414144,
+      "grad_norm": 0.6395701766014099,
+      "learning_rate": 8.30078125e-06,
+      "loss": 1.1856,
+      "mean_token_accuracy": 0.7422183871269226,
+      "num_tokens": 1873568.0,
+      "step": 310
+    },
+    {
+      "entropy": 1.234528934955597,
+      "epoch": 0.40404040404040403,
+      "grad_norm": 0.6168740391731262,
+      "learning_rate": 8.235677083333334e-06,
+      "loss": 1.1951,
+      "mean_token_accuracy": 0.7380509555339814,
+      "num_tokens": 1935518.0,
+      "step": 320
+    },
+    {
+      "entropy": 1.226365676522255,
+      "epoch": 0.4166666666666667,
+      "grad_norm": 0.611132800579071,
+      "learning_rate": 8.170572916666666e-06,
+      "loss": 1.2078,
+      "mean_token_accuracy": 0.739974245429039,
+      "num_tokens": 1995604.0,
+      "step": 330
+    },
+    {
+      "entropy": 1.2132183194160462,
+      "epoch": 0.4292929292929293,
+      "grad_norm": 0.6103131771087646,
+      "learning_rate": 8.10546875e-06,
+      "loss": 1.1642,
+      "mean_token_accuracy": 0.7444137379527092,
+      "num_tokens": 2056484.0,
+      "step": 340
+    },
+    {
+      "entropy": 1.223158246278763,
+      "epoch": 0.44191919191919193,
+      "grad_norm": 0.6188805103302002,
+      "learning_rate": 8.040364583333334e-06,
+      "loss": 1.2001,
+      "mean_token_accuracy": 0.7385613292455673,
+      "num_tokens": 2118437.0,
+      "step": 350
+    },
+    {
+      "entropy": 1.2154075980186463,
+      "epoch": 0.45454545454545453,
+      "grad_norm": 0.6238694190979004,
+      "learning_rate": 7.975260416666668e-06,
+      "loss": 1.1848,
+      "mean_token_accuracy": 0.7404153689742088,
+      "num_tokens": 2179333.0,
+      "step": 360
+    },
+    {
+      "entropy": 1.197928261756897,
+      "epoch": 0.4671717171717172,
+      "grad_norm": 0.6028566956520081,
+      "learning_rate": 7.910156250000001e-06,
+      "loss": 1.1597,
+      "mean_token_accuracy": 0.7475010469555855,
+      "num_tokens": 2239604.0,
+      "step": 370
+    },
+    {
+      "entropy": 1.189855706691742,
+      "epoch": 0.4797979797979798,
+      "grad_norm": 0.6569434404373169,
+      "learning_rate": 7.845052083333335e-06,
+      "loss": 1.1805,
+      "mean_token_accuracy": 0.7427218139171601,
+      "num_tokens": 2300936.0,
+      "step": 380
+    },
+    {
+      "entropy": 1.2076493889093398,
+      "epoch": 0.49242424242424243,
+      "grad_norm": 0.6351733207702637,
+      "learning_rate": 7.779947916666667e-06,
+      "loss": 1.1759,
+      "mean_token_accuracy": 0.7408407002687454,
+      "num_tokens": 2361539.0,
+      "step": 390
+    },
+    {
+      "entropy": 1.2223081022500992,
+      "epoch": 0.5050505050505051,
+      "grad_norm": 0.6327986121177673,
+      "learning_rate": 7.71484375e-06,
+      "loss": 1.1874,
+      "mean_token_accuracy": 0.7407250568270684,
+      "num_tokens": 2422162.0,
+      "step": 400
+    },
+    {
+      "entropy": 1.2013412863016129,
+      "epoch": 0.5176767676767676,
+      "grad_norm": 0.622104823589325,
+      "learning_rate": 7.649739583333334e-06,
+      "loss": 1.1726,
+      "mean_token_accuracy": 0.744242025911808,
+      "num_tokens": 2483447.0,
+      "step": 410
+    },
+    {
+      "entropy": 1.2116613179445266,
+      "epoch": 0.5303030303030303,
+      "grad_norm": 0.637651264667511,
+      "learning_rate": 7.5846354166666665e-06,
+      "loss": 1.1838,
+      "mean_token_accuracy": 0.7405030101537704,
+      "num_tokens": 2544848.0,
+      "step": 420
+    },
+    {
+      "entropy": 1.2024697184562683,
+      "epoch": 0.5429292929292929,
+      "grad_norm": 0.6252374649047852,
+      "learning_rate": 7.51953125e-06,
+      "loss": 1.1681,
+      "mean_token_accuracy": 0.7458183988928795,
+      "num_tokens": 2605232.0,
+      "step": 430
+    },
+    {
+      "entropy": 1.1797083109617232,
+      "epoch": 0.5555555555555556,
+      "grad_norm": 0.6502755284309387,
+      "learning_rate": 7.454427083333334e-06,
+      "loss": 1.1452,
+      "mean_token_accuracy": 0.7477341219782829,
+      "num_tokens": 2664276.0,
+      "step": 440
+    },
+    {
+      "entropy": 1.1964200481772422,
+      "epoch": 0.5681818181818182,
+      "grad_norm": 0.639979362487793,
+      "learning_rate": 7.389322916666667e-06,
+      "loss": 1.1665,
+      "mean_token_accuracy": 0.7431837096810341,
+      "num_tokens": 2724073.0,
+      "step": 450
+    },
+    {
+      "entropy": 1.1795272737741471,
+      "epoch": 0.5808080808080808,
+      "grad_norm": 0.6212354302406311,
+      "learning_rate": 7.3242187500000006e-06,
+      "loss": 1.1529,
+      "mean_token_accuracy": 0.7479098170995713,
+      "num_tokens": 2784262.0,
+      "step": 460
+    },
+    {
+      "entropy": 1.1908618807792664,
+      "epoch": 0.5934343434343434,
+      "grad_norm": 0.6528693437576294,
+      "learning_rate": 7.259114583333334e-06,
+      "loss": 1.1678,
+      "mean_token_accuracy": 0.745174677670002,
+      "num_tokens": 2843804.0,
+      "step": 470
+    },
+    {
+      "entropy": 1.1862946093082427,
+      "epoch": 0.6060606060606061,
+      "grad_norm": 0.639481246471405,
+      "learning_rate": 7.194010416666667e-06,
+      "loss": 1.1565,
+      "mean_token_accuracy": 0.7461866185069084,
+      "num_tokens": 2903408.0,
+      "step": 480
+    },
+    {
+      "entropy": 1.151743534207344,
+      "epoch": 0.6186868686868687,
+      "grad_norm": 0.6332777142524719,
+      "learning_rate": 7.128906250000001e-06,
+      "loss": 1.1251,
+      "mean_token_accuracy": 0.7535199671983719,
+      "num_tokens": 2963401.0,
+      "step": 490
+    },
+    {
+      "entropy": 1.1778477430343628,
+      "epoch": 0.6313131313131313,
+      "grad_norm": 0.5991836190223694,
+      "learning_rate": 7.063802083333335e-06,
+      "loss": 1.1407,
+      "mean_token_accuracy": 0.7491110354661942,
+      "num_tokens": 3023530.0,
+      "step": 500
+    },
+    {
+      "entropy": 1.2023035794496537,
+      "epoch": 0.6439393939393939,
+      "grad_norm": 0.6293458938598633,
+      "learning_rate": 6.998697916666667e-06,
+      "loss": 1.1724,
+      "mean_token_accuracy": 0.7405256554484367,
+      "num_tokens": 3085225.0,
+      "step": 510
+    },
+    {
+      "entropy": 1.1997334092855454,
+      "epoch": 0.6565656565656566,
+      "grad_norm": 0.6213802695274353,
+      "learning_rate": 6.93359375e-06,
+      "loss": 1.1604,
+      "mean_token_accuracy": 0.7435309410095214,
+      "num_tokens": 3145749.0,
+      "step": 520
+    },
+    {
+      "entropy": 1.1643184214830398,
+      "epoch": 0.6691919191919192,
+      "grad_norm": 0.6495156288146973,
+      "learning_rate": 6.868489583333334e-06,
+      "loss": 1.1381,
+      "mean_token_accuracy": 0.7496751576662064,
+      "num_tokens": 3205761.0,
+      "step": 530
+    },
+    {
+      "entropy": 1.1837532848119736,
+      "epoch": 0.6818181818181818,
+      "grad_norm": 0.6004510521888733,
+      "learning_rate": 6.803385416666667e-06,
+      "loss": 1.1563,
+      "mean_token_accuracy": 0.7447851061820984,
+      "num_tokens": 3267024.0,
+      "step": 540
+    },
+    {
+      "entropy": 1.201677542924881,
+      "epoch": 0.6944444444444444,
+      "grad_norm": 0.607467532157898,
+      "learning_rate": 6.738281250000001e-06,
+      "loss": 1.1776,
+      "mean_token_accuracy": 0.7406648993492126,
+      "num_tokens": 3329070.0,
+      "step": 550
+    },
+    {
+      "entropy": 1.1659786373376846,
+      "epoch": 0.7070707070707071,
+      "grad_norm": 0.6079947352409363,
+      "learning_rate": 6.6731770833333345e-06,
+      "loss": 1.1298,
+      "mean_token_accuracy": 0.7505511298775673,
+      "num_tokens": 3389505.0,
+      "step": 560
+    },
+    {
+      "entropy": 1.1714913487434386,
+      "epoch": 0.7196969696969697,
+      "grad_norm": 0.6534572839736938,
+      "learning_rate": 6.6080729166666665e-06,
+      "loss": 1.144,
+      "mean_token_accuracy": 0.7481625184416771,
+      "num_tokens": 3449834.0,
+      "step": 570
+    },
+    {
+      "entropy": 1.1694963037967683,
+      "epoch": 0.7323232323232324,
+      "grad_norm": 0.5903164744377136,
+      "learning_rate": 6.54296875e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7476441130042076,
+      "num_tokens": 3510701.0,
+      "step": 580
+    },
+    {
+      "entropy": 1.1777653217315673,
+      "epoch": 0.7449494949494949,
+      "grad_norm": 0.6284182071685791,
+      "learning_rate": 6.477864583333334e-06,
+      "loss": 1.1422,
+      "mean_token_accuracy": 0.7490358456969262,
+      "num_tokens": 3570992.0,
+      "step": 590
+    },
+    {
+      "entropy": 1.1588268011808396,
+      "epoch": 0.7575757575757576,
+      "grad_norm": 0.6250146627426147,
+      "learning_rate": 6.412760416666667e-06,
+      "loss": 1.1336,
+      "mean_token_accuracy": 0.7504108369350433,
+      "num_tokens": 3631087.0,
+      "step": 600
+    },
+    {
+      "entropy": 1.1890273630619048,
+      "epoch": 0.7702020202020202,
+      "grad_norm": 0.6420578956604004,
+      "learning_rate": 6.3476562500000006e-06,
+      "loss": 1.1534,
+      "mean_token_accuracy": 0.7442372158169747,
+      "num_tokens": 3692871.0,
+      "step": 610
+    },
+    {
+      "entropy": 1.1817798465490341,
+      "epoch": 0.7828282828282829,
+      "grad_norm": 0.6156490445137024,
+      "learning_rate": 6.282552083333334e-06,
+      "loss": 1.1477,
+      "mean_token_accuracy": 0.7468263059854507,
+      "num_tokens": 3753671.0,
+      "step": 620
+    },
+    {
+      "entropy": 1.1686612635850906,
+      "epoch": 0.7954545454545454,
+      "grad_norm": 0.6248748898506165,
+      "learning_rate": 6.217447916666667e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7478924334049225,
+      "num_tokens": 3813110.0,
+      "step": 630
+    },
+    {
+      "entropy": 1.1387953266501427,
+      "epoch": 0.8080808080808081,
+      "grad_norm": 0.6052266359329224,
+      "learning_rate": 6.152343750000001e-06,
+      "loss": 1.1118,
+      "mean_token_accuracy": 0.7538487210869789,
+      "num_tokens": 3873477.0,
+      "step": 640
+    },
+    {
+      "entropy": 1.1343814879655838,
+      "epoch": 0.8207070707070707,
+      "grad_norm": 0.6769536137580872,
+      "learning_rate": 6.087239583333335e-06,
+      "loss": 1.1108,
+      "mean_token_accuracy": 0.7548464313149452,
+      "num_tokens": 3932873.0,
+      "step": 650
+    },
+    {
+      "entropy": 1.1686519652605056,
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.6545736789703369,
+      "learning_rate": 6.022135416666667e-06,
+      "loss": 1.134,
+      "mean_token_accuracy": 0.7495195478200912,
+      "num_tokens": 3992754.0,
+      "step": 660
+    },
+    {
+      "entropy": 1.1596283346414566,
+      "epoch": 0.8459595959595959,
+      "grad_norm": 0.6192017793655396,
+      "learning_rate": 5.95703125e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7510297149419785,
+      "num_tokens": 4053540.0,
+      "step": 670
+    },
+    {
+      "entropy": 1.1484344542026519,
+      "epoch": 0.8585858585858586,
+      "grad_norm": 0.6520631909370422,
+      "learning_rate": 5.891927083333334e-06,
+      "loss": 1.1181,
+      "mean_token_accuracy": 0.751795919239521,
+      "num_tokens": 4113582.0,
+      "step": 680
+    },
+    {
+      "entropy": 1.1519750133156776,
+      "epoch": 0.8712121212121212,
+      "grad_norm": 0.6247655153274536,
+      "learning_rate": 5.826822916666667e-06,
+      "loss": 1.1204,
+      "mean_token_accuracy": 0.7506368085741997,
+      "num_tokens": 4174983.0,
+      "step": 690
+    },
+    {
+      "entropy": 1.1486561581492425,
+      "epoch": 0.8838383838383839,
+      "grad_norm": 0.620272159576416,
+      "learning_rate": 5.761718750000001e-06,
+      "loss": 1.1191,
+      "mean_token_accuracy": 0.7536391675472259,
+      "num_tokens": 4234465.0,
+      "step": 700
+    },
+    {
+      "entropy": 1.1504923462867738,
+      "epoch": 0.8964646464646465,
+      "grad_norm": 0.6308649182319641,
+      "learning_rate": 5.6966145833333344e-06,
+      "loss": 1.1224,
+      "mean_token_accuracy": 0.750646598637104,
+      "num_tokens": 4295955.0,
+      "step": 710
+    },
+    {
+      "entropy": 1.1561898440122604,
+      "epoch": 0.9090909090909091,
+      "grad_norm": 0.6629899740219116,
+      "learning_rate": 5.6315104166666665e-06,
+      "loss": 1.1238,
+      "mean_token_accuracy": 0.7507557719945908,
+      "num_tokens": 4357171.0,
+      "step": 720
+    },
+    {
+      "entropy": 1.1413449853658677,
+      "epoch": 0.9217171717171717,
+      "grad_norm": 0.5972346067428589,
+      "learning_rate": 5.56640625e-06,
+      "loss": 1.1038,
+      "mean_token_accuracy": 0.7554104939103127,
+      "num_tokens": 4417636.0,
+      "step": 730
+    },
+    {
+      "entropy": 1.1356119453907012,
+      "epoch": 0.9343434343434344,
+      "grad_norm": 0.6356479525566101,
+      "learning_rate": 5.501302083333334e-06,
+      "loss": 1.1005,
+      "mean_token_accuracy": 0.7565629109740257,
+      "num_tokens": 4477294.0,
+      "step": 740
+    },
+    {
+      "entropy": 1.1633600294589996,
+      "epoch": 0.946969696969697,
+      "grad_norm": 0.6416464447975159,
+      "learning_rate": 5.436197916666667e-06,
+      "loss": 1.1225,
+      "mean_token_accuracy": 0.7515855401754379,
+      "num_tokens": 4537503.0,
+      "step": 750
+    },
+    {
+      "entropy": 1.1527763932943345,
+      "epoch": 0.9595959595959596,
+      "grad_norm": 0.6126084327697754,
+      "learning_rate": 5.3710937500000005e-06,
+      "loss": 1.1184,
+      "mean_token_accuracy": 0.7526160582900048,
+      "num_tokens": 4598778.0,
+      "step": 760
+    },
+    {
+      "entropy": 1.1397768080234527,
+      "epoch": 0.9722222222222222,
+      "grad_norm": 0.6359922289848328,
+      "learning_rate": 5.305989583333334e-06,
+      "loss": 1.1144,
+      "mean_token_accuracy": 0.7548302739858628,
+      "num_tokens": 4658978.0,
+      "step": 770
+    },
+    {
+      "entropy": 1.1569962561130525,
+      "epoch": 0.9848484848484849,
+      "grad_norm": 0.6260409951210022,
+      "learning_rate": 5.240885416666667e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7512885302305221,
+      "num_tokens": 4720500.0,
+      "step": 780
+    },
+    {
+      "entropy": 1.1509152203798294,
+      "epoch": 0.9974747474747475,
+      "grad_norm": 0.6293452978134155,
+      "learning_rate": 5.17578125e-06,
+      "loss": 1.1227,
+      "mean_token_accuracy": 0.7519564241170883,
+      "num_tokens": 4781612.0,
+      "step": 790
+    },
+    {
+      "entropy": 1.1399024561047555,
+      "epoch": 1.0101010101010102,
+      "grad_norm": 0.664761483669281,
+      "learning_rate": 5.110677083333334e-06,
+      "loss": 1.1034,
+      "mean_token_accuracy": 0.7526706486940384,
+      "num_tokens": 4841359.0,
+      "step": 800
+    },
+    {
+      "entropy": 1.120214229822159,
+      "epoch": 1.0227272727272727,
+      "grad_norm": 0.5934865474700928,
+      "learning_rate": 5.045572916666667e-06,
+      "loss": 1.0857,
+      "mean_token_accuracy": 0.7593594208359719,
+      "num_tokens": 4901016.0,
+      "step": 810
+    },
+    {
+      "entropy": 1.1430341199040412,
+      "epoch": 1.0353535353535352,
+      "grad_norm": 0.6040735840797424,
+      "learning_rate": 4.98046875e-06,
+      "loss": 1.1165,
+      "mean_token_accuracy": 0.7525988414883613,
+      "num_tokens": 4961646.0,
+      "step": 820
+    },
+    {
+      "entropy": 1.1245349109172822,
+      "epoch": 1.047979797979798,
+      "grad_norm": 0.6277610063552856,
+      "learning_rate": 4.915364583333333e-06,
+      "loss": 1.0851,
+      "mean_token_accuracy": 0.7577921718358993,
+      "num_tokens": 5022365.0,
+      "step": 830
+    },
+    {
+      "entropy": 1.1204675793647767,
+      "epoch": 1.0606060606060606,
+      "grad_norm": 0.6260582804679871,
+      "learning_rate": 4.850260416666667e-06,
+      "loss": 1.0813,
+      "mean_token_accuracy": 0.7580071151256561,
+      "num_tokens": 5081972.0,
+      "step": 840
+    },
+    {
+      "entropy": 1.1222782507538795,
+      "epoch": 1.0732323232323233,
+      "grad_norm": 0.6023226976394653,
+      "learning_rate": 4.785156250000001e-06,
+      "loss": 1.0922,
+      "mean_token_accuracy": 0.7566258609294891,
+      "num_tokens": 5142184.0,
+      "step": 850
+    },
+    {
+      "entropy": 1.1227335944771766,
+      "epoch": 1.0858585858585859,
+      "grad_norm": 0.6206791996955872,
+      "learning_rate": 4.7200520833333336e-06,
+      "loss": 1.0994,
+      "mean_token_accuracy": 0.7540625646710396,
+      "num_tokens": 5203020.0,
+      "step": 860
+    },
+    {
+      "entropy": 1.1352888554334641,
+      "epoch": 1.0984848484848484,
+      "grad_norm": 0.6301055550575256,
+      "learning_rate": 4.654947916666667e-06,
+      "loss": 1.0958,
+      "mean_token_accuracy": 0.7562039017677307,
+      "num_tokens": 5263291.0,
+      "step": 870
+    },
+    {
+      "entropy": 1.1120088309049607,
+      "epoch": 1.1111111111111112,
+      "grad_norm": 0.6210020780563354,
+      "learning_rate": 4.58984375e-06,
+      "loss": 1.0793,
+      "mean_token_accuracy": 0.7590557768940925,
+      "num_tokens": 5323972.0,
+      "step": 880
+    },
+    {
+      "entropy": 1.0996666207909584,
+      "epoch": 1.1237373737373737,
+      "grad_norm": 0.6332690715789795,
+      "learning_rate": 4.524739583333334e-06,
+      "loss": 1.0717,
+      "mean_token_accuracy": 0.7615471586585045,
+      "num_tokens": 5383911.0,
+      "step": 890
+    },
+    {
+      "entropy": 1.127107810974121,
+      "epoch": 1.1363636363636362,
+      "grad_norm": 0.6505516767501831,
+      "learning_rate": 4.459635416666668e-06,
+      "loss": 1.1027,
+      "mean_token_accuracy": 0.7562421515583992,
+      "num_tokens": 5445417.0,
+      "step": 900
+    },
+    {
+      "entropy": 1.129740473628044,
+      "epoch": 1.148989898989899,
+      "grad_norm": 0.6406158804893494,
+      "learning_rate": 4.3945312500000005e-06,
+      "loss": 1.0879,
+      "mean_token_accuracy": 0.7587148532271385,
+      "num_tokens": 5505455.0,
+      "step": 910
+    },
+    {
+      "entropy": 1.1167259424924851,
+      "epoch": 1.1616161616161615,
+      "grad_norm": 0.6297397613525391,
+      "learning_rate": 4.329427083333333e-06,
+      "loss": 1.0752,
+      "mean_token_accuracy": 0.7604142814874649,
+      "num_tokens": 5565311.0,
+      "step": 920
+    },
+    {
+      "entropy": 1.1037891641259194,
+      "epoch": 1.1742424242424243,
+      "grad_norm": 0.6490073204040527,
+      "learning_rate": 4.264322916666667e-06,
+      "loss": 1.0686,
+      "mean_token_accuracy": 0.7610052570700645,
+      "num_tokens": 5625358.0,
+      "step": 930
+    },
+    {
+      "entropy": 1.1061881184577942,
+      "epoch": 1.1868686868686869,
+      "grad_norm": 0.6366387009620667,
+      "learning_rate": 4.19921875e-06,
+      "loss": 1.0868,
+      "mean_token_accuracy": 0.7574937298893929,
+      "num_tokens": 5686421.0,
+      "step": 940
+    },
+    {
+      "entropy": 1.1124324068427085,
+      "epoch": 1.1994949494949494,
+      "grad_norm": 0.6556055545806885,
+      "learning_rate": 4.134114583333334e-06,
+      "loss": 1.0694,
+      "mean_token_accuracy": 0.7602224007248879,
+      "num_tokens": 5745891.0,
+      "step": 950
+    },
+    {
+      "entropy": 1.1175200879573821,
+      "epoch": 1.2121212121212122,
+      "grad_norm": 0.6404849886894226,
+      "learning_rate": 4.0690104166666675e-06,
+      "loss": 1.081,
+      "mean_token_accuracy": 0.7568994402885437,
+      "num_tokens": 5806078.0,
+      "step": 960
+    },
+    {
+      "entropy": 1.1186978340148925,
+      "epoch": 1.2247474747474747,
+      "grad_norm": 0.6227584481239319,
+      "learning_rate": 4.00390625e-06,
+      "loss": 1.0791,
+      "mean_token_accuracy": 0.759756401181221,
+      "num_tokens": 5866369.0,
+      "step": 970
+    },
+    {
+      "entropy": 1.122128139436245,
+      "epoch": 1.2373737373737375,
+      "grad_norm": 0.6616361141204834,
+      "learning_rate": 3.938802083333333e-06,
+      "loss": 1.0937,
+      "mean_token_accuracy": 0.7582718566060066,
+      "num_tokens": 5926217.0,
+      "step": 980
+    },
+    {
+      "entropy": 1.122861033678055,
+      "epoch": 1.25,
+      "grad_norm": 0.6384168267250061,
+      "learning_rate": 3.873697916666667e-06,
+      "loss": 1.0978,
+      "mean_token_accuracy": 0.7549964562058449,
+      "num_tokens": 5987666.0,
+      "step": 990
+    },
+    {
+      "entropy": 1.1277505576610565,
+      "epoch": 1.2626262626262625,
+      "grad_norm": 0.6038117408752441,
+      "learning_rate": 3.8085937500000002e-06,
+      "loss": 1.0952,
+      "mean_token_accuracy": 0.755272176861763,
+      "num_tokens": 6048708.0,
+      "step": 1000
+    },
+    {
+      "entropy": 1.1120157346129418,
+      "epoch": 1.2752525252525253,
+      "grad_norm": 0.6418159604072571,
+      "learning_rate": 3.7434895833333336e-06,
+      "loss": 1.078,
+      "mean_token_accuracy": 0.7594122514128685,
+      "num_tokens": 6109652.0,
+      "step": 1010
+    },
+    {
+      "entropy": 1.101425115764141,
+      "epoch": 1.2878787878787878,
+      "grad_norm": 0.6218425035476685,
+      "learning_rate": 3.6783854166666673e-06,
+      "loss": 1.0688,
+      "mean_token_accuracy": 0.7604865297675133,
+      "num_tokens": 6169125.0,
+      "step": 1020
+    },
+    {
+      "entropy": 1.1007713869214057,
+      "epoch": 1.3005050505050506,
+      "grad_norm": 0.6429149508476257,
+      "learning_rate": 3.61328125e-06,
+      "loss": 1.0581,
+      "mean_token_accuracy": 0.7621071562170982,
+      "num_tokens": 6230303.0,
+      "step": 1030
+    },
+    {
+      "entropy": 1.1094096556305886,
+      "epoch": 1.3131313131313131,
+      "grad_norm": 0.6489748358726501,
+      "learning_rate": 3.5481770833333335e-06,
+      "loss": 1.0715,
+      "mean_token_accuracy": 0.7599423810839653,
+      "num_tokens": 6291396.0,
+      "step": 1040
+    },
+    {
+      "entropy": 1.0827289715409278,
+      "epoch": 1.3257575757575757,
+      "grad_norm": 0.6485461592674255,
+      "learning_rate": 3.483072916666667e-06,
+      "loss": 1.0584,
+      "mean_token_accuracy": 0.7630694910883904,
+      "num_tokens": 6351579.0,
+      "step": 1050
+    },
+    {
+      "entropy": 1.114325873553753,
+      "epoch": 1.3383838383838385,
+      "grad_norm": 0.6261104941368103,
+      "learning_rate": 3.41796875e-06,
+      "loss": 1.0764,
+      "mean_token_accuracy": 0.7585488513112069,
+      "num_tokens": 6411662.0,
+      "step": 1060
+    },
+    {
+      "entropy": 1.1271554425358772,
+      "epoch": 1.351010101010101,
+      "grad_norm": 0.6522034406661987,
+      "learning_rate": 3.3528645833333334e-06,
+      "loss": 1.0902,
+      "mean_token_accuracy": 0.7562535598874092,
+      "num_tokens": 6473505.0,
+      "step": 1070
+    },
+    {
+      "entropy": 1.1013643085956573,
+      "epoch": 1.3636363636363638,
+      "grad_norm": 0.6176674962043762,
+      "learning_rate": 3.287760416666667e-06,
+      "loss": 1.065,
+      "mean_token_accuracy": 0.763075165450573,
+      "num_tokens": 6533580.0,
+      "step": 1080
+    },
+    {
+      "entropy": 1.098090337216854,
+      "epoch": 1.3762626262626263,
+      "grad_norm": 0.6540253758430481,
+      "learning_rate": 3.2226562500000004e-06,
+      "loss": 1.0596,
+      "mean_token_accuracy": 0.7616770043969154,
+      "num_tokens": 6593481.0,
+      "step": 1090
+    },
+    {
+      "entropy": 1.1176372200250626,
+      "epoch": 1.3888888888888888,
+      "grad_norm": 0.6754550933837891,
+      "learning_rate": 3.1575520833333333e-06,
+      "loss": 1.0861,
+      "mean_token_accuracy": 0.7573029339313507,
+      "num_tokens": 6653967.0,
+      "step": 1100
+    },
+    {
+      "entropy": 1.1040414482355119,
+      "epoch": 1.4015151515151514,
+      "grad_norm": 0.6022531986236572,
+      "learning_rate": 3.092447916666667e-06,
+      "loss": 1.0573,
+      "mean_token_accuracy": 0.7612267225980759,
+      "num_tokens": 6714685.0,
+      "step": 1110
+    },
+    {
+      "entropy": 1.0926991075277328,
+      "epoch": 1.4141414141414141,
+      "grad_norm": 0.6621010303497314,
+      "learning_rate": 3.0273437500000003e-06,
+      "loss": 1.0637,
+      "mean_token_accuracy": 0.7612977519631385,
+      "num_tokens": 6774659.0,
+      "step": 1120
+    },
+    {
+      "entropy": 1.1095534324645997,
+      "epoch": 1.4267676767676767,
+      "grad_norm": 0.62503981590271,
+      "learning_rate": 2.962239583333333e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7618053883314133,
+      "num_tokens": 6834579.0,
+      "step": 1130
+    },
+    {
+      "entropy": 1.117809349298477,
+      "epoch": 1.4393939393939394,
+      "grad_norm": 0.6527109742164612,
+      "learning_rate": 2.897135416666667e-06,
+      "loss": 1.0747,
+      "mean_token_accuracy": 0.759482853114605,
+      "num_tokens": 6894074.0,
+      "step": 1140
+    },
+    {
+      "entropy": 1.1005077749490737,
+      "epoch": 1.452020202020202,
+      "grad_norm": 0.6720954775810242,
+      "learning_rate": 2.8320312500000002e-06,
+      "loss": 1.0607,
+      "mean_token_accuracy": 0.7621246844530105,
+      "num_tokens": 6953870.0,
+      "step": 1150
+    },
+    {
+      "entropy": 1.1236482918262483,
+      "epoch": 1.4646464646464645,
+      "grad_norm": 0.658524215221405,
+      "learning_rate": 2.7669270833333335e-06,
+      "loss": 1.0884,
+      "mean_token_accuracy": 0.7560836613178253,
+      "num_tokens": 7014553.0,
+      "step": 1160
+    },
+    {
+      "entropy": 1.1116504594683647,
+      "epoch": 1.4772727272727273,
+      "grad_norm": 0.6261802911758423,
+      "learning_rate": 2.7018229166666673e-06,
+      "loss": 1.0659,
+      "mean_token_accuracy": 0.7597616642713547,
+      "num_tokens": 7076291.0,
+      "step": 1170
+    },
+    {
+      "entropy": 1.073892480134964,
+      "epoch": 1.4898989898989898,
+      "grad_norm": 0.6310375332832336,
+      "learning_rate": 2.63671875e-06,
+      "loss": 1.0524,
+      "mean_token_accuracy": 0.7628733053803444,
+      "num_tokens": 7137305.0,
+      "step": 1180
+    },
+    {
+      "entropy": 1.0975843235850333,
+      "epoch": 1.5025252525252526,
+      "grad_norm": 0.638482391834259,
+      "learning_rate": 2.5716145833333334e-06,
+      "loss": 1.0679,
+      "mean_token_accuracy": 0.7603248566389084,
+      "num_tokens": 7198239.0,
+      "step": 1190
+    },
+    {
+      "entropy": 1.0986508697271347,
+      "epoch": 1.5151515151515151,
+      "grad_norm": 0.640065610408783,
+      "learning_rate": 2.506510416666667e-06,
+      "loss": 1.0666,
+      "mean_token_accuracy": 0.7622530281543731,
+      "num_tokens": 7257847.0,
+      "step": 1200
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1584,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.110405675434312e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eeb71c14deb91ac5fd11522db45cb3275c9164415fcbefc9d00cac27a27f0a3
+size 6417

checkpoint-1400/README.md

@@ -0,0 +1,209 @@
+---
+base_model: CohereLabs/aya-expanse-8b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:CohereLabs/aya-expanse-8b
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-1400/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "CohereLabs/aya-expanse-8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "o_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1400/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee5846528f7fce0a4cf0270a4dc36986f041f0754c9eefb430c289323f104d4c
+size 167832240

checkpoint-1400/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true %}{% set loop_messages = messages %}{% set system_message = 'You are Aya, a brilliant, sophisticated, multilingual AI-assistant trained to assist human users by providing thorough responses. You are able to interact and respond to questions in 23 languages and you are powered by a multilingual model built by Cohere For AI.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% if system_message != false %}{{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% elif message['role'] == 'assistant' %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'  + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }}{% endif %}

checkpoint-1400/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74c2cee5bcd67acd6c6931a316bfc1a8a46e05c52a8664bd86dddda4f410978a
+size 335929123

checkpoint-1400/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7dff23631b1d1432ccf216a29a64a0894fa08d99cdc2ae64b29a790d479eb958
+size 14645

checkpoint-1400/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:026b9ae9780a61b08bc12971cc3b0b0cd0ad05bdb25e4c31d15359a55fdf2292
+size 1465

checkpoint-1400/special_tokens_map.json

@@ -0,0 +1,17 @@
+{
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<PAD>"
+}

checkpoint-1400/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:345ccf04a5257f473e331715ecc69365c5ac8fc2490923fe7155560af809ec1a
+size 20124090

checkpoint-1400/tokenizer_config.json

@@ -0,0 +1,317 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<CLS>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<SEP>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<EOP_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255000": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255001": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255002": {
+      "content": "<|YES_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255003": {
+      "content": "<|NO_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255004": {
+      "content": "<|GOOD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255005": {
+      "content": "<|BAD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255006": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255007": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255008": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255009": {
+      "content": "<|USER_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255010": {
+      "content": "<|USER_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255011": {
+      "content": "<|USER_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255012": {
+      "content": "<|USER_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255013": {
+      "content": "<|USER_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255014": {
+      "content": "<|USER_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255015": {
+      "content": "<|USER_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255016": {
+      "content": "<|USER_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255017": {
+      "content": "<|USER_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255018": {
+      "content": "<|USER_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255019": {
+      "content": "<|EXTRA_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255020": {
+      "content": "<|EXTRA_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255021": {
+      "content": "<|EXTRA_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255022": {
+      "content": "<|EXTRA_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255023": {
+      "content": "<|EXTRA_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255024": {
+      "content": "<|EXTRA_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255025": {
+      "content": "<|EXTRA_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255026": {
+      "content": "<|EXTRA_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255027": {
+      "content": "<|EXTRA_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255028": {
+      "content": "<|EXTRA_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<BOS_TOKEN>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|END_OF_TURN_TOKEN|>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "merges_file": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "CohereTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false,
+  "vocab_file": null
+}

checkpoint-1400/trainer_state.json

@@ -0,0 +1,1434 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.7676767676767677,
+  "eval_steps": 200,
+  "global_step": 1400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.4040649354457857,
+      "epoch": 0.012626262626262626,
+      "grad_norm": 4.66565465927124,
+      "learning_rate": 1.8750000000000003e-06,
+      "loss": 3.6021,
+      "mean_token_accuracy": 0.4221017129719257,
+      "num_tokens": 61199.0,
+      "step": 10
+    },
+    {
+      "entropy": 2.3836746215820312,
+      "epoch": 0.025252525252525252,
+      "grad_norm": 3.8161869049072266,
+      "learning_rate": 3.958333333333333e-06,
+      "loss": 3.3432,
+      "mean_token_accuracy": 0.44042530804872515,
+      "num_tokens": 122423.0,
+      "step": 20
+    },
+    {
+      "entropy": 2.355724626779556,
+      "epoch": 0.03787878787878788,
+      "grad_norm": 3.8800699710845947,
+      "learning_rate": 6.041666666666667e-06,
+      "loss": 2.9033,
+      "mean_token_accuracy": 0.48426677361130716,
+      "num_tokens": 182649.0,
+      "step": 30
+    },
+    {
+      "entropy": 2.092331054806709,
+      "epoch": 0.050505050505050504,
+      "grad_norm": 2.8217720985412598,
+      "learning_rate": 8.125000000000001e-06,
+      "loss": 2.356,
+      "mean_token_accuracy": 0.5772452697157859,
+      "num_tokens": 243049.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.6766322344541549,
+      "epoch": 0.06313131313131314,
+      "grad_norm": 1.4623568058013916,
+      "learning_rate": 9.993489583333334e-06,
+      "loss": 1.8899,
+      "mean_token_accuracy": 0.6480962842702865,
+      "num_tokens": 304326.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.5568815559148788,
+      "epoch": 0.07575757575757576,
+      "grad_norm": 1.171562671661377,
+      "learning_rate": 9.928385416666668e-06,
+      "loss": 1.677,
+      "mean_token_accuracy": 0.6776855796575546,
+      "num_tokens": 364866.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.48199902176857,
+      "epoch": 0.08838383838383838,
+      "grad_norm": 0.9904961585998535,
+      "learning_rate": 9.863281250000001e-06,
+      "loss": 1.5337,
+      "mean_token_accuracy": 0.697019773721695,
+      "num_tokens": 423478.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.497376424074173,
+      "epoch": 0.10101010101010101,
+      "grad_norm": 0.9454260468482971,
+      "learning_rate": 9.798177083333335e-06,
+      "loss": 1.4953,
+      "mean_token_accuracy": 0.6976823702454567,
+      "num_tokens": 483659.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.4664768785238267,
+      "epoch": 0.11363636363636363,
+      "grad_norm": 0.8955270648002625,
+      "learning_rate": 9.733072916666667e-06,
+      "loss": 1.4356,
+      "mean_token_accuracy": 0.7069446608424187,
+      "num_tokens": 544453.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.4269085675477982,
+      "epoch": 0.12626262626262627,
+      "grad_norm": 0.9242203235626221,
+      "learning_rate": 9.66796875e-06,
+      "loss": 1.4106,
+      "mean_token_accuracy": 0.7122392952442169,
+      "num_tokens": 604546.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.4060751020908355,
+      "epoch": 0.1388888888888889,
+      "grad_norm": 0.8968560695648193,
+      "learning_rate": 9.602864583333335e-06,
+      "loss": 1.3487,
+      "mean_token_accuracy": 0.7178552970290184,
+      "num_tokens": 664860.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.4018951296806335,
+      "epoch": 0.15151515151515152,
+      "grad_norm": 0.9047113656997681,
+      "learning_rate": 9.537760416666667e-06,
+      "loss": 1.3347,
+      "mean_token_accuracy": 0.7208079636096955,
+      "num_tokens": 725022.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.3809731483459473,
+      "epoch": 0.16414141414141414,
+      "grad_norm": 0.8915444016456604,
+      "learning_rate": 9.47265625e-06,
+      "loss": 1.3155,
+      "mean_token_accuracy": 0.7267520889639855,
+      "num_tokens": 785586.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.3699676394462585,
+      "epoch": 0.17676767676767677,
+      "grad_norm": 0.8574295043945312,
+      "learning_rate": 9.407552083333334e-06,
+      "loss": 1.3016,
+      "mean_token_accuracy": 0.7266673430800438,
+      "num_tokens": 845790.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.3425012439489366,
+      "epoch": 0.1893939393939394,
+      "grad_norm": 0.8231800198554993,
+      "learning_rate": 9.342447916666668e-06,
+      "loss": 1.2823,
+      "mean_token_accuracy": 0.7277117937803268,
+      "num_tokens": 905842.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.3314216613769532,
+      "epoch": 0.20202020202020202,
+      "grad_norm": 0.8166369795799255,
+      "learning_rate": 9.277343750000001e-06,
+      "loss": 1.2917,
+      "mean_token_accuracy": 0.7278609350323677,
+      "num_tokens": 966487.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.30389544069767,
+      "epoch": 0.21464646464646464,
+      "grad_norm": 0.7738587260246277,
+      "learning_rate": 9.212239583333335e-06,
+      "loss": 1.2481,
+      "mean_token_accuracy": 0.7335339426994324,
+      "num_tokens": 1025558.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.311449444293976,
+      "epoch": 0.22727272727272727,
+      "grad_norm": 0.7718328833580017,
+      "learning_rate": 9.147135416666667e-06,
+      "loss": 1.2643,
+      "mean_token_accuracy": 0.7285080313682556,
+      "num_tokens": 1086677.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.3079361200332642,
+      "epoch": 0.2398989898989899,
+      "grad_norm": 0.7341915369033813,
+      "learning_rate": 9.082031250000001e-06,
+      "loss": 1.2641,
+      "mean_token_accuracy": 0.729550538957119,
+      "num_tokens": 1147885.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.2794219702482224,
+      "epoch": 0.25252525252525254,
+      "grad_norm": 0.748540997505188,
+      "learning_rate": 9.016927083333335e-06,
+      "loss": 1.2397,
+      "mean_token_accuracy": 0.7351120054721832,
+      "num_tokens": 1207321.0,
+      "step": 200
+    },
+    {
+      "entropy": 1.294454461336136,
+      "epoch": 0.26515151515151514,
+      "grad_norm": 0.7585553526878357,
+      "learning_rate": 8.951822916666667e-06,
+      "loss": 1.2489,
+      "mean_token_accuracy": 0.7322148531675339,
+      "num_tokens": 1267837.0,
+      "step": 210
+    },
+    {
+      "entropy": 1.2862246632575989,
+      "epoch": 0.2777777777777778,
+      "grad_norm": 0.6937864422798157,
+      "learning_rate": 8.88671875e-06,
+      "loss": 1.2383,
+      "mean_token_accuracy": 0.7358245223760604,
+      "num_tokens": 1328436.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.2531811505556107,
+      "epoch": 0.2904040404040404,
+      "grad_norm": 0.6792387366294861,
+      "learning_rate": 8.821614583333334e-06,
+      "loss": 1.2007,
+      "mean_token_accuracy": 0.7360369265079498,
+      "num_tokens": 1389877.0,
+      "step": 230
+    },
+    {
+      "entropy": 1.2815489560365676,
+      "epoch": 0.30303030303030304,
+      "grad_norm": 0.6865427494049072,
+      "learning_rate": 8.756510416666666e-06,
+      "loss": 1.2474,
+      "mean_token_accuracy": 0.7304804190993309,
+      "num_tokens": 1450927.0,
+      "step": 240
+    },
+    {
+      "entropy": 1.2568059146404267,
+      "epoch": 0.31565656565656564,
+      "grad_norm": 0.669840395450592,
+      "learning_rate": 8.69140625e-06,
+      "loss": 1.2172,
+      "mean_token_accuracy": 0.7385278165340423,
+      "num_tokens": 1511174.0,
+      "step": 250
+    },
+    {
+      "entropy": 1.254868358373642,
+      "epoch": 0.3282828282828283,
+      "grad_norm": 0.6434893012046814,
+      "learning_rate": 8.626302083333334e-06,
+      "loss": 1.213,
+      "mean_token_accuracy": 0.7380544006824493,
+      "num_tokens": 1570570.0,
+      "step": 260
+    },
+    {
+      "entropy": 1.2532441645860672,
+      "epoch": 0.3409090909090909,
+      "grad_norm": 0.6034978032112122,
+      "learning_rate": 8.561197916666667e-06,
+      "loss": 1.2116,
+      "mean_token_accuracy": 0.7382062628865242,
+      "num_tokens": 1630930.0,
+      "step": 270
+    },
+    {
+      "entropy": 1.2605280816555022,
+      "epoch": 0.35353535353535354,
+      "grad_norm": 0.6371450424194336,
+      "learning_rate": 8.496093750000001e-06,
+      "loss": 1.2313,
+      "mean_token_accuracy": 0.7328852489590645,
+      "num_tokens": 1692624.0,
+      "step": 280
+    },
+    {
+      "entropy": 1.248606452345848,
+      "epoch": 0.3661616161616162,
+      "grad_norm": 0.6300278306007385,
+      "learning_rate": 8.430989583333335e-06,
+      "loss": 1.2195,
+      "mean_token_accuracy": 0.7370449885725975,
+      "num_tokens": 1754213.0,
+      "step": 290
+    },
+    {
+      "entropy": 1.2246102809906005,
+      "epoch": 0.3787878787878788,
+      "grad_norm": 0.6430155634880066,
+      "learning_rate": 8.365885416666667e-06,
+      "loss": 1.1845,
+      "mean_token_accuracy": 0.7439196646213532,
+      "num_tokens": 1813841.0,
+      "step": 300
+    },
+    {
+      "entropy": 1.2188815206289292,
+      "epoch": 0.39141414141414144,
+      "grad_norm": 0.6395701766014099,
+      "learning_rate": 8.30078125e-06,
+      "loss": 1.1856,
+      "mean_token_accuracy": 0.7422183871269226,
+      "num_tokens": 1873568.0,
+      "step": 310
+    },
+    {
+      "entropy": 1.234528934955597,
+      "epoch": 0.40404040404040403,
+      "grad_norm": 0.6168740391731262,
+      "learning_rate": 8.235677083333334e-06,
+      "loss": 1.1951,
+      "mean_token_accuracy": 0.7380509555339814,
+      "num_tokens": 1935518.0,
+      "step": 320
+    },
+    {
+      "entropy": 1.226365676522255,
+      "epoch": 0.4166666666666667,
+      "grad_norm": 0.611132800579071,
+      "learning_rate": 8.170572916666666e-06,
+      "loss": 1.2078,
+      "mean_token_accuracy": 0.739974245429039,
+      "num_tokens": 1995604.0,
+      "step": 330
+    },
+    {
+      "entropy": 1.2132183194160462,
+      "epoch": 0.4292929292929293,
+      "grad_norm": 0.6103131771087646,
+      "learning_rate": 8.10546875e-06,
+      "loss": 1.1642,
+      "mean_token_accuracy": 0.7444137379527092,
+      "num_tokens": 2056484.0,
+      "step": 340
+    },
+    {
+      "entropy": 1.223158246278763,
+      "epoch": 0.44191919191919193,
+      "grad_norm": 0.6188805103302002,
+      "learning_rate": 8.040364583333334e-06,
+      "loss": 1.2001,
+      "mean_token_accuracy": 0.7385613292455673,
+      "num_tokens": 2118437.0,
+      "step": 350
+    },
+    {
+      "entropy": 1.2154075980186463,
+      "epoch": 0.45454545454545453,
+      "grad_norm": 0.6238694190979004,
+      "learning_rate": 7.975260416666668e-06,
+      "loss": 1.1848,
+      "mean_token_accuracy": 0.7404153689742088,
+      "num_tokens": 2179333.0,
+      "step": 360
+    },
+    {
+      "entropy": 1.197928261756897,
+      "epoch": 0.4671717171717172,
+      "grad_norm": 0.6028566956520081,
+      "learning_rate": 7.910156250000001e-06,
+      "loss": 1.1597,
+      "mean_token_accuracy": 0.7475010469555855,
+      "num_tokens": 2239604.0,
+      "step": 370
+    },
+    {
+      "entropy": 1.189855706691742,
+      "epoch": 0.4797979797979798,
+      "grad_norm": 0.6569434404373169,
+      "learning_rate": 7.845052083333335e-06,
+      "loss": 1.1805,
+      "mean_token_accuracy": 0.7427218139171601,
+      "num_tokens": 2300936.0,
+      "step": 380
+    },
+    {
+      "entropy": 1.2076493889093398,
+      "epoch": 0.49242424242424243,
+      "grad_norm": 0.6351733207702637,
+      "learning_rate": 7.779947916666667e-06,
+      "loss": 1.1759,
+      "mean_token_accuracy": 0.7408407002687454,
+      "num_tokens": 2361539.0,
+      "step": 390
+    },
+    {
+      "entropy": 1.2223081022500992,
+      "epoch": 0.5050505050505051,
+      "grad_norm": 0.6327986121177673,
+      "learning_rate": 7.71484375e-06,
+      "loss": 1.1874,
+      "mean_token_accuracy": 0.7407250568270684,
+      "num_tokens": 2422162.0,
+      "step": 400
+    },
+    {
+      "entropy": 1.2013412863016129,
+      "epoch": 0.5176767676767676,
+      "grad_norm": 0.622104823589325,
+      "learning_rate": 7.649739583333334e-06,
+      "loss": 1.1726,
+      "mean_token_accuracy": 0.744242025911808,
+      "num_tokens": 2483447.0,
+      "step": 410
+    },
+    {
+      "entropy": 1.2116613179445266,
+      "epoch": 0.5303030303030303,
+      "grad_norm": 0.637651264667511,
+      "learning_rate": 7.5846354166666665e-06,
+      "loss": 1.1838,
+      "mean_token_accuracy": 0.7405030101537704,
+      "num_tokens": 2544848.0,
+      "step": 420
+    },
+    {
+      "entropy": 1.2024697184562683,
+      "epoch": 0.5429292929292929,
+      "grad_norm": 0.6252374649047852,
+      "learning_rate": 7.51953125e-06,
+      "loss": 1.1681,
+      "mean_token_accuracy": 0.7458183988928795,
+      "num_tokens": 2605232.0,
+      "step": 430
+    },
+    {
+      "entropy": 1.1797083109617232,
+      "epoch": 0.5555555555555556,
+      "grad_norm": 0.6502755284309387,
+      "learning_rate": 7.454427083333334e-06,
+      "loss": 1.1452,
+      "mean_token_accuracy": 0.7477341219782829,
+      "num_tokens": 2664276.0,
+      "step": 440
+    },
+    {
+      "entropy": 1.1964200481772422,
+      "epoch": 0.5681818181818182,
+      "grad_norm": 0.639979362487793,
+      "learning_rate": 7.389322916666667e-06,
+      "loss": 1.1665,
+      "mean_token_accuracy": 0.7431837096810341,
+      "num_tokens": 2724073.0,
+      "step": 450
+    },
+    {
+      "entropy": 1.1795272737741471,
+      "epoch": 0.5808080808080808,
+      "grad_norm": 0.6212354302406311,
+      "learning_rate": 7.3242187500000006e-06,
+      "loss": 1.1529,
+      "mean_token_accuracy": 0.7479098170995713,
+      "num_tokens": 2784262.0,
+      "step": 460
+    },
+    {
+      "entropy": 1.1908618807792664,
+      "epoch": 0.5934343434343434,
+      "grad_norm": 0.6528693437576294,
+      "learning_rate": 7.259114583333334e-06,
+      "loss": 1.1678,
+      "mean_token_accuracy": 0.745174677670002,
+      "num_tokens": 2843804.0,
+      "step": 470
+    },
+    {
+      "entropy": 1.1862946093082427,
+      "epoch": 0.6060606060606061,
+      "grad_norm": 0.639481246471405,
+      "learning_rate": 7.194010416666667e-06,
+      "loss": 1.1565,
+      "mean_token_accuracy": 0.7461866185069084,
+      "num_tokens": 2903408.0,
+      "step": 480
+    },
+    {
+      "entropy": 1.151743534207344,
+      "epoch": 0.6186868686868687,
+      "grad_norm": 0.6332777142524719,
+      "learning_rate": 7.128906250000001e-06,
+      "loss": 1.1251,
+      "mean_token_accuracy": 0.7535199671983719,
+      "num_tokens": 2963401.0,
+      "step": 490
+    },
+    {
+      "entropy": 1.1778477430343628,
+      "epoch": 0.6313131313131313,
+      "grad_norm": 0.5991836190223694,
+      "learning_rate": 7.063802083333335e-06,
+      "loss": 1.1407,
+      "mean_token_accuracy": 0.7491110354661942,
+      "num_tokens": 3023530.0,
+      "step": 500
+    },
+    {
+      "entropy": 1.2023035794496537,
+      "epoch": 0.6439393939393939,
+      "grad_norm": 0.6293458938598633,
+      "learning_rate": 6.998697916666667e-06,
+      "loss": 1.1724,
+      "mean_token_accuracy": 0.7405256554484367,
+      "num_tokens": 3085225.0,
+      "step": 510
+    },
+    {
+      "entropy": 1.1997334092855454,
+      "epoch": 0.6565656565656566,
+      "grad_norm": 0.6213802695274353,
+      "learning_rate": 6.93359375e-06,
+      "loss": 1.1604,
+      "mean_token_accuracy": 0.7435309410095214,
+      "num_tokens": 3145749.0,
+      "step": 520
+    },
+    {
+      "entropy": 1.1643184214830398,
+      "epoch": 0.6691919191919192,
+      "grad_norm": 0.6495156288146973,
+      "learning_rate": 6.868489583333334e-06,
+      "loss": 1.1381,
+      "mean_token_accuracy": 0.7496751576662064,
+      "num_tokens": 3205761.0,
+      "step": 530
+    },
+    {
+      "entropy": 1.1837532848119736,
+      "epoch": 0.6818181818181818,
+      "grad_norm": 0.6004510521888733,
+      "learning_rate": 6.803385416666667e-06,
+      "loss": 1.1563,
+      "mean_token_accuracy": 0.7447851061820984,
+      "num_tokens": 3267024.0,
+      "step": 540
+    },
+    {
+      "entropy": 1.201677542924881,
+      "epoch": 0.6944444444444444,
+      "grad_norm": 0.607467532157898,
+      "learning_rate": 6.738281250000001e-06,
+      "loss": 1.1776,
+      "mean_token_accuracy": 0.7406648993492126,
+      "num_tokens": 3329070.0,
+      "step": 550
+    },
+    {
+      "entropy": 1.1659786373376846,
+      "epoch": 0.7070707070707071,
+      "grad_norm": 0.6079947352409363,
+      "learning_rate": 6.6731770833333345e-06,
+      "loss": 1.1298,
+      "mean_token_accuracy": 0.7505511298775673,
+      "num_tokens": 3389505.0,
+      "step": 560
+    },
+    {
+      "entropy": 1.1714913487434386,
+      "epoch": 0.7196969696969697,
+      "grad_norm": 0.6534572839736938,
+      "learning_rate": 6.6080729166666665e-06,
+      "loss": 1.144,
+      "mean_token_accuracy": 0.7481625184416771,
+      "num_tokens": 3449834.0,
+      "step": 570
+    },
+    {
+      "entropy": 1.1694963037967683,
+      "epoch": 0.7323232323232324,
+      "grad_norm": 0.5903164744377136,
+      "learning_rate": 6.54296875e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7476441130042076,
+      "num_tokens": 3510701.0,
+      "step": 580
+    },
+    {
+      "entropy": 1.1777653217315673,
+      "epoch": 0.7449494949494949,
+      "grad_norm": 0.6284182071685791,
+      "learning_rate": 6.477864583333334e-06,
+      "loss": 1.1422,
+      "mean_token_accuracy": 0.7490358456969262,
+      "num_tokens": 3570992.0,
+      "step": 590
+    },
+    {
+      "entropy": 1.1588268011808396,
+      "epoch": 0.7575757575757576,
+      "grad_norm": 0.6250146627426147,
+      "learning_rate": 6.412760416666667e-06,
+      "loss": 1.1336,
+      "mean_token_accuracy": 0.7504108369350433,
+      "num_tokens": 3631087.0,
+      "step": 600
+    },
+    {
+      "entropy": 1.1890273630619048,
+      "epoch": 0.7702020202020202,
+      "grad_norm": 0.6420578956604004,
+      "learning_rate": 6.3476562500000006e-06,
+      "loss": 1.1534,
+      "mean_token_accuracy": 0.7442372158169747,
+      "num_tokens": 3692871.0,
+      "step": 610
+    },
+    {
+      "entropy": 1.1817798465490341,
+      "epoch": 0.7828282828282829,
+      "grad_norm": 0.6156490445137024,
+      "learning_rate": 6.282552083333334e-06,
+      "loss": 1.1477,
+      "mean_token_accuracy": 0.7468263059854507,
+      "num_tokens": 3753671.0,
+      "step": 620
+    },
+    {
+      "entropy": 1.1686612635850906,
+      "epoch": 0.7954545454545454,
+      "grad_norm": 0.6248748898506165,
+      "learning_rate": 6.217447916666667e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7478924334049225,
+      "num_tokens": 3813110.0,
+      "step": 630
+    },
+    {
+      "entropy": 1.1387953266501427,
+      "epoch": 0.8080808080808081,
+      "grad_norm": 0.6052266359329224,
+      "learning_rate": 6.152343750000001e-06,
+      "loss": 1.1118,
+      "mean_token_accuracy": 0.7538487210869789,
+      "num_tokens": 3873477.0,
+      "step": 640
+    },
+    {
+      "entropy": 1.1343814879655838,
+      "epoch": 0.8207070707070707,
+      "grad_norm": 0.6769536137580872,
+      "learning_rate": 6.087239583333335e-06,
+      "loss": 1.1108,
+      "mean_token_accuracy": 0.7548464313149452,
+      "num_tokens": 3932873.0,
+      "step": 650
+    },
+    {
+      "entropy": 1.1686519652605056,
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.6545736789703369,
+      "learning_rate": 6.022135416666667e-06,
+      "loss": 1.134,
+      "mean_token_accuracy": 0.7495195478200912,
+      "num_tokens": 3992754.0,
+      "step": 660
+    },
+    {
+      "entropy": 1.1596283346414566,
+      "epoch": 0.8459595959595959,
+      "grad_norm": 0.6192017793655396,
+      "learning_rate": 5.95703125e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7510297149419785,
+      "num_tokens": 4053540.0,
+      "step": 670
+    },
+    {
+      "entropy": 1.1484344542026519,
+      "epoch": 0.8585858585858586,
+      "grad_norm": 0.6520631909370422,
+      "learning_rate": 5.891927083333334e-06,
+      "loss": 1.1181,
+      "mean_token_accuracy": 0.751795919239521,
+      "num_tokens": 4113582.0,
+      "step": 680
+    },
+    {
+      "entropy": 1.1519750133156776,
+      "epoch": 0.8712121212121212,
+      "grad_norm": 0.6247655153274536,
+      "learning_rate": 5.826822916666667e-06,
+      "loss": 1.1204,
+      "mean_token_accuracy": 0.7506368085741997,
+      "num_tokens": 4174983.0,
+      "step": 690
+    },
+    {
+      "entropy": 1.1486561581492425,
+      "epoch": 0.8838383838383839,
+      "grad_norm": 0.620272159576416,
+      "learning_rate": 5.761718750000001e-06,
+      "loss": 1.1191,
+      "mean_token_accuracy": 0.7536391675472259,
+      "num_tokens": 4234465.0,
+      "step": 700
+    },
+    {
+      "entropy": 1.1504923462867738,
+      "epoch": 0.8964646464646465,
+      "grad_norm": 0.6308649182319641,
+      "learning_rate": 5.6966145833333344e-06,
+      "loss": 1.1224,
+      "mean_token_accuracy": 0.750646598637104,
+      "num_tokens": 4295955.0,
+      "step": 710
+    },
+    {
+      "entropy": 1.1561898440122604,
+      "epoch": 0.9090909090909091,
+      "grad_norm": 0.6629899740219116,
+      "learning_rate": 5.6315104166666665e-06,
+      "loss": 1.1238,
+      "mean_token_accuracy": 0.7507557719945908,
+      "num_tokens": 4357171.0,
+      "step": 720
+    },
+    {
+      "entropy": 1.1413449853658677,
+      "epoch": 0.9217171717171717,
+      "grad_norm": 0.5972346067428589,
+      "learning_rate": 5.56640625e-06,
+      "loss": 1.1038,
+      "mean_token_accuracy": 0.7554104939103127,
+      "num_tokens": 4417636.0,
+      "step": 730
+    },
+    {
+      "entropy": 1.1356119453907012,
+      "epoch": 0.9343434343434344,
+      "grad_norm": 0.6356479525566101,
+      "learning_rate": 5.501302083333334e-06,
+      "loss": 1.1005,
+      "mean_token_accuracy": 0.7565629109740257,
+      "num_tokens": 4477294.0,
+      "step": 740
+    },
+    {
+      "entropy": 1.1633600294589996,
+      "epoch": 0.946969696969697,
+      "grad_norm": 0.6416464447975159,
+      "learning_rate": 5.436197916666667e-06,
+      "loss": 1.1225,
+      "mean_token_accuracy": 0.7515855401754379,
+      "num_tokens": 4537503.0,
+      "step": 750
+    },
+    {
+      "entropy": 1.1527763932943345,
+      "epoch": 0.9595959595959596,
+      "grad_norm": 0.6126084327697754,
+      "learning_rate": 5.3710937500000005e-06,
+      "loss": 1.1184,
+      "mean_token_accuracy": 0.7526160582900048,
+      "num_tokens": 4598778.0,
+      "step": 760
+    },
+    {
+      "entropy": 1.1397768080234527,
+      "epoch": 0.9722222222222222,
+      "grad_norm": 0.6359922289848328,
+      "learning_rate": 5.305989583333334e-06,
+      "loss": 1.1144,
+      "mean_token_accuracy": 0.7548302739858628,
+      "num_tokens": 4658978.0,
+      "step": 770
+    },
+    {
+      "entropy": 1.1569962561130525,
+      "epoch": 0.9848484848484849,
+      "grad_norm": 0.6260409951210022,
+      "learning_rate": 5.240885416666667e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7512885302305221,
+      "num_tokens": 4720500.0,
+      "step": 780
+    },
+    {
+      "entropy": 1.1509152203798294,
+      "epoch": 0.9974747474747475,
+      "grad_norm": 0.6293452978134155,
+      "learning_rate": 5.17578125e-06,
+      "loss": 1.1227,
+      "mean_token_accuracy": 0.7519564241170883,
+      "num_tokens": 4781612.0,
+      "step": 790
+    },
+    {
+      "entropy": 1.1399024561047555,
+      "epoch": 1.0101010101010102,
+      "grad_norm": 0.664761483669281,
+      "learning_rate": 5.110677083333334e-06,
+      "loss": 1.1034,
+      "mean_token_accuracy": 0.7526706486940384,
+      "num_tokens": 4841359.0,
+      "step": 800
+    },
+    {
+      "entropy": 1.120214229822159,
+      "epoch": 1.0227272727272727,
+      "grad_norm": 0.5934865474700928,
+      "learning_rate": 5.045572916666667e-06,
+      "loss": 1.0857,
+      "mean_token_accuracy": 0.7593594208359719,
+      "num_tokens": 4901016.0,
+      "step": 810
+    },
+    {
+      "entropy": 1.1430341199040412,
+      "epoch": 1.0353535353535352,
+      "grad_norm": 0.6040735840797424,
+      "learning_rate": 4.98046875e-06,
+      "loss": 1.1165,
+      "mean_token_accuracy": 0.7525988414883613,
+      "num_tokens": 4961646.0,
+      "step": 820
+    },
+    {
+      "entropy": 1.1245349109172822,
+      "epoch": 1.047979797979798,
+      "grad_norm": 0.6277610063552856,
+      "learning_rate": 4.915364583333333e-06,
+      "loss": 1.0851,
+      "mean_token_accuracy": 0.7577921718358993,
+      "num_tokens": 5022365.0,
+      "step": 830
+    },
+    {
+      "entropy": 1.1204675793647767,
+      "epoch": 1.0606060606060606,
+      "grad_norm": 0.6260582804679871,
+      "learning_rate": 4.850260416666667e-06,
+      "loss": 1.0813,
+      "mean_token_accuracy": 0.7580071151256561,
+      "num_tokens": 5081972.0,
+      "step": 840
+    },
+    {
+      "entropy": 1.1222782507538795,
+      "epoch": 1.0732323232323233,
+      "grad_norm": 0.6023226976394653,
+      "learning_rate": 4.785156250000001e-06,
+      "loss": 1.0922,
+      "mean_token_accuracy": 0.7566258609294891,
+      "num_tokens": 5142184.0,
+      "step": 850
+    },
+    {
+      "entropy": 1.1227335944771766,
+      "epoch": 1.0858585858585859,
+      "grad_norm": 0.6206791996955872,
+      "learning_rate": 4.7200520833333336e-06,
+      "loss": 1.0994,
+      "mean_token_accuracy": 0.7540625646710396,
+      "num_tokens": 5203020.0,
+      "step": 860
+    },
+    {
+      "entropy": 1.1352888554334641,
+      "epoch": 1.0984848484848484,
+      "grad_norm": 0.6301055550575256,
+      "learning_rate": 4.654947916666667e-06,
+      "loss": 1.0958,
+      "mean_token_accuracy": 0.7562039017677307,
+      "num_tokens": 5263291.0,
+      "step": 870
+    },
+    {
+      "entropy": 1.1120088309049607,
+      "epoch": 1.1111111111111112,
+      "grad_norm": 0.6210020780563354,
+      "learning_rate": 4.58984375e-06,
+      "loss": 1.0793,
+      "mean_token_accuracy": 0.7590557768940925,
+      "num_tokens": 5323972.0,
+      "step": 880
+    },
+    {
+      "entropy": 1.0996666207909584,
+      "epoch": 1.1237373737373737,
+      "grad_norm": 0.6332690715789795,
+      "learning_rate": 4.524739583333334e-06,
+      "loss": 1.0717,
+      "mean_token_accuracy": 0.7615471586585045,
+      "num_tokens": 5383911.0,
+      "step": 890
+    },
+    {
+      "entropy": 1.127107810974121,
+      "epoch": 1.1363636363636362,
+      "grad_norm": 0.6505516767501831,
+      "learning_rate": 4.459635416666668e-06,
+      "loss": 1.1027,
+      "mean_token_accuracy": 0.7562421515583992,
+      "num_tokens": 5445417.0,
+      "step": 900
+    },
+    {
+      "entropy": 1.129740473628044,
+      "epoch": 1.148989898989899,
+      "grad_norm": 0.6406158804893494,
+      "learning_rate": 4.3945312500000005e-06,
+      "loss": 1.0879,
+      "mean_token_accuracy": 0.7587148532271385,
+      "num_tokens": 5505455.0,
+      "step": 910
+    },
+    {
+      "entropy": 1.1167259424924851,
+      "epoch": 1.1616161616161615,
+      "grad_norm": 0.6297397613525391,
+      "learning_rate": 4.329427083333333e-06,
+      "loss": 1.0752,
+      "mean_token_accuracy": 0.7604142814874649,
+      "num_tokens": 5565311.0,
+      "step": 920
+    },
+    {
+      "entropy": 1.1037891641259194,
+      "epoch": 1.1742424242424243,
+      "grad_norm": 0.6490073204040527,
+      "learning_rate": 4.264322916666667e-06,
+      "loss": 1.0686,
+      "mean_token_accuracy": 0.7610052570700645,
+      "num_tokens": 5625358.0,
+      "step": 930
+    },
+    {
+      "entropy": 1.1061881184577942,
+      "epoch": 1.1868686868686869,
+      "grad_norm": 0.6366387009620667,
+      "learning_rate": 4.19921875e-06,
+      "loss": 1.0868,
+      "mean_token_accuracy": 0.7574937298893929,
+      "num_tokens": 5686421.0,
+      "step": 940
+    },
+    {
+      "entropy": 1.1124324068427085,
+      "epoch": 1.1994949494949494,
+      "grad_norm": 0.6556055545806885,
+      "learning_rate": 4.134114583333334e-06,
+      "loss": 1.0694,
+      "mean_token_accuracy": 0.7602224007248879,
+      "num_tokens": 5745891.0,
+      "step": 950
+    },
+    {
+      "entropy": 1.1175200879573821,
+      "epoch": 1.2121212121212122,
+      "grad_norm": 0.6404849886894226,
+      "learning_rate": 4.0690104166666675e-06,
+      "loss": 1.081,
+      "mean_token_accuracy": 0.7568994402885437,
+      "num_tokens": 5806078.0,
+      "step": 960
+    },
+    {
+      "entropy": 1.1186978340148925,
+      "epoch": 1.2247474747474747,
+      "grad_norm": 0.6227584481239319,
+      "learning_rate": 4.00390625e-06,
+      "loss": 1.0791,
+      "mean_token_accuracy": 0.759756401181221,
+      "num_tokens": 5866369.0,
+      "step": 970
+    },
+    {
+      "entropy": 1.122128139436245,
+      "epoch": 1.2373737373737375,
+      "grad_norm": 0.6616361141204834,
+      "learning_rate": 3.938802083333333e-06,
+      "loss": 1.0937,
+      "mean_token_accuracy": 0.7582718566060066,
+      "num_tokens": 5926217.0,
+      "step": 980
+    },
+    {
+      "entropy": 1.122861033678055,
+      "epoch": 1.25,
+      "grad_norm": 0.6384168267250061,
+      "learning_rate": 3.873697916666667e-06,
+      "loss": 1.0978,
+      "mean_token_accuracy": 0.7549964562058449,
+      "num_tokens": 5987666.0,
+      "step": 990
+    },
+    {
+      "entropy": 1.1277505576610565,
+      "epoch": 1.2626262626262625,
+      "grad_norm": 0.6038117408752441,
+      "learning_rate": 3.8085937500000002e-06,
+      "loss": 1.0952,
+      "mean_token_accuracy": 0.755272176861763,
+      "num_tokens": 6048708.0,
+      "step": 1000
+    },
+    {
+      "entropy": 1.1120157346129418,
+      "epoch": 1.2752525252525253,
+      "grad_norm": 0.6418159604072571,
+      "learning_rate": 3.7434895833333336e-06,
+      "loss": 1.078,
+      "mean_token_accuracy": 0.7594122514128685,
+      "num_tokens": 6109652.0,
+      "step": 1010
+    },
+    {
+      "entropy": 1.101425115764141,
+      "epoch": 1.2878787878787878,
+      "grad_norm": 0.6218425035476685,
+      "learning_rate": 3.6783854166666673e-06,
+      "loss": 1.0688,
+      "mean_token_accuracy": 0.7604865297675133,
+      "num_tokens": 6169125.0,
+      "step": 1020
+    },
+    {
+      "entropy": 1.1007713869214057,
+      "epoch": 1.3005050505050506,
+      "grad_norm": 0.6429149508476257,
+      "learning_rate": 3.61328125e-06,
+      "loss": 1.0581,
+      "mean_token_accuracy": 0.7621071562170982,
+      "num_tokens": 6230303.0,
+      "step": 1030
+    },
+    {
+      "entropy": 1.1094096556305886,
+      "epoch": 1.3131313131313131,
+      "grad_norm": 0.6489748358726501,
+      "learning_rate": 3.5481770833333335e-06,
+      "loss": 1.0715,
+      "mean_token_accuracy": 0.7599423810839653,
+      "num_tokens": 6291396.0,
+      "step": 1040
+    },
+    {
+      "entropy": 1.0827289715409278,
+      "epoch": 1.3257575757575757,
+      "grad_norm": 0.6485461592674255,
+      "learning_rate": 3.483072916666667e-06,
+      "loss": 1.0584,
+      "mean_token_accuracy": 0.7630694910883904,
+      "num_tokens": 6351579.0,
+      "step": 1050
+    },
+    {
+      "entropy": 1.114325873553753,
+      "epoch": 1.3383838383838385,
+      "grad_norm": 0.6261104941368103,
+      "learning_rate": 3.41796875e-06,
+      "loss": 1.0764,
+      "mean_token_accuracy": 0.7585488513112069,
+      "num_tokens": 6411662.0,
+      "step": 1060
+    },
+    {
+      "entropy": 1.1271554425358772,
+      "epoch": 1.351010101010101,
+      "grad_norm": 0.6522034406661987,
+      "learning_rate": 3.3528645833333334e-06,
+      "loss": 1.0902,
+      "mean_token_accuracy": 0.7562535598874092,
+      "num_tokens": 6473505.0,
+      "step": 1070
+    },
+    {
+      "entropy": 1.1013643085956573,
+      "epoch": 1.3636363636363638,
+      "grad_norm": 0.6176674962043762,
+      "learning_rate": 3.287760416666667e-06,
+      "loss": 1.065,
+      "mean_token_accuracy": 0.763075165450573,
+      "num_tokens": 6533580.0,
+      "step": 1080
+    },
+    {
+      "entropy": 1.098090337216854,
+      "epoch": 1.3762626262626263,
+      "grad_norm": 0.6540253758430481,
+      "learning_rate": 3.2226562500000004e-06,
+      "loss": 1.0596,
+      "mean_token_accuracy": 0.7616770043969154,
+      "num_tokens": 6593481.0,
+      "step": 1090
+    },
+    {
+      "entropy": 1.1176372200250626,
+      "epoch": 1.3888888888888888,
+      "grad_norm": 0.6754550933837891,
+      "learning_rate": 3.1575520833333333e-06,
+      "loss": 1.0861,
+      "mean_token_accuracy": 0.7573029339313507,
+      "num_tokens": 6653967.0,
+      "step": 1100
+    },
+    {
+      "entropy": 1.1040414482355119,
+      "epoch": 1.4015151515151514,
+      "grad_norm": 0.6022531986236572,
+      "learning_rate": 3.092447916666667e-06,
+      "loss": 1.0573,
+      "mean_token_accuracy": 0.7612267225980759,
+      "num_tokens": 6714685.0,
+      "step": 1110
+    },
+    {
+      "entropy": 1.0926991075277328,
+      "epoch": 1.4141414141414141,
+      "grad_norm": 0.6621010303497314,
+      "learning_rate": 3.0273437500000003e-06,
+      "loss": 1.0637,
+      "mean_token_accuracy": 0.7612977519631385,
+      "num_tokens": 6774659.0,
+      "step": 1120
+    },
+    {
+      "entropy": 1.1095534324645997,
+      "epoch": 1.4267676767676767,
+      "grad_norm": 0.62503981590271,
+      "learning_rate": 2.962239583333333e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7618053883314133,
+      "num_tokens": 6834579.0,
+      "step": 1130
+    },
+    {
+      "entropy": 1.117809349298477,
+      "epoch": 1.4393939393939394,
+      "grad_norm": 0.6527109742164612,
+      "learning_rate": 2.897135416666667e-06,
+      "loss": 1.0747,
+      "mean_token_accuracy": 0.759482853114605,
+      "num_tokens": 6894074.0,
+      "step": 1140
+    },
+    {
+      "entropy": 1.1005077749490737,
+      "epoch": 1.452020202020202,
+      "grad_norm": 0.6720954775810242,
+      "learning_rate": 2.8320312500000002e-06,
+      "loss": 1.0607,
+      "mean_token_accuracy": 0.7621246844530105,
+      "num_tokens": 6953870.0,
+      "step": 1150
+    },
+    {
+      "entropy": 1.1236482918262483,
+      "epoch": 1.4646464646464645,
+      "grad_norm": 0.658524215221405,
+      "learning_rate": 2.7669270833333335e-06,
+      "loss": 1.0884,
+      "mean_token_accuracy": 0.7560836613178253,
+      "num_tokens": 7014553.0,
+      "step": 1160
+    },
+    {
+      "entropy": 1.1116504594683647,
+      "epoch": 1.4772727272727273,
+      "grad_norm": 0.6261802911758423,
+      "learning_rate": 2.7018229166666673e-06,
+      "loss": 1.0659,
+      "mean_token_accuracy": 0.7597616642713547,
+      "num_tokens": 7076291.0,
+      "step": 1170
+    },
+    {
+      "entropy": 1.073892480134964,
+      "epoch": 1.4898989898989898,
+      "grad_norm": 0.6310375332832336,
+      "learning_rate": 2.63671875e-06,
+      "loss": 1.0524,
+      "mean_token_accuracy": 0.7628733053803444,
+      "num_tokens": 7137305.0,
+      "step": 1180
+    },
+    {
+      "entropy": 1.0975843235850333,
+      "epoch": 1.5025252525252526,
+      "grad_norm": 0.638482391834259,
+      "learning_rate": 2.5716145833333334e-06,
+      "loss": 1.0679,
+      "mean_token_accuracy": 0.7603248566389084,
+      "num_tokens": 7198239.0,
+      "step": 1190
+    },
+    {
+      "entropy": 1.0986508697271347,
+      "epoch": 1.5151515151515151,
+      "grad_norm": 0.640065610408783,
+      "learning_rate": 2.506510416666667e-06,
+      "loss": 1.0666,
+      "mean_token_accuracy": 0.7622530281543731,
+      "num_tokens": 7257847.0,
+      "step": 1200
+    },
+    {
+      "entropy": 1.0971406906843186,
+      "epoch": 1.5277777777777777,
+      "grad_norm": 0.6437165141105652,
+      "learning_rate": 2.44140625e-06,
+      "loss": 1.0587,
+      "mean_token_accuracy": 0.7623877748847008,
+      "num_tokens": 7317615.0,
+      "step": 1210
+    },
+    {
+      "entropy": 1.1032136514782906,
+      "epoch": 1.5404040404040404,
+      "grad_norm": 0.6590547561645508,
+      "learning_rate": 2.3763020833333338e-06,
+      "loss": 1.0569,
+      "mean_token_accuracy": 0.7616146191954613,
+      "num_tokens": 7377946.0,
+      "step": 1220
+    },
+    {
+      "entropy": 1.0922824308276176,
+      "epoch": 1.553030303030303,
+      "grad_norm": 0.6317723989486694,
+      "learning_rate": 2.3111979166666667e-06,
+      "loss": 1.0616,
+      "mean_token_accuracy": 0.7611089378595353,
+      "num_tokens": 7438666.0,
+      "step": 1230
+    },
+    {
+      "entropy": 1.1092566132545472,
+      "epoch": 1.5656565656565657,
+      "grad_norm": 0.66637122631073,
+      "learning_rate": 2.2460937500000004e-06,
+      "loss": 1.0725,
+      "mean_token_accuracy": 0.7594954133033752,
+      "num_tokens": 7499747.0,
+      "step": 1240
+    },
+    {
+      "entropy": 1.111878876388073,
+      "epoch": 1.5782828282828283,
+      "grad_norm": 0.6520881652832031,
+      "learning_rate": 2.1809895833333337e-06,
+      "loss": 1.074,
+      "mean_token_accuracy": 0.7558068126440048,
+      "num_tokens": 7561004.0,
+      "step": 1250
+    },
+    {
+      "entropy": 1.1105972841382026,
+      "epoch": 1.5909090909090908,
+      "grad_norm": 0.6495437622070312,
+      "learning_rate": 2.1158854166666666e-06,
+      "loss": 1.0844,
+      "mean_token_accuracy": 0.7563070356845856,
+      "num_tokens": 7622115.0,
+      "step": 1260
+    },
+    {
+      "entropy": 1.0932338371872903,
+      "epoch": 1.6035353535353534,
+      "grad_norm": 0.6420316696166992,
+      "learning_rate": 2.0507812500000003e-06,
+      "loss": 1.0649,
+      "mean_token_accuracy": 0.7622412323951722,
+      "num_tokens": 7682210.0,
+      "step": 1270
+    },
+    {
+      "entropy": 1.0919055327773095,
+      "epoch": 1.6161616161616161,
+      "grad_norm": 0.6192623972892761,
+      "learning_rate": 1.9856770833333336e-06,
+      "loss": 1.0427,
+      "mean_token_accuracy": 0.7657591253519058,
+      "num_tokens": 7742745.0,
+      "step": 1280
+    },
+    {
+      "entropy": 1.1026942864060403,
+      "epoch": 1.628787878787879,
+      "grad_norm": 0.6355161666870117,
+      "learning_rate": 1.920572916666667e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7609635755419731,
+      "num_tokens": 7802902.0,
+      "step": 1290
+    },
+    {
+      "entropy": 1.1140723824501038,
+      "epoch": 1.6414141414141414,
+      "grad_norm": 0.6254522800445557,
+      "learning_rate": 1.8554687500000002e-06,
+      "loss": 1.0789,
+      "mean_token_accuracy": 0.7592580512166023,
+      "num_tokens": 7863576.0,
+      "step": 1300
+    },
+    {
+      "entropy": 1.108394268155098,
+      "epoch": 1.654040404040404,
+      "grad_norm": 0.633172333240509,
+      "learning_rate": 1.7903645833333335e-06,
+      "loss": 1.0714,
+      "mean_token_accuracy": 0.7598690986633301,
+      "num_tokens": 7925518.0,
+      "step": 1310
+    },
+    {
+      "entropy": 1.1097407966852189,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 0.6279735565185547,
+      "learning_rate": 1.7252604166666668e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7611567705869675,
+      "num_tokens": 7987388.0,
+      "step": 1320
+    },
+    {
+      "entropy": 1.1071213275194167,
+      "epoch": 1.6792929292929293,
+      "grad_norm": 0.6425778269767761,
+      "learning_rate": 1.6601562500000001e-06,
+      "loss": 1.0786,
+      "mean_token_accuracy": 0.7576610520482063,
+      "num_tokens": 8048853.0,
+      "step": 1330
+    },
+    {
+      "entropy": 1.0931925728917122,
+      "epoch": 1.691919191919192,
+      "grad_norm": 0.666192889213562,
+      "learning_rate": 1.5950520833333336e-06,
+      "loss": 1.0604,
+      "mean_token_accuracy": 0.7616572439670563,
+      "num_tokens": 8108967.0,
+      "step": 1340
+    },
+    {
+      "entropy": 1.0973364993929864,
+      "epoch": 1.7045454545454546,
+      "grad_norm": 0.6348255276679993,
+      "learning_rate": 1.5299479166666667e-06,
+      "loss": 1.0769,
+      "mean_token_accuracy": 0.7596119627356529,
+      "num_tokens": 8169700.0,
+      "step": 1350
+    },
+    {
+      "entropy": 1.1137071400880814,
+      "epoch": 1.7171717171717171,
+      "grad_norm": 0.6510699391365051,
+      "learning_rate": 1.46484375e-06,
+      "loss": 1.0731,
+      "mean_token_accuracy": 0.7593250289559365,
+      "num_tokens": 8229676.0,
+      "step": 1360
+    },
+    {
+      "entropy": 1.1052428260445595,
+      "epoch": 1.7297979797979797,
+      "grad_norm": 0.6622318625450134,
+      "learning_rate": 1.3997395833333335e-06,
+      "loss": 1.069,
+      "mean_token_accuracy": 0.7627501472830772,
+      "num_tokens": 8289396.0,
+      "step": 1370
+    },
+    {
+      "entropy": 1.0989198789000512,
+      "epoch": 1.7424242424242424,
+      "grad_norm": 0.6430277824401855,
+      "learning_rate": 1.3346354166666666e-06,
+      "loss": 1.0506,
+      "mean_token_accuracy": 0.7634323209524154,
+      "num_tokens": 8351152.0,
+      "step": 1380
+    },
+    {
+      "entropy": 1.0913211867213248,
+      "epoch": 1.7550505050505052,
+      "grad_norm": 0.639707088470459,
+      "learning_rate": 1.2695312500000002e-06,
+      "loss": 1.0505,
+      "mean_token_accuracy": 0.763850274682045,
+      "num_tokens": 8411357.0,
+      "step": 1390
+    },
+    {
+      "entropy": 1.1044820442795753,
+      "epoch": 1.7676767676767677,
+      "grad_norm": 0.680479109287262,
+      "learning_rate": 1.2044270833333335e-06,
+      "loss": 1.0756,
+      "mean_token_accuracy": 0.7598197475075722,
+      "num_tokens": 8471975.0,
+      "step": 1400
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1584,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.796684058233733e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1400/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eeb71c14deb91ac5fd11522db45cb3275c9164415fcbefc9d00cac27a27f0a3
+size 6417

checkpoint-1584/README.md

@@ -0,0 +1,209 @@
+---
+base_model: CohereLabs/aya-expanse-8b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:CohereLabs/aya-expanse-8b
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-1584/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "CohereLabs/aya-expanse-8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "o_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1584/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4ccb6b6fb84ee6cc158f9a20474ec7ec29ec158992cc2108c5575301c294143
+size 167832240

checkpoint-1584/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true %}{% set loop_messages = messages %}{% set system_message = 'You are Aya, a brilliant, sophisticated, multilingual AI-assistant trained to assist human users by providing thorough responses. You are able to interact and respond to questions in 23 languages and you are powered by a multilingual model built by Cohere For AI.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% if system_message != false %}{{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% elif message['role'] == 'assistant' %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'  + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }}{% endif %}

checkpoint-1584/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad3c6bd1873cc7822b9cfcc9d20d53f10797d182583ad873b056e3fce9975cec
+size 335929123

checkpoint-1584/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb890208f872a9bc21144563ccd67590ecc434a4bdb0af2e1f0f0e89facf8648
+size 14645

checkpoint-1584/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad40633bbc30d3679cb64e4eb23a2afc463f00f1a22fd86e76f1a61cdf8ca9d6
+size 1465

checkpoint-1584/special_tokens_map.json

@@ -0,0 +1,17 @@
+{
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<PAD>"
+}

checkpoint-1584/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:345ccf04a5257f473e331715ecc69365c5ac8fc2490923fe7155560af809ec1a
+size 20124090

checkpoint-1584/tokenizer_config.json

@@ -0,0 +1,317 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<CLS>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<SEP>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<EOP_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255000": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255001": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255002": {
+      "content": "<|YES_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255003": {
+      "content": "<|NO_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255004": {
+      "content": "<|GOOD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255005": {
+      "content": "<|BAD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255006": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255007": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255008": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255009": {
+      "content": "<|USER_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255010": {
+      "content": "<|USER_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255011": {
+      "content": "<|USER_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255012": {
+      "content": "<|USER_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255013": {
+      "content": "<|USER_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255014": {
+      "content": "<|USER_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255015": {
+      "content": "<|USER_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255016": {
+      "content": "<|USER_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255017": {
+      "content": "<|USER_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255018": {
+      "content": "<|USER_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255019": {
+      "content": "<|EXTRA_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255020": {
+      "content": "<|EXTRA_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255021": {
+      "content": "<|EXTRA_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255022": {
+      "content": "<|EXTRA_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255023": {
+      "content": "<|EXTRA_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255024": {
+      "content": "<|EXTRA_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255025": {
+      "content": "<|EXTRA_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255026": {
+      "content": "<|EXTRA_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255027": {
+      "content": "<|EXTRA_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255028": {
+      "content": "<|EXTRA_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<BOS_TOKEN>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|END_OF_TURN_TOKEN|>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "merges_file": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "CohereTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false,
+  "vocab_file": null
+}

checkpoint-1584/trainer_state.json

@@ -0,0 +1,1614 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.0,
+  "eval_steps": 200,
+  "global_step": 1584,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.4040649354457857,
+      "epoch": 0.012626262626262626,
+      "grad_norm": 4.66565465927124,
+      "learning_rate": 1.8750000000000003e-06,
+      "loss": 3.6021,
+      "mean_token_accuracy": 0.4221017129719257,
+      "num_tokens": 61199.0,
+      "step": 10
+    },
+    {
+      "entropy": 2.3836746215820312,
+      "epoch": 0.025252525252525252,
+      "grad_norm": 3.8161869049072266,
+      "learning_rate": 3.958333333333333e-06,
+      "loss": 3.3432,
+      "mean_token_accuracy": 0.44042530804872515,
+      "num_tokens": 122423.0,
+      "step": 20
+    },
+    {
+      "entropy": 2.355724626779556,
+      "epoch": 0.03787878787878788,
+      "grad_norm": 3.8800699710845947,
+      "learning_rate": 6.041666666666667e-06,
+      "loss": 2.9033,
+      "mean_token_accuracy": 0.48426677361130716,
+      "num_tokens": 182649.0,
+      "step": 30
+    },
+    {
+      "entropy": 2.092331054806709,
+      "epoch": 0.050505050505050504,
+      "grad_norm": 2.8217720985412598,
+      "learning_rate": 8.125000000000001e-06,
+      "loss": 2.356,
+      "mean_token_accuracy": 0.5772452697157859,
+      "num_tokens": 243049.0,
+      "step": 40
+    },
+    {
+      "entropy": 1.6766322344541549,
+      "epoch": 0.06313131313131314,
+      "grad_norm": 1.4623568058013916,
+      "learning_rate": 9.993489583333334e-06,
+      "loss": 1.8899,
+      "mean_token_accuracy": 0.6480962842702865,
+      "num_tokens": 304326.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.5568815559148788,
+      "epoch": 0.07575757575757576,
+      "grad_norm": 1.171562671661377,
+      "learning_rate": 9.928385416666668e-06,
+      "loss": 1.677,
+      "mean_token_accuracy": 0.6776855796575546,
+      "num_tokens": 364866.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.48199902176857,
+      "epoch": 0.08838383838383838,
+      "grad_norm": 0.9904961585998535,
+      "learning_rate": 9.863281250000001e-06,
+      "loss": 1.5337,
+      "mean_token_accuracy": 0.697019773721695,
+      "num_tokens": 423478.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.497376424074173,
+      "epoch": 0.10101010101010101,
+      "grad_norm": 0.9454260468482971,
+      "learning_rate": 9.798177083333335e-06,
+      "loss": 1.4953,
+      "mean_token_accuracy": 0.6976823702454567,
+      "num_tokens": 483659.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.4664768785238267,
+      "epoch": 0.11363636363636363,
+      "grad_norm": 0.8955270648002625,
+      "learning_rate": 9.733072916666667e-06,
+      "loss": 1.4356,
+      "mean_token_accuracy": 0.7069446608424187,
+      "num_tokens": 544453.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.4269085675477982,
+      "epoch": 0.12626262626262627,
+      "grad_norm": 0.9242203235626221,
+      "learning_rate": 9.66796875e-06,
+      "loss": 1.4106,
+      "mean_token_accuracy": 0.7122392952442169,
+      "num_tokens": 604546.0,
+      "step": 100
+    },
+    {
+      "entropy": 1.4060751020908355,
+      "epoch": 0.1388888888888889,
+      "grad_norm": 0.8968560695648193,
+      "learning_rate": 9.602864583333335e-06,
+      "loss": 1.3487,
+      "mean_token_accuracy": 0.7178552970290184,
+      "num_tokens": 664860.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.4018951296806335,
+      "epoch": 0.15151515151515152,
+      "grad_norm": 0.9047113656997681,
+      "learning_rate": 9.537760416666667e-06,
+      "loss": 1.3347,
+      "mean_token_accuracy": 0.7208079636096955,
+      "num_tokens": 725022.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.3809731483459473,
+      "epoch": 0.16414141414141414,
+      "grad_norm": 0.8915444016456604,
+      "learning_rate": 9.47265625e-06,
+      "loss": 1.3155,
+      "mean_token_accuracy": 0.7267520889639855,
+      "num_tokens": 785586.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.3699676394462585,
+      "epoch": 0.17676767676767677,
+      "grad_norm": 0.8574295043945312,
+      "learning_rate": 9.407552083333334e-06,
+      "loss": 1.3016,
+      "mean_token_accuracy": 0.7266673430800438,
+      "num_tokens": 845790.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.3425012439489366,
+      "epoch": 0.1893939393939394,
+      "grad_norm": 0.8231800198554993,
+      "learning_rate": 9.342447916666668e-06,
+      "loss": 1.2823,
+      "mean_token_accuracy": 0.7277117937803268,
+      "num_tokens": 905842.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.3314216613769532,
+      "epoch": 0.20202020202020202,
+      "grad_norm": 0.8166369795799255,
+      "learning_rate": 9.277343750000001e-06,
+      "loss": 1.2917,
+      "mean_token_accuracy": 0.7278609350323677,
+      "num_tokens": 966487.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.30389544069767,
+      "epoch": 0.21464646464646464,
+      "grad_norm": 0.7738587260246277,
+      "learning_rate": 9.212239583333335e-06,
+      "loss": 1.2481,
+      "mean_token_accuracy": 0.7335339426994324,
+      "num_tokens": 1025558.0,
+      "step": 170
+    },
+    {
+      "entropy": 1.311449444293976,
+      "epoch": 0.22727272727272727,
+      "grad_norm": 0.7718328833580017,
+      "learning_rate": 9.147135416666667e-06,
+      "loss": 1.2643,
+      "mean_token_accuracy": 0.7285080313682556,
+      "num_tokens": 1086677.0,
+      "step": 180
+    },
+    {
+      "entropy": 1.3079361200332642,
+      "epoch": 0.2398989898989899,
+      "grad_norm": 0.7341915369033813,
+      "learning_rate": 9.082031250000001e-06,
+      "loss": 1.2641,
+      "mean_token_accuracy": 0.729550538957119,
+      "num_tokens": 1147885.0,
+      "step": 190
+    },
+    {
+      "entropy": 1.2794219702482224,
+      "epoch": 0.25252525252525254,
+      "grad_norm": 0.748540997505188,
+      "learning_rate": 9.016927083333335e-06,
+      "loss": 1.2397,
+      "mean_token_accuracy": 0.7351120054721832,
+      "num_tokens": 1207321.0,
+      "step": 200
+    },
+    {
+      "entropy": 1.294454461336136,
+      "epoch": 0.26515151515151514,
+      "grad_norm": 0.7585553526878357,
+      "learning_rate": 8.951822916666667e-06,
+      "loss": 1.2489,
+      "mean_token_accuracy": 0.7322148531675339,
+      "num_tokens": 1267837.0,
+      "step": 210
+    },
+    {
+      "entropy": 1.2862246632575989,
+      "epoch": 0.2777777777777778,
+      "grad_norm": 0.6937864422798157,
+      "learning_rate": 8.88671875e-06,
+      "loss": 1.2383,
+      "mean_token_accuracy": 0.7358245223760604,
+      "num_tokens": 1328436.0,
+      "step": 220
+    },
+    {
+      "entropy": 1.2531811505556107,
+      "epoch": 0.2904040404040404,
+      "grad_norm": 0.6792387366294861,
+      "learning_rate": 8.821614583333334e-06,
+      "loss": 1.2007,
+      "mean_token_accuracy": 0.7360369265079498,
+      "num_tokens": 1389877.0,
+      "step": 230
+    },
+    {
+      "entropy": 1.2815489560365676,
+      "epoch": 0.30303030303030304,
+      "grad_norm": 0.6865427494049072,
+      "learning_rate": 8.756510416666666e-06,
+      "loss": 1.2474,
+      "mean_token_accuracy": 0.7304804190993309,
+      "num_tokens": 1450927.0,
+      "step": 240
+    },
+    {
+      "entropy": 1.2568059146404267,
+      "epoch": 0.31565656565656564,
+      "grad_norm": 0.669840395450592,
+      "learning_rate": 8.69140625e-06,
+      "loss": 1.2172,
+      "mean_token_accuracy": 0.7385278165340423,
+      "num_tokens": 1511174.0,
+      "step": 250
+    },
+    {
+      "entropy": 1.254868358373642,
+      "epoch": 0.3282828282828283,
+      "grad_norm": 0.6434893012046814,
+      "learning_rate": 8.626302083333334e-06,
+      "loss": 1.213,
+      "mean_token_accuracy": 0.7380544006824493,
+      "num_tokens": 1570570.0,
+      "step": 260
+    },
+    {
+      "entropy": 1.2532441645860672,
+      "epoch": 0.3409090909090909,
+      "grad_norm": 0.6034978032112122,
+      "learning_rate": 8.561197916666667e-06,
+      "loss": 1.2116,
+      "mean_token_accuracy": 0.7382062628865242,
+      "num_tokens": 1630930.0,
+      "step": 270
+    },
+    {
+      "entropy": 1.2605280816555022,
+      "epoch": 0.35353535353535354,
+      "grad_norm": 0.6371450424194336,
+      "learning_rate": 8.496093750000001e-06,
+      "loss": 1.2313,
+      "mean_token_accuracy": 0.7328852489590645,
+      "num_tokens": 1692624.0,
+      "step": 280
+    },
+    {
+      "entropy": 1.248606452345848,
+      "epoch": 0.3661616161616162,
+      "grad_norm": 0.6300278306007385,
+      "learning_rate": 8.430989583333335e-06,
+      "loss": 1.2195,
+      "mean_token_accuracy": 0.7370449885725975,
+      "num_tokens": 1754213.0,
+      "step": 290
+    },
+    {
+      "entropy": 1.2246102809906005,
+      "epoch": 0.3787878787878788,
+      "grad_norm": 0.6430155634880066,
+      "learning_rate": 8.365885416666667e-06,
+      "loss": 1.1845,
+      "mean_token_accuracy": 0.7439196646213532,
+      "num_tokens": 1813841.0,
+      "step": 300
+    },
+    {
+      "entropy": 1.2188815206289292,
+      "epoch": 0.39141414141414144,
+      "grad_norm": 0.6395701766014099,
+      "learning_rate": 8.30078125e-06,
+      "loss": 1.1856,
+      "mean_token_accuracy": 0.7422183871269226,
+      "num_tokens": 1873568.0,
+      "step": 310
+    },
+    {
+      "entropy": 1.234528934955597,
+      "epoch": 0.40404040404040403,
+      "grad_norm": 0.6168740391731262,
+      "learning_rate": 8.235677083333334e-06,
+      "loss": 1.1951,
+      "mean_token_accuracy": 0.7380509555339814,
+      "num_tokens": 1935518.0,
+      "step": 320
+    },
+    {
+      "entropy": 1.226365676522255,
+      "epoch": 0.4166666666666667,
+      "grad_norm": 0.611132800579071,
+      "learning_rate": 8.170572916666666e-06,
+      "loss": 1.2078,
+      "mean_token_accuracy": 0.739974245429039,
+      "num_tokens": 1995604.0,
+      "step": 330
+    },
+    {
+      "entropy": 1.2132183194160462,
+      "epoch": 0.4292929292929293,
+      "grad_norm": 0.6103131771087646,
+      "learning_rate": 8.10546875e-06,
+      "loss": 1.1642,
+      "mean_token_accuracy": 0.7444137379527092,
+      "num_tokens": 2056484.0,
+      "step": 340
+    },
+    {
+      "entropy": 1.223158246278763,
+      "epoch": 0.44191919191919193,
+      "grad_norm": 0.6188805103302002,
+      "learning_rate": 8.040364583333334e-06,
+      "loss": 1.2001,
+      "mean_token_accuracy": 0.7385613292455673,
+      "num_tokens": 2118437.0,
+      "step": 350
+    },
+    {
+      "entropy": 1.2154075980186463,
+      "epoch": 0.45454545454545453,
+      "grad_norm": 0.6238694190979004,
+      "learning_rate": 7.975260416666668e-06,
+      "loss": 1.1848,
+      "mean_token_accuracy": 0.7404153689742088,
+      "num_tokens": 2179333.0,
+      "step": 360
+    },
+    {
+      "entropy": 1.197928261756897,
+      "epoch": 0.4671717171717172,
+      "grad_norm": 0.6028566956520081,
+      "learning_rate": 7.910156250000001e-06,
+      "loss": 1.1597,
+      "mean_token_accuracy": 0.7475010469555855,
+      "num_tokens": 2239604.0,
+      "step": 370
+    },
+    {
+      "entropy": 1.189855706691742,
+      "epoch": 0.4797979797979798,
+      "grad_norm": 0.6569434404373169,
+      "learning_rate": 7.845052083333335e-06,
+      "loss": 1.1805,
+      "mean_token_accuracy": 0.7427218139171601,
+      "num_tokens": 2300936.0,
+      "step": 380
+    },
+    {
+      "entropy": 1.2076493889093398,
+      "epoch": 0.49242424242424243,
+      "grad_norm": 0.6351733207702637,
+      "learning_rate": 7.779947916666667e-06,
+      "loss": 1.1759,
+      "mean_token_accuracy": 0.7408407002687454,
+      "num_tokens": 2361539.0,
+      "step": 390
+    },
+    {
+      "entropy": 1.2223081022500992,
+      "epoch": 0.5050505050505051,
+      "grad_norm": 0.6327986121177673,
+      "learning_rate": 7.71484375e-06,
+      "loss": 1.1874,
+      "mean_token_accuracy": 0.7407250568270684,
+      "num_tokens": 2422162.0,
+      "step": 400
+    },
+    {
+      "entropy": 1.2013412863016129,
+      "epoch": 0.5176767676767676,
+      "grad_norm": 0.622104823589325,
+      "learning_rate": 7.649739583333334e-06,
+      "loss": 1.1726,
+      "mean_token_accuracy": 0.744242025911808,
+      "num_tokens": 2483447.0,
+      "step": 410
+    },
+    {
+      "entropy": 1.2116613179445266,
+      "epoch": 0.5303030303030303,
+      "grad_norm": 0.637651264667511,
+      "learning_rate": 7.5846354166666665e-06,
+      "loss": 1.1838,
+      "mean_token_accuracy": 0.7405030101537704,
+      "num_tokens": 2544848.0,
+      "step": 420
+    },
+    {
+      "entropy": 1.2024697184562683,
+      "epoch": 0.5429292929292929,
+      "grad_norm": 0.6252374649047852,
+      "learning_rate": 7.51953125e-06,
+      "loss": 1.1681,
+      "mean_token_accuracy": 0.7458183988928795,
+      "num_tokens": 2605232.0,
+      "step": 430
+    },
+    {
+      "entropy": 1.1797083109617232,
+      "epoch": 0.5555555555555556,
+      "grad_norm": 0.6502755284309387,
+      "learning_rate": 7.454427083333334e-06,
+      "loss": 1.1452,
+      "mean_token_accuracy": 0.7477341219782829,
+      "num_tokens": 2664276.0,
+      "step": 440
+    },
+    {
+      "entropy": 1.1964200481772422,
+      "epoch": 0.5681818181818182,
+      "grad_norm": 0.639979362487793,
+      "learning_rate": 7.389322916666667e-06,
+      "loss": 1.1665,
+      "mean_token_accuracy": 0.7431837096810341,
+      "num_tokens": 2724073.0,
+      "step": 450
+    },
+    {
+      "entropy": 1.1795272737741471,
+      "epoch": 0.5808080808080808,
+      "grad_norm": 0.6212354302406311,
+      "learning_rate": 7.3242187500000006e-06,
+      "loss": 1.1529,
+      "mean_token_accuracy": 0.7479098170995713,
+      "num_tokens": 2784262.0,
+      "step": 460
+    },
+    {
+      "entropy": 1.1908618807792664,
+      "epoch": 0.5934343434343434,
+      "grad_norm": 0.6528693437576294,
+      "learning_rate": 7.259114583333334e-06,
+      "loss": 1.1678,
+      "mean_token_accuracy": 0.745174677670002,
+      "num_tokens": 2843804.0,
+      "step": 470
+    },
+    {
+      "entropy": 1.1862946093082427,
+      "epoch": 0.6060606060606061,
+      "grad_norm": 0.639481246471405,
+      "learning_rate": 7.194010416666667e-06,
+      "loss": 1.1565,
+      "mean_token_accuracy": 0.7461866185069084,
+      "num_tokens": 2903408.0,
+      "step": 480
+    },
+    {
+      "entropy": 1.151743534207344,
+      "epoch": 0.6186868686868687,
+      "grad_norm": 0.6332777142524719,
+      "learning_rate": 7.128906250000001e-06,
+      "loss": 1.1251,
+      "mean_token_accuracy": 0.7535199671983719,
+      "num_tokens": 2963401.0,
+      "step": 490
+    },
+    {
+      "entropy": 1.1778477430343628,
+      "epoch": 0.6313131313131313,
+      "grad_norm": 0.5991836190223694,
+      "learning_rate": 7.063802083333335e-06,
+      "loss": 1.1407,
+      "mean_token_accuracy": 0.7491110354661942,
+      "num_tokens": 3023530.0,
+      "step": 500
+    },
+    {
+      "entropy": 1.2023035794496537,
+      "epoch": 0.6439393939393939,
+      "grad_norm": 0.6293458938598633,
+      "learning_rate": 6.998697916666667e-06,
+      "loss": 1.1724,
+      "mean_token_accuracy": 0.7405256554484367,
+      "num_tokens": 3085225.0,
+      "step": 510
+    },
+    {
+      "entropy": 1.1997334092855454,
+      "epoch": 0.6565656565656566,
+      "grad_norm": 0.6213802695274353,
+      "learning_rate": 6.93359375e-06,
+      "loss": 1.1604,
+      "mean_token_accuracy": 0.7435309410095214,
+      "num_tokens": 3145749.0,
+      "step": 520
+    },
+    {
+      "entropy": 1.1643184214830398,
+      "epoch": 0.6691919191919192,
+      "grad_norm": 0.6495156288146973,
+      "learning_rate": 6.868489583333334e-06,
+      "loss": 1.1381,
+      "mean_token_accuracy": 0.7496751576662064,
+      "num_tokens": 3205761.0,
+      "step": 530
+    },
+    {
+      "entropy": 1.1837532848119736,
+      "epoch": 0.6818181818181818,
+      "grad_norm": 0.6004510521888733,
+      "learning_rate": 6.803385416666667e-06,
+      "loss": 1.1563,
+      "mean_token_accuracy": 0.7447851061820984,
+      "num_tokens": 3267024.0,
+      "step": 540
+    },
+    {
+      "entropy": 1.201677542924881,
+      "epoch": 0.6944444444444444,
+      "grad_norm": 0.607467532157898,
+      "learning_rate": 6.738281250000001e-06,
+      "loss": 1.1776,
+      "mean_token_accuracy": 0.7406648993492126,
+      "num_tokens": 3329070.0,
+      "step": 550
+    },
+    {
+      "entropy": 1.1659786373376846,
+      "epoch": 0.7070707070707071,
+      "grad_norm": 0.6079947352409363,
+      "learning_rate": 6.6731770833333345e-06,
+      "loss": 1.1298,
+      "mean_token_accuracy": 0.7505511298775673,
+      "num_tokens": 3389505.0,
+      "step": 560
+    },
+    {
+      "entropy": 1.1714913487434386,
+      "epoch": 0.7196969696969697,
+      "grad_norm": 0.6534572839736938,
+      "learning_rate": 6.6080729166666665e-06,
+      "loss": 1.144,
+      "mean_token_accuracy": 0.7481625184416771,
+      "num_tokens": 3449834.0,
+      "step": 570
+    },
+    {
+      "entropy": 1.1694963037967683,
+      "epoch": 0.7323232323232324,
+      "grad_norm": 0.5903164744377136,
+      "learning_rate": 6.54296875e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7476441130042076,
+      "num_tokens": 3510701.0,
+      "step": 580
+    },
+    {
+      "entropy": 1.1777653217315673,
+      "epoch": 0.7449494949494949,
+      "grad_norm": 0.6284182071685791,
+      "learning_rate": 6.477864583333334e-06,
+      "loss": 1.1422,
+      "mean_token_accuracy": 0.7490358456969262,
+      "num_tokens": 3570992.0,
+      "step": 590
+    },
+    {
+      "entropy": 1.1588268011808396,
+      "epoch": 0.7575757575757576,
+      "grad_norm": 0.6250146627426147,
+      "learning_rate": 6.412760416666667e-06,
+      "loss": 1.1336,
+      "mean_token_accuracy": 0.7504108369350433,
+      "num_tokens": 3631087.0,
+      "step": 600
+    },
+    {
+      "entropy": 1.1890273630619048,
+      "epoch": 0.7702020202020202,
+      "grad_norm": 0.6420578956604004,
+      "learning_rate": 6.3476562500000006e-06,
+      "loss": 1.1534,
+      "mean_token_accuracy": 0.7442372158169747,
+      "num_tokens": 3692871.0,
+      "step": 610
+    },
+    {
+      "entropy": 1.1817798465490341,
+      "epoch": 0.7828282828282829,
+      "grad_norm": 0.6156490445137024,
+      "learning_rate": 6.282552083333334e-06,
+      "loss": 1.1477,
+      "mean_token_accuracy": 0.7468263059854507,
+      "num_tokens": 3753671.0,
+      "step": 620
+    },
+    {
+      "entropy": 1.1686612635850906,
+      "epoch": 0.7954545454545454,
+      "grad_norm": 0.6248748898506165,
+      "learning_rate": 6.217447916666667e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7478924334049225,
+      "num_tokens": 3813110.0,
+      "step": 630
+    },
+    {
+      "entropy": 1.1387953266501427,
+      "epoch": 0.8080808080808081,
+      "grad_norm": 0.6052266359329224,
+      "learning_rate": 6.152343750000001e-06,
+      "loss": 1.1118,
+      "mean_token_accuracy": 0.7538487210869789,
+      "num_tokens": 3873477.0,
+      "step": 640
+    },
+    {
+      "entropy": 1.1343814879655838,
+      "epoch": 0.8207070707070707,
+      "grad_norm": 0.6769536137580872,
+      "learning_rate": 6.087239583333335e-06,
+      "loss": 1.1108,
+      "mean_token_accuracy": 0.7548464313149452,
+      "num_tokens": 3932873.0,
+      "step": 650
+    },
+    {
+      "entropy": 1.1686519652605056,
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.6545736789703369,
+      "learning_rate": 6.022135416666667e-06,
+      "loss": 1.134,
+      "mean_token_accuracy": 0.7495195478200912,
+      "num_tokens": 3992754.0,
+      "step": 660
+    },
+    {
+      "entropy": 1.1596283346414566,
+      "epoch": 0.8459595959595959,
+      "grad_norm": 0.6192017793655396,
+      "learning_rate": 5.95703125e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7510297149419785,
+      "num_tokens": 4053540.0,
+      "step": 670
+    },
+    {
+      "entropy": 1.1484344542026519,
+      "epoch": 0.8585858585858586,
+      "grad_norm": 0.6520631909370422,
+      "learning_rate": 5.891927083333334e-06,
+      "loss": 1.1181,
+      "mean_token_accuracy": 0.751795919239521,
+      "num_tokens": 4113582.0,
+      "step": 680
+    },
+    {
+      "entropy": 1.1519750133156776,
+      "epoch": 0.8712121212121212,
+      "grad_norm": 0.6247655153274536,
+      "learning_rate": 5.826822916666667e-06,
+      "loss": 1.1204,
+      "mean_token_accuracy": 0.7506368085741997,
+      "num_tokens": 4174983.0,
+      "step": 690
+    },
+    {
+      "entropy": 1.1486561581492425,
+      "epoch": 0.8838383838383839,
+      "grad_norm": 0.620272159576416,
+      "learning_rate": 5.761718750000001e-06,
+      "loss": 1.1191,
+      "mean_token_accuracy": 0.7536391675472259,
+      "num_tokens": 4234465.0,
+      "step": 700
+    },
+    {
+      "entropy": 1.1504923462867738,
+      "epoch": 0.8964646464646465,
+      "grad_norm": 0.6308649182319641,
+      "learning_rate": 5.6966145833333344e-06,
+      "loss": 1.1224,
+      "mean_token_accuracy": 0.750646598637104,
+      "num_tokens": 4295955.0,
+      "step": 710
+    },
+    {
+      "entropy": 1.1561898440122604,
+      "epoch": 0.9090909090909091,
+      "grad_norm": 0.6629899740219116,
+      "learning_rate": 5.6315104166666665e-06,
+      "loss": 1.1238,
+      "mean_token_accuracy": 0.7507557719945908,
+      "num_tokens": 4357171.0,
+      "step": 720
+    },
+    {
+      "entropy": 1.1413449853658677,
+      "epoch": 0.9217171717171717,
+      "grad_norm": 0.5972346067428589,
+      "learning_rate": 5.56640625e-06,
+      "loss": 1.1038,
+      "mean_token_accuracy": 0.7554104939103127,
+      "num_tokens": 4417636.0,
+      "step": 730
+    },
+    {
+      "entropy": 1.1356119453907012,
+      "epoch": 0.9343434343434344,
+      "grad_norm": 0.6356479525566101,
+      "learning_rate": 5.501302083333334e-06,
+      "loss": 1.1005,
+      "mean_token_accuracy": 0.7565629109740257,
+      "num_tokens": 4477294.0,
+      "step": 740
+    },
+    {
+      "entropy": 1.1633600294589996,
+      "epoch": 0.946969696969697,
+      "grad_norm": 0.6416464447975159,
+      "learning_rate": 5.436197916666667e-06,
+      "loss": 1.1225,
+      "mean_token_accuracy": 0.7515855401754379,
+      "num_tokens": 4537503.0,
+      "step": 750
+    },
+    {
+      "entropy": 1.1527763932943345,
+      "epoch": 0.9595959595959596,
+      "grad_norm": 0.6126084327697754,
+      "learning_rate": 5.3710937500000005e-06,
+      "loss": 1.1184,
+      "mean_token_accuracy": 0.7526160582900048,
+      "num_tokens": 4598778.0,
+      "step": 760
+    },
+    {
+      "entropy": 1.1397768080234527,
+      "epoch": 0.9722222222222222,
+      "grad_norm": 0.6359922289848328,
+      "learning_rate": 5.305989583333334e-06,
+      "loss": 1.1144,
+      "mean_token_accuracy": 0.7548302739858628,
+      "num_tokens": 4658978.0,
+      "step": 770
+    },
+    {
+      "entropy": 1.1569962561130525,
+      "epoch": 0.9848484848484849,
+      "grad_norm": 0.6260409951210022,
+      "learning_rate": 5.240885416666667e-06,
+      "loss": 1.1213,
+      "mean_token_accuracy": 0.7512885302305221,
+      "num_tokens": 4720500.0,
+      "step": 780
+    },
+    {
+      "entropy": 1.1509152203798294,
+      "epoch": 0.9974747474747475,
+      "grad_norm": 0.6293452978134155,
+      "learning_rate": 5.17578125e-06,
+      "loss": 1.1227,
+      "mean_token_accuracy": 0.7519564241170883,
+      "num_tokens": 4781612.0,
+      "step": 790
+    },
+    {
+      "entropy": 1.1399024561047555,
+      "epoch": 1.0101010101010102,
+      "grad_norm": 0.664761483669281,
+      "learning_rate": 5.110677083333334e-06,
+      "loss": 1.1034,
+      "mean_token_accuracy": 0.7526706486940384,
+      "num_tokens": 4841359.0,
+      "step": 800
+    },
+    {
+      "entropy": 1.120214229822159,
+      "epoch": 1.0227272727272727,
+      "grad_norm": 0.5934865474700928,
+      "learning_rate": 5.045572916666667e-06,
+      "loss": 1.0857,
+      "mean_token_accuracy": 0.7593594208359719,
+      "num_tokens": 4901016.0,
+      "step": 810
+    },
+    {
+      "entropy": 1.1430341199040412,
+      "epoch": 1.0353535353535352,
+      "grad_norm": 0.6040735840797424,
+      "learning_rate": 4.98046875e-06,
+      "loss": 1.1165,
+      "mean_token_accuracy": 0.7525988414883613,
+      "num_tokens": 4961646.0,
+      "step": 820
+    },
+    {
+      "entropy": 1.1245349109172822,
+      "epoch": 1.047979797979798,
+      "grad_norm": 0.6277610063552856,
+      "learning_rate": 4.915364583333333e-06,
+      "loss": 1.0851,
+      "mean_token_accuracy": 0.7577921718358993,
+      "num_tokens": 5022365.0,
+      "step": 830
+    },
+    {
+      "entropy": 1.1204675793647767,
+      "epoch": 1.0606060606060606,
+      "grad_norm": 0.6260582804679871,
+      "learning_rate": 4.850260416666667e-06,
+      "loss": 1.0813,
+      "mean_token_accuracy": 0.7580071151256561,
+      "num_tokens": 5081972.0,
+      "step": 840
+    },
+    {
+      "entropy": 1.1222782507538795,
+      "epoch": 1.0732323232323233,
+      "grad_norm": 0.6023226976394653,
+      "learning_rate": 4.785156250000001e-06,
+      "loss": 1.0922,
+      "mean_token_accuracy": 0.7566258609294891,
+      "num_tokens": 5142184.0,
+      "step": 850
+    },
+    {
+      "entropy": 1.1227335944771766,
+      "epoch": 1.0858585858585859,
+      "grad_norm": 0.6206791996955872,
+      "learning_rate": 4.7200520833333336e-06,
+      "loss": 1.0994,
+      "mean_token_accuracy": 0.7540625646710396,
+      "num_tokens": 5203020.0,
+      "step": 860
+    },
+    {
+      "entropy": 1.1352888554334641,
+      "epoch": 1.0984848484848484,
+      "grad_norm": 0.6301055550575256,
+      "learning_rate": 4.654947916666667e-06,
+      "loss": 1.0958,
+      "mean_token_accuracy": 0.7562039017677307,
+      "num_tokens": 5263291.0,
+      "step": 870
+    },
+    {
+      "entropy": 1.1120088309049607,
+      "epoch": 1.1111111111111112,
+      "grad_norm": 0.6210020780563354,
+      "learning_rate": 4.58984375e-06,
+      "loss": 1.0793,
+      "mean_token_accuracy": 0.7590557768940925,
+      "num_tokens": 5323972.0,
+      "step": 880
+    },
+    {
+      "entropy": 1.0996666207909584,
+      "epoch": 1.1237373737373737,
+      "grad_norm": 0.6332690715789795,
+      "learning_rate": 4.524739583333334e-06,
+      "loss": 1.0717,
+      "mean_token_accuracy": 0.7615471586585045,
+      "num_tokens": 5383911.0,
+      "step": 890
+    },
+    {
+      "entropy": 1.127107810974121,
+      "epoch": 1.1363636363636362,
+      "grad_norm": 0.6505516767501831,
+      "learning_rate": 4.459635416666668e-06,
+      "loss": 1.1027,
+      "mean_token_accuracy": 0.7562421515583992,
+      "num_tokens": 5445417.0,
+      "step": 900
+    },
+    {
+      "entropy": 1.129740473628044,
+      "epoch": 1.148989898989899,
+      "grad_norm": 0.6406158804893494,
+      "learning_rate": 4.3945312500000005e-06,
+      "loss": 1.0879,
+      "mean_token_accuracy": 0.7587148532271385,
+      "num_tokens": 5505455.0,
+      "step": 910
+    },
+    {
+      "entropy": 1.1167259424924851,
+      "epoch": 1.1616161616161615,
+      "grad_norm": 0.6297397613525391,
+      "learning_rate": 4.329427083333333e-06,
+      "loss": 1.0752,
+      "mean_token_accuracy": 0.7604142814874649,
+      "num_tokens": 5565311.0,
+      "step": 920
+    },
+    {
+      "entropy": 1.1037891641259194,
+      "epoch": 1.1742424242424243,
+      "grad_norm": 0.6490073204040527,
+      "learning_rate": 4.264322916666667e-06,
+      "loss": 1.0686,
+      "mean_token_accuracy": 0.7610052570700645,
+      "num_tokens": 5625358.0,
+      "step": 930
+    },
+    {
+      "entropy": 1.1061881184577942,
+      "epoch": 1.1868686868686869,
+      "grad_norm": 0.6366387009620667,
+      "learning_rate": 4.19921875e-06,
+      "loss": 1.0868,
+      "mean_token_accuracy": 0.7574937298893929,
+      "num_tokens": 5686421.0,
+      "step": 940
+    },
+    {
+      "entropy": 1.1124324068427085,
+      "epoch": 1.1994949494949494,
+      "grad_norm": 0.6556055545806885,
+      "learning_rate": 4.134114583333334e-06,
+      "loss": 1.0694,
+      "mean_token_accuracy": 0.7602224007248879,
+      "num_tokens": 5745891.0,
+      "step": 950
+    },
+    {
+      "entropy": 1.1175200879573821,
+      "epoch": 1.2121212121212122,
+      "grad_norm": 0.6404849886894226,
+      "learning_rate": 4.0690104166666675e-06,
+      "loss": 1.081,
+      "mean_token_accuracy": 0.7568994402885437,
+      "num_tokens": 5806078.0,
+      "step": 960
+    },
+    {
+      "entropy": 1.1186978340148925,
+      "epoch": 1.2247474747474747,
+      "grad_norm": 0.6227584481239319,
+      "learning_rate": 4.00390625e-06,
+      "loss": 1.0791,
+      "mean_token_accuracy": 0.759756401181221,
+      "num_tokens": 5866369.0,
+      "step": 970
+    },
+    {
+      "entropy": 1.122128139436245,
+      "epoch": 1.2373737373737375,
+      "grad_norm": 0.6616361141204834,
+      "learning_rate": 3.938802083333333e-06,
+      "loss": 1.0937,
+      "mean_token_accuracy": 0.7582718566060066,
+      "num_tokens": 5926217.0,
+      "step": 980
+    },
+    {
+      "entropy": 1.122861033678055,
+      "epoch": 1.25,
+      "grad_norm": 0.6384168267250061,
+      "learning_rate": 3.873697916666667e-06,
+      "loss": 1.0978,
+      "mean_token_accuracy": 0.7549964562058449,
+      "num_tokens": 5987666.0,
+      "step": 990
+    },
+    {
+      "entropy": 1.1277505576610565,
+      "epoch": 1.2626262626262625,
+      "grad_norm": 0.6038117408752441,
+      "learning_rate": 3.8085937500000002e-06,
+      "loss": 1.0952,
+      "mean_token_accuracy": 0.755272176861763,
+      "num_tokens": 6048708.0,
+      "step": 1000
+    },
+    {
+      "entropy": 1.1120157346129418,
+      "epoch": 1.2752525252525253,
+      "grad_norm": 0.6418159604072571,
+      "learning_rate": 3.7434895833333336e-06,
+      "loss": 1.078,
+      "mean_token_accuracy": 0.7594122514128685,
+      "num_tokens": 6109652.0,
+      "step": 1010
+    },
+    {
+      "entropy": 1.101425115764141,
+      "epoch": 1.2878787878787878,
+      "grad_norm": 0.6218425035476685,
+      "learning_rate": 3.6783854166666673e-06,
+      "loss": 1.0688,
+      "mean_token_accuracy": 0.7604865297675133,
+      "num_tokens": 6169125.0,
+      "step": 1020
+    },
+    {
+      "entropy": 1.1007713869214057,
+      "epoch": 1.3005050505050506,
+      "grad_norm": 0.6429149508476257,
+      "learning_rate": 3.61328125e-06,
+      "loss": 1.0581,
+      "mean_token_accuracy": 0.7621071562170982,
+      "num_tokens": 6230303.0,
+      "step": 1030
+    },
+    {
+      "entropy": 1.1094096556305886,
+      "epoch": 1.3131313131313131,
+      "grad_norm": 0.6489748358726501,
+      "learning_rate": 3.5481770833333335e-06,
+      "loss": 1.0715,
+      "mean_token_accuracy": 0.7599423810839653,
+      "num_tokens": 6291396.0,
+      "step": 1040
+    },
+    {
+      "entropy": 1.0827289715409278,
+      "epoch": 1.3257575757575757,
+      "grad_norm": 0.6485461592674255,
+      "learning_rate": 3.483072916666667e-06,
+      "loss": 1.0584,
+      "mean_token_accuracy": 0.7630694910883904,
+      "num_tokens": 6351579.0,
+      "step": 1050
+    },
+    {
+      "entropy": 1.114325873553753,
+      "epoch": 1.3383838383838385,
+      "grad_norm": 0.6261104941368103,
+      "learning_rate": 3.41796875e-06,
+      "loss": 1.0764,
+      "mean_token_accuracy": 0.7585488513112069,
+      "num_tokens": 6411662.0,
+      "step": 1060
+    },
+    {
+      "entropy": 1.1271554425358772,
+      "epoch": 1.351010101010101,
+      "grad_norm": 0.6522034406661987,
+      "learning_rate": 3.3528645833333334e-06,
+      "loss": 1.0902,
+      "mean_token_accuracy": 0.7562535598874092,
+      "num_tokens": 6473505.0,
+      "step": 1070
+    },
+    {
+      "entropy": 1.1013643085956573,
+      "epoch": 1.3636363636363638,
+      "grad_norm": 0.6176674962043762,
+      "learning_rate": 3.287760416666667e-06,
+      "loss": 1.065,
+      "mean_token_accuracy": 0.763075165450573,
+      "num_tokens": 6533580.0,
+      "step": 1080
+    },
+    {
+      "entropy": 1.098090337216854,
+      "epoch": 1.3762626262626263,
+      "grad_norm": 0.6540253758430481,
+      "learning_rate": 3.2226562500000004e-06,
+      "loss": 1.0596,
+      "mean_token_accuracy": 0.7616770043969154,
+      "num_tokens": 6593481.0,
+      "step": 1090
+    },
+    {
+      "entropy": 1.1176372200250626,
+      "epoch": 1.3888888888888888,
+      "grad_norm": 0.6754550933837891,
+      "learning_rate": 3.1575520833333333e-06,
+      "loss": 1.0861,
+      "mean_token_accuracy": 0.7573029339313507,
+      "num_tokens": 6653967.0,
+      "step": 1100
+    },
+    {
+      "entropy": 1.1040414482355119,
+      "epoch": 1.4015151515151514,
+      "grad_norm": 0.6022531986236572,
+      "learning_rate": 3.092447916666667e-06,
+      "loss": 1.0573,
+      "mean_token_accuracy": 0.7612267225980759,
+      "num_tokens": 6714685.0,
+      "step": 1110
+    },
+    {
+      "entropy": 1.0926991075277328,
+      "epoch": 1.4141414141414141,
+      "grad_norm": 0.6621010303497314,
+      "learning_rate": 3.0273437500000003e-06,
+      "loss": 1.0637,
+      "mean_token_accuracy": 0.7612977519631385,
+      "num_tokens": 6774659.0,
+      "step": 1120
+    },
+    {
+      "entropy": 1.1095534324645997,
+      "epoch": 1.4267676767676767,
+      "grad_norm": 0.62503981590271,
+      "learning_rate": 2.962239583333333e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7618053883314133,
+      "num_tokens": 6834579.0,
+      "step": 1130
+    },
+    {
+      "entropy": 1.117809349298477,
+      "epoch": 1.4393939393939394,
+      "grad_norm": 0.6527109742164612,
+      "learning_rate": 2.897135416666667e-06,
+      "loss": 1.0747,
+      "mean_token_accuracy": 0.759482853114605,
+      "num_tokens": 6894074.0,
+      "step": 1140
+    },
+    {
+      "entropy": 1.1005077749490737,
+      "epoch": 1.452020202020202,
+      "grad_norm": 0.6720954775810242,
+      "learning_rate": 2.8320312500000002e-06,
+      "loss": 1.0607,
+      "mean_token_accuracy": 0.7621246844530105,
+      "num_tokens": 6953870.0,
+      "step": 1150
+    },
+    {
+      "entropy": 1.1236482918262483,
+      "epoch": 1.4646464646464645,
+      "grad_norm": 0.658524215221405,
+      "learning_rate": 2.7669270833333335e-06,
+      "loss": 1.0884,
+      "mean_token_accuracy": 0.7560836613178253,
+      "num_tokens": 7014553.0,
+      "step": 1160
+    },
+    {
+      "entropy": 1.1116504594683647,
+      "epoch": 1.4772727272727273,
+      "grad_norm": 0.6261802911758423,
+      "learning_rate": 2.7018229166666673e-06,
+      "loss": 1.0659,
+      "mean_token_accuracy": 0.7597616642713547,
+      "num_tokens": 7076291.0,
+      "step": 1170
+    },
+    {
+      "entropy": 1.073892480134964,
+      "epoch": 1.4898989898989898,
+      "grad_norm": 0.6310375332832336,
+      "learning_rate": 2.63671875e-06,
+      "loss": 1.0524,
+      "mean_token_accuracy": 0.7628733053803444,
+      "num_tokens": 7137305.0,
+      "step": 1180
+    },
+    {
+      "entropy": 1.0975843235850333,
+      "epoch": 1.5025252525252526,
+      "grad_norm": 0.638482391834259,
+      "learning_rate": 2.5716145833333334e-06,
+      "loss": 1.0679,
+      "mean_token_accuracy": 0.7603248566389084,
+      "num_tokens": 7198239.0,
+      "step": 1190
+    },
+    {
+      "entropy": 1.0986508697271347,
+      "epoch": 1.5151515151515151,
+      "grad_norm": 0.640065610408783,
+      "learning_rate": 2.506510416666667e-06,
+      "loss": 1.0666,
+      "mean_token_accuracy": 0.7622530281543731,
+      "num_tokens": 7257847.0,
+      "step": 1200
+    },
+    {
+      "entropy": 1.0971406906843186,
+      "epoch": 1.5277777777777777,
+      "grad_norm": 0.6437165141105652,
+      "learning_rate": 2.44140625e-06,
+      "loss": 1.0587,
+      "mean_token_accuracy": 0.7623877748847008,
+      "num_tokens": 7317615.0,
+      "step": 1210
+    },
+    {
+      "entropy": 1.1032136514782906,
+      "epoch": 1.5404040404040404,
+      "grad_norm": 0.6590547561645508,
+      "learning_rate": 2.3763020833333338e-06,
+      "loss": 1.0569,
+      "mean_token_accuracy": 0.7616146191954613,
+      "num_tokens": 7377946.0,
+      "step": 1220
+    },
+    {
+      "entropy": 1.0922824308276176,
+      "epoch": 1.553030303030303,
+      "grad_norm": 0.6317723989486694,
+      "learning_rate": 2.3111979166666667e-06,
+      "loss": 1.0616,
+      "mean_token_accuracy": 0.7611089378595353,
+      "num_tokens": 7438666.0,
+      "step": 1230
+    },
+    {
+      "entropy": 1.1092566132545472,
+      "epoch": 1.5656565656565657,
+      "grad_norm": 0.66637122631073,
+      "learning_rate": 2.2460937500000004e-06,
+      "loss": 1.0725,
+      "mean_token_accuracy": 0.7594954133033752,
+      "num_tokens": 7499747.0,
+      "step": 1240
+    },
+    {
+      "entropy": 1.111878876388073,
+      "epoch": 1.5782828282828283,
+      "grad_norm": 0.6520881652832031,
+      "learning_rate": 2.1809895833333337e-06,
+      "loss": 1.074,
+      "mean_token_accuracy": 0.7558068126440048,
+      "num_tokens": 7561004.0,
+      "step": 1250
+    },
+    {
+      "entropy": 1.1105972841382026,
+      "epoch": 1.5909090909090908,
+      "grad_norm": 0.6495437622070312,
+      "learning_rate": 2.1158854166666666e-06,
+      "loss": 1.0844,
+      "mean_token_accuracy": 0.7563070356845856,
+      "num_tokens": 7622115.0,
+      "step": 1260
+    },
+    {
+      "entropy": 1.0932338371872903,
+      "epoch": 1.6035353535353534,
+      "grad_norm": 0.6420316696166992,
+      "learning_rate": 2.0507812500000003e-06,
+      "loss": 1.0649,
+      "mean_token_accuracy": 0.7622412323951722,
+      "num_tokens": 7682210.0,
+      "step": 1270
+    },
+    {
+      "entropy": 1.0919055327773095,
+      "epoch": 1.6161616161616161,
+      "grad_norm": 0.6192623972892761,
+      "learning_rate": 1.9856770833333336e-06,
+      "loss": 1.0427,
+      "mean_token_accuracy": 0.7657591253519058,
+      "num_tokens": 7742745.0,
+      "step": 1280
+    },
+    {
+      "entropy": 1.1026942864060403,
+      "epoch": 1.628787878787879,
+      "grad_norm": 0.6355161666870117,
+      "learning_rate": 1.920572916666667e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7609635755419731,
+      "num_tokens": 7802902.0,
+      "step": 1290
+    },
+    {
+      "entropy": 1.1140723824501038,
+      "epoch": 1.6414141414141414,
+      "grad_norm": 0.6254522800445557,
+      "learning_rate": 1.8554687500000002e-06,
+      "loss": 1.0789,
+      "mean_token_accuracy": 0.7592580512166023,
+      "num_tokens": 7863576.0,
+      "step": 1300
+    },
+    {
+      "entropy": 1.108394268155098,
+      "epoch": 1.654040404040404,
+      "grad_norm": 0.633172333240509,
+      "learning_rate": 1.7903645833333335e-06,
+      "loss": 1.0714,
+      "mean_token_accuracy": 0.7598690986633301,
+      "num_tokens": 7925518.0,
+      "step": 1310
+    },
+    {
+      "entropy": 1.1097407966852189,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 0.6279735565185547,
+      "learning_rate": 1.7252604166666668e-06,
+      "loss": 1.0701,
+      "mean_token_accuracy": 0.7611567705869675,
+      "num_tokens": 7987388.0,
+      "step": 1320
+    },
+    {
+      "entropy": 1.1071213275194167,
+      "epoch": 1.6792929292929293,
+      "grad_norm": 0.6425778269767761,
+      "learning_rate": 1.6601562500000001e-06,
+      "loss": 1.0786,
+      "mean_token_accuracy": 0.7576610520482063,
+      "num_tokens": 8048853.0,
+      "step": 1330
+    },
+    {
+      "entropy": 1.0931925728917122,
+      "epoch": 1.691919191919192,
+      "grad_norm": 0.666192889213562,
+      "learning_rate": 1.5950520833333336e-06,
+      "loss": 1.0604,
+      "mean_token_accuracy": 0.7616572439670563,
+      "num_tokens": 8108967.0,
+      "step": 1340
+    },
+    {
+      "entropy": 1.0973364993929864,
+      "epoch": 1.7045454545454546,
+      "grad_norm": 0.6348255276679993,
+      "learning_rate": 1.5299479166666667e-06,
+      "loss": 1.0769,
+      "mean_token_accuracy": 0.7596119627356529,
+      "num_tokens": 8169700.0,
+      "step": 1350
+    },
+    {
+      "entropy": 1.1137071400880814,
+      "epoch": 1.7171717171717171,
+      "grad_norm": 0.6510699391365051,
+      "learning_rate": 1.46484375e-06,
+      "loss": 1.0731,
+      "mean_token_accuracy": 0.7593250289559365,
+      "num_tokens": 8229676.0,
+      "step": 1360
+    },
+    {
+      "entropy": 1.1052428260445595,
+      "epoch": 1.7297979797979797,
+      "grad_norm": 0.6622318625450134,
+      "learning_rate": 1.3997395833333335e-06,
+      "loss": 1.069,
+      "mean_token_accuracy": 0.7627501472830772,
+      "num_tokens": 8289396.0,
+      "step": 1370
+    },
+    {
+      "entropy": 1.0989198789000512,
+      "epoch": 1.7424242424242424,
+      "grad_norm": 0.6430277824401855,
+      "learning_rate": 1.3346354166666666e-06,
+      "loss": 1.0506,
+      "mean_token_accuracy": 0.7634323209524154,
+      "num_tokens": 8351152.0,
+      "step": 1380
+    },
+    {
+      "entropy": 1.0913211867213248,
+      "epoch": 1.7550505050505052,
+      "grad_norm": 0.639707088470459,
+      "learning_rate": 1.2695312500000002e-06,
+      "loss": 1.0505,
+      "mean_token_accuracy": 0.763850274682045,
+      "num_tokens": 8411357.0,
+      "step": 1390
+    },
+    {
+      "entropy": 1.1044820442795753,
+      "epoch": 1.7676767676767677,
+      "grad_norm": 0.680479109287262,
+      "learning_rate": 1.2044270833333335e-06,
+      "loss": 1.0756,
+      "mean_token_accuracy": 0.7598197475075722,
+      "num_tokens": 8471975.0,
+      "step": 1400
+    },
+    {
+      "entropy": 1.0821994885802269,
+      "epoch": 1.7803030303030303,
+      "grad_norm": 0.651622474193573,
+      "learning_rate": 1.1393229166666668e-06,
+      "loss": 1.048,
+      "mean_token_accuracy": 0.7638402819633484,
+      "num_tokens": 8532480.0,
+      "step": 1410
+    },
+    {
+      "entropy": 1.0933044001460075,
+      "epoch": 1.7929292929292928,
+      "grad_norm": 0.6294305920600891,
+      "learning_rate": 1.07421875e-06,
+      "loss": 1.0545,
+      "mean_token_accuracy": 0.7633207753300667,
+      "num_tokens": 8593538.0,
+      "step": 1420
+    },
+    {
+      "entropy": 1.1002878457307816,
+      "epoch": 1.8055555555555556,
+      "grad_norm": 0.6396600008010864,
+      "learning_rate": 1.0091145833333334e-06,
+      "loss": 1.065,
+      "mean_token_accuracy": 0.7610213488340378,
+      "num_tokens": 8654446.0,
+      "step": 1430
+    },
+    {
+      "entropy": 1.1098253890872,
+      "epoch": 1.8181818181818183,
+      "grad_norm": 0.6585692167282104,
+      "learning_rate": 9.440104166666668e-07,
+      "loss": 1.079,
+      "mean_token_accuracy": 0.7590781077742577,
+      "num_tokens": 8715305.0,
+      "step": 1440
+    },
+    {
+      "entropy": 1.0894212126731873,
+      "epoch": 1.8308080808080809,
+      "grad_norm": 0.6637106537818909,
+      "learning_rate": 8.789062500000001e-07,
+      "loss": 1.0555,
+      "mean_token_accuracy": 0.7621971383690834,
+      "num_tokens": 8775351.0,
+      "step": 1450
+    },
+    {
+      "entropy": 1.0821923539042473,
+      "epoch": 1.8434343434343434,
+      "grad_norm": 0.6491685509681702,
+      "learning_rate": 8.138020833333334e-07,
+      "loss": 1.0464,
+      "mean_token_accuracy": 0.7639922067523003,
+      "num_tokens": 8836131.0,
+      "step": 1460
+    },
+    {
+      "entropy": 1.1058385655283929,
+      "epoch": 1.856060606060606,
+      "grad_norm": 0.6781795024871826,
+      "learning_rate": 7.486979166666668e-07,
+      "loss": 1.0682,
+      "mean_token_accuracy": 0.7595004603266716,
+      "num_tokens": 8896334.0,
+      "step": 1470
+    },
+    {
+      "entropy": 1.0934822604060173,
+      "epoch": 1.8686868686868687,
+      "grad_norm": 0.652746319770813,
+      "learning_rate": 6.835937500000001e-07,
+      "loss": 1.0625,
+      "mean_token_accuracy": 0.7614723727107048,
+      "num_tokens": 8956948.0,
+      "step": 1480
+    },
+    {
+      "entropy": 1.0955146595835685,
+      "epoch": 1.8813131313131313,
+      "grad_norm": 0.6350075006484985,
+      "learning_rate": 6.184895833333334e-07,
+      "loss": 1.0677,
+      "mean_token_accuracy": 0.75977371186018,
+      "num_tokens": 9017330.0,
+      "step": 1490
+    },
+    {
+      "entropy": 1.1074895232915878,
+      "epoch": 1.893939393939394,
+      "grad_norm": 0.6651970744132996,
+      "learning_rate": 5.533854166666667e-07,
+      "loss": 1.0778,
+      "mean_token_accuracy": 0.7592803448438644,
+      "num_tokens": 9077561.0,
+      "step": 1500
+    },
+    {
+      "entropy": 1.088778705894947,
+      "epoch": 1.9065656565656566,
+      "grad_norm": 0.6216638684272766,
+      "learning_rate": 4.8828125e-07,
+      "loss": 1.0517,
+      "mean_token_accuracy": 0.7628024965524673,
+      "num_tokens": 9137424.0,
+      "step": 1510
+    },
+    {
+      "entropy": 1.0963940545916557,
+      "epoch": 1.9191919191919191,
+      "grad_norm": 0.6801443099975586,
+      "learning_rate": 4.2317708333333337e-07,
+      "loss": 1.0657,
+      "mean_token_accuracy": 0.7611119478940964,
+      "num_tokens": 9198465.0,
+      "step": 1520
+    },
+    {
+      "entropy": 1.107513566315174,
+      "epoch": 1.9318181818181817,
+      "grad_norm": 0.6482690572738647,
+      "learning_rate": 3.5807291666666667e-07,
+      "loss": 1.0657,
+      "mean_token_accuracy": 0.7620691254734993,
+      "num_tokens": 9258486.0,
+      "step": 1530
+    },
+    {
+      "entropy": 1.1056584566831589,
+      "epoch": 1.9444444444444444,
+      "grad_norm": 0.6314805746078491,
+      "learning_rate": 2.9296875000000003e-07,
+      "loss": 1.0758,
+      "mean_token_accuracy": 0.7584919854998589,
+      "num_tokens": 9319619.0,
+      "step": 1540
+    },
+    {
+      "entropy": 1.1117310538887977,
+      "epoch": 1.9570707070707072,
+      "grad_norm": 0.6383644938468933,
+      "learning_rate": 2.2786458333333333e-07,
+      "loss": 1.0745,
+      "mean_token_accuracy": 0.7591810420155525,
+      "num_tokens": 9380350.0,
+      "step": 1550
+    },
+    {
+      "entropy": 1.0970366701483727,
+      "epoch": 1.9696969696969697,
+      "grad_norm": 0.6331989169120789,
+      "learning_rate": 1.627604166666667e-07,
+      "loss": 1.0557,
+      "mean_token_accuracy": 0.7604442983865738,
+      "num_tokens": 9441627.0,
+      "step": 1560
+    },
+    {
+      "entropy": 1.0917240902781487,
+      "epoch": 1.9823232323232323,
+      "grad_norm": 0.6618102192878723,
+      "learning_rate": 9.765625e-08,
+      "loss": 1.0546,
+      "mean_token_accuracy": 0.7638352930545806,
+      "num_tokens": 9501728.0,
+      "step": 1570
+    },
+    {
+      "entropy": 1.096452857553959,
+      "epoch": 1.9949494949494948,
+      "grad_norm": 0.6413847804069519,
+      "learning_rate": 3.2552083333333335e-08,
+      "loss": 1.0615,
+      "mean_token_accuracy": 0.7619210347533226,
+      "num_tokens": 9562238.0,
+      "step": 1580
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1584,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.4295385739381965e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1584/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eeb71c14deb91ac5fd11522db45cb3275c9164415fcbefc9d00cac27a27f0a3
+size 6417

sft_train.log

@@ -0,0 +1,157 @@
+2026-05-07 13:27:47,763 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:27:47,763 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:27:47,763 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:27:47,763 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:27:47,763 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:27:47,763 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:27:47,804 | INFO | CUDA available: True
+2026-05-07 13:27:47,825 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:27:47,826 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:27:47,827 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:27:47,827 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:27:47,827 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:27:49,034 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 13:30:30,504 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:30:34,280 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:30:40,321 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:30:40,717 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:30:40,818 | INFO | Train rows: 25340
+2026-05-07 13:30:40,819 | INFO | Eval rows: 6336
+2026-05-07 13:30:40,819 | INFO | Sample train example: [{'role': 'system', 'content': 'أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.'}, {'role': 'user', 'content': 'نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟'}, {'role': 'assistant', 'content': 'البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.'}]
+2026-05-07 13:30:40,819 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:34:03,570 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:34:03,570 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:34:03,570 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:34:03,570 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:34:03,570 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:34:03,570 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:34:03,612 | INFO | CUDA available: True
+2026-05-07 13:34:03,631 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:34:03,633 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:34:03,633 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:34:03,633 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:34:03,633 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:34:04,862 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 13:36:22,953 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:36:26,760 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:36:28,674 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:36:28,675 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:36:28,676 | INFO | Train rows: 25340
+2026-05-07 13:36:28,676 | INFO | Eval rows: 6336
+2026-05-07 13:36:28,677 | INFO | Sample train example: [{'role': 'system', 'content': 'أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.'}, {'role': 'user', 'content': 'نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟'}, {'role': 'assistant', 'content': 'البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش ��خون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.'}]
+2026-05-07 13:36:28,677 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:41:24,497 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:41:24,497 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:41:24,497 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:41:24,497 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:41:24,497 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:41:24,497 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:41:24,548 | INFO | CUDA available: True
+2026-05-07 13:41:24,576 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:41:24,577 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:41:24,578 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:41:24,578 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:41:24,578 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:41:25,691 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 13:44:02,778 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:44:06,613 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:44:08,694 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:44:08,697 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:44:08,698 | INFO | Train rows: 25340
+2026-05-07 13:44:08,699 | INFO | Eval rows: 6336
+2026-05-07 13:44:08,699 | INFO | Sample train example: [{'role': 'system', 'content': 'أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.'}, {'role': 'user', 'content': 'نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟'}, {'role': 'assistant', 'content': 'البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.'}]
+2026-05-07 13:44:08,699 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:44:08,752 | INFO | Training started
+2026-05-07 13:46:02,478 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:46:02,479 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:46:02,479 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:46:02,479 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:46:02,479 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:46:02,479 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:46:02,523 | INFO | CUDA available: True
+2026-05-07 13:46:02,552 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:46:02,554 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:46:02,555 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:46:02,555 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:46:02,555 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:46:03,698 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 13:48:19,475 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:48:23,292 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:48:25,199 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:48:25,202 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:48:25,203 | INFO | Train rows: 25340
+2026-05-07 13:48:25,203 | INFO | Eval rows: 6336
+2026-05-07 13:48:25,204 | INFO | Sample train example: [{'role': 'system', 'content': 'أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.'}, {'role': 'user', 'content': 'نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟'}, {'role': 'assistant', 'content': 'البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.'}]
+2026-05-07 13:48:25,204 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:48:25,244 | INFO | Training started
+2026-05-07 13:55:23,761 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:55:23,761 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:55:23,761 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:55:23,761 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:55:23,761 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:55:23,761 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:55:23,803 | INFO | CUDA available: True
+2026-05-07 13:55:23,826 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:55:23,828 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:55:23,829 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:55:23,829 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:55:23,829 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:55:24,929 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 13:58:01,693 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:58:05,420 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:58:07,816 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:58:08,954 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 13:58:09,429 | INFO | Train rows: 25340
+2026-05-07 13:58:09,430 | INFO | Eval rows: 6336
+2026-05-07 13:58:09,430 | INFO | Sample train example text: <BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
+2026-05-07 13:58:09,430 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:59:25,638 | INFO | Starting SFT fine-tuning job
+2026-05-07 13:59:25,639 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 13:59:25,639 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:59:25,639 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 13:59:25,639 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 13:59:25,639 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 13:59:25,683 | INFO | CUDA available: True
+2026-05-07 13:59:25,709 | INFO | GPU: NVIDIA GB10
+2026-05-07 13:59:25,710 | INFO | No Hugging Face token provided; skipping login
+2026-05-07 13:59:25,711 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 13:59:25,711 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 13:59:25,711 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 13:59:26,819 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 14:02:05,419 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 14:02:09,265 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 14:02:11,637 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 14:02:11,918 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 14:02:12,159 | INFO | Train rows: 25340
+2026-05-07 14:02:12,160 | INFO | Eval rows: 6336
+2026-05-07 14:02:12,160 | INFO | Sample train example text: <BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
+2026-05-07 14:02:12,161 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 14:02:23,459 | INFO | Training started
+2026-05-07 14:08:07,243 | INFO | Starting SFT fine-tuning job
+2026-05-07 14:08:07,244 | INFO | Output directory: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 14:08:07,244 | INFO | Base model: CohereLabs/aya-expanse-8b
+2026-05-07 14:08:07,244 | INFO | CPT adapter: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 14:08:07,244 | INFO | Dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 14:08:07,244 | INFO | Torch version: 2.11.0+cu130
+2026-05-07 14:08:07,286 | INFO | CUDA available: True
+2026-05-07 14:08:07,309 | INFO | GPU: NVIDIA GB10
+2026-05-07 14:08:07,499 | INFO | Hugging Face login succeeded
+2026-05-07 14:08:07,500 | INFO | trl version 1.3.0: assistant_only_loss is fully supported
+2026-05-07 14:08:07,500 | INFO | Loading configuration: batch_size=8, accumulation=4, warmup=0.03, weight_decay=0.0100
+2026-05-07 14:08:07,500 | INFO | Loading tokenizer from base model: CohereLabs/aya-expanse-8b
+2026-05-07 14:08:08,618 | INFO | Loading base model: CohereLabs/aya-expanse-8b with full bf16 precision (no quantization)
+2026-05-07 14:10:19,279 | INFO | Loading CPT adapter from: /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-cpt-tunisian
+2026-05-07 14:10:23,076 | INFO | Loading dataset: Syrinesmati/tunisian-question-response-dataset
+2026-05-07 14:10:25,353 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 14:10:25,621 | INFO | Using dataset fields: question=instruction, answer=response
+2026-05-07 14:10:25,861 | INFO | Train rows: 25340
+2026-05-07 14:10:25,862 | INFO | Eval rows: 6336
+2026-05-07 14:10:25,862 | INFO | Sample train example text: <BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>نحس بوجيعة في العينين من كثرة غسلان الماعون بالماء السخون؟<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>البخار الصاعد من الماء السخون ينجم يتعب العين ويسبب احمرار، هذاكا علاش حاول تستعمل ماء دافي مش سخون برشة. اغسل وجهك بماء بارد بعد ما تكمل باش تبرد عينيك.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
+2026-05-07 14:10:25,862 | INFO | No checkpoint found in /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-07 14:10:29,187 | INFO | Training started
+2026-05-08 04:09:43,113 | INFO | Training finished
+2026-05-08 04:09:43,113 | INFO | Saving model and tokenizer to /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft
+2026-05-08 04:09:45,243 | INFO | Saved training metrics to /home/ala/TunisianDialogSystem/outputs/checkpoints/aya-expanse-8b-tunisian-sft/training_metrics.json
+2026-05-08 04:09:45,243 | INFO | Running preview generation on a Tunisian prompt
+2026-05-08 04:09:48,487 | INFO | Preview prompt: عسلامة، شنوة تنصحني نعمل كي نكون تعبان وبرشة؟
+2026-05-08 04:09:48,487 | INFO | Preview output: <|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>أنت "التيجاني"، مساعد ذكاء اصطناعي تونسي 100%. جاوب بالتونسي الدارج فقط، وبالطول المناسب للسؤال: كان يلزم قصّر، وكان يلزم فسّر أكثر. ممنوع الهلوسة أو الخروج على الموضوع.<|START_OF_TURN_TOKEN|><|USER_TOKEN|>عسلامة، شنوة تنصحني نعمل كي نكون تعبان وبرشة؟<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>ح في في في في في في في في في في<|START_OF_TURN_TOKEN|>

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<PAD>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:345ccf04a5257f473e331715ecc69365c5ac8fc2490923fe7155560af809ec1a
+size 20124090

tokenizer_config.json

@@ -0,0 +1,317 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<CLS>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<SEP>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<EOP_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255000": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255001": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255002": {
+      "content": "<|YES_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255003": {
+      "content": "<|NO_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255004": {
+      "content": "<|GOOD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255005": {
+      "content": "<|BAD_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255006": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255007": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255008": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255009": {
+      "content": "<|USER_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255010": {
+      "content": "<|USER_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255011": {
+      "content": "<|USER_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255012": {
+      "content": "<|USER_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255013": {
+      "content": "<|USER_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255014": {
+      "content": "<|USER_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255015": {
+      "content": "<|USER_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255016": {
+      "content": "<|USER_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255017": {
+      "content": "<|USER_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255018": {
+      "content": "<|USER_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255019": {
+      "content": "<|EXTRA_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255020": {
+      "content": "<|EXTRA_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255021": {
+      "content": "<|EXTRA_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255022": {
+      "content": "<|EXTRA_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255023": {
+      "content": "<|EXTRA_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255024": {
+      "content": "<|EXTRA_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255025": {
+      "content": "<|EXTRA_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255026": {
+      "content": "<|EXTRA_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255027": {
+      "content": "<|EXTRA_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255028": {
+      "content": "<|EXTRA_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<BOS_TOKEN>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|END_OF_TURN_TOKEN|>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "merges_file": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "CohereTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false,
+  "vocab_file": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eeb71c14deb91ac5fd11522db45cb3275c9164415fcbefc9d00cac27a27f0a3
+size 6417

training_metrics.json

@@ -0,0 +1,1600 @@
+{
+  "log_history": [
+    {
+      "loss": 3.6021,
+      "grad_norm": 4.66565465927124,
+      "learning_rate": 1.8750000000000003e-06,
+      "entropy": 2.4040649354457857,
+      "num_tokens": 61199.0,
+      "mean_token_accuracy": 0.4221017129719257,
+      "epoch": 0.012626262626262626,
+      "step": 10
+    },
+    {
+      "loss": 3.3432,
+      "grad_norm": 3.8161869049072266,
+      "learning_rate": 3.958333333333333e-06,
+      "entropy": 2.3836746215820312,
+      "num_tokens": 122423.0,
+      "mean_token_accuracy": 0.44042530804872515,
+      "epoch": 0.025252525252525252,
+      "step": 20
+    },
+    {
+      "loss": 2.9033,
+      "grad_norm": 3.8800699710845947,
+      "learning_rate": 6.041666666666667e-06,
+      "entropy": 2.355724626779556,
+      "num_tokens": 182649.0,
+      "mean_token_accuracy": 0.48426677361130716,
+      "epoch": 0.03787878787878788,
+      "step": 30
+    },
+    {
+      "loss": 2.356,
+      "grad_norm": 2.8217720985412598,
+      "learning_rate": 8.125000000000001e-06,
+      "entropy": 2.092331054806709,
+      "num_tokens": 243049.0,
+      "mean_token_accuracy": 0.5772452697157859,
+      "epoch": 0.050505050505050504,
+      "step": 40
+    },
+    {
+      "loss": 1.8899,
+      "grad_norm": 1.4623568058013916,
+      "learning_rate": 9.993489583333334e-06,
+      "entropy": 1.6766322344541549,
+      "num_tokens": 304326.0,
+      "mean_token_accuracy": 0.6480962842702865,
+      "epoch": 0.06313131313131314,
+      "step": 50
+    },
+    {
+      "loss": 1.677,
+      "grad_norm": 1.171562671661377,
+      "learning_rate": 9.928385416666668e-06,
+      "entropy": 1.5568815559148788,
+      "num_tokens": 364866.0,
+      "mean_token_accuracy": 0.6776855796575546,
+      "epoch": 0.07575757575757576,
+      "step": 60
+    },
+    {
+      "loss": 1.5337,
+      "grad_norm": 0.9904961585998535,
+      "learning_rate": 9.863281250000001e-06,
+      "entropy": 1.48199902176857,
+      "num_tokens": 423478.0,
+      "mean_token_accuracy": 0.697019773721695,
+      "epoch": 0.08838383838383838,
+      "step": 70
+    },
+    {
+      "loss": 1.4953,
+      "grad_norm": 0.9454260468482971,
+      "learning_rate": 9.798177083333335e-06,
+      "entropy": 1.497376424074173,
+      "num_tokens": 483659.0,
+      "mean_token_accuracy": 0.6976823702454567,
+      "epoch": 0.10101010101010101,
+      "step": 80
+    },
+    {
+      "loss": 1.4356,
+      "grad_norm": 0.8955270648002625,
+      "learning_rate": 9.733072916666667e-06,
+      "entropy": 1.4664768785238267,
+      "num_tokens": 544453.0,
+      "mean_token_accuracy": 0.7069446608424187,
+      "epoch": 0.11363636363636363,
+      "step": 90
+    },
+    {
+      "loss": 1.4106,
+      "grad_norm": 0.9242203235626221,
+      "learning_rate": 9.66796875e-06,
+      "entropy": 1.4269085675477982,
+      "num_tokens": 604546.0,
+      "mean_token_accuracy": 0.7122392952442169,
+      "epoch": 0.12626262626262627,
+      "step": 100
+    },
+    {
+      "loss": 1.3487,
+      "grad_norm": 0.8968560695648193,
+      "learning_rate": 9.602864583333335e-06,
+      "entropy": 1.4060751020908355,
+      "num_tokens": 664860.0,
+      "mean_token_accuracy": 0.7178552970290184,
+      "epoch": 0.1388888888888889,
+      "step": 110
+    },
+    {
+      "loss": 1.3347,
+      "grad_norm": 0.9047113656997681,
+      "learning_rate": 9.537760416666667e-06,
+      "entropy": 1.4018951296806335,
+      "num_tokens": 725022.0,
+      "mean_token_accuracy": 0.7208079636096955,
+      "epoch": 0.15151515151515152,
+      "step": 120
+    },
+    {
+      "loss": 1.3155,
+      "grad_norm": 0.8915444016456604,
+      "learning_rate": 9.47265625e-06,
+      "entropy": 1.3809731483459473,
+      "num_tokens": 785586.0,
+      "mean_token_accuracy": 0.7267520889639855,
+      "epoch": 0.16414141414141414,
+      "step": 130
+    },
+    {
+      "loss": 1.3016,
+      "grad_norm": 0.8574295043945312,
+      "learning_rate": 9.407552083333334e-06,
+      "entropy": 1.3699676394462585,
+      "num_tokens": 845790.0,
+      "mean_token_accuracy": 0.7266673430800438,
+      "epoch": 0.17676767676767677,
+      "step": 140
+    },
+    {
+      "loss": 1.2823,
+      "grad_norm": 0.8231800198554993,
+      "learning_rate": 9.342447916666668e-06,
+      "entropy": 1.3425012439489366,
+      "num_tokens": 905842.0,
+      "mean_token_accuracy": 0.7277117937803268,
+      "epoch": 0.1893939393939394,
+      "step": 150
+    },
+    {
+      "loss": 1.2917,
+      "grad_norm": 0.8166369795799255,
+      "learning_rate": 9.277343750000001e-06,
+      "entropy": 1.3314216613769532,
+      "num_tokens": 966487.0,
+      "mean_token_accuracy": 0.7278609350323677,
+      "epoch": 0.20202020202020202,
+      "step": 160
+    },
+    {
+      "loss": 1.2481,
+      "grad_norm": 0.7738587260246277,
+      "learning_rate": 9.212239583333335e-06,
+      "entropy": 1.30389544069767,
+      "num_tokens": 1025558.0,
+      "mean_token_accuracy": 0.7335339426994324,
+      "epoch": 0.21464646464646464,
+      "step": 170
+    },
+    {
+      "loss": 1.2643,
+      "grad_norm": 0.7718328833580017,
+      "learning_rate": 9.147135416666667e-06,
+      "entropy": 1.311449444293976,
+      "num_tokens": 1086677.0,
+      "mean_token_accuracy": 0.7285080313682556,
+      "epoch": 0.22727272727272727,
+      "step": 180
+    },
+    {
+      "loss": 1.2641,
+      "grad_norm": 0.7341915369033813,
+      "learning_rate": 9.082031250000001e-06,
+      "entropy": 1.3079361200332642,
+      "num_tokens": 1147885.0,
+      "mean_token_accuracy": 0.729550538957119,
+      "epoch": 0.2398989898989899,
+      "step": 190
+    },
+    {
+      "loss": 1.2397,
+      "grad_norm": 0.748540997505188,
+      "learning_rate": 9.016927083333335e-06,
+      "entropy": 1.2794219702482224,
+      "num_tokens": 1207321.0,
+      "mean_token_accuracy": 0.7351120054721832,
+      "epoch": 0.25252525252525254,
+      "step": 200
+    },
+    {
+      "loss": 1.2489,
+      "grad_norm": 0.7585553526878357,
+      "learning_rate": 8.951822916666667e-06,
+      "entropy": 1.294454461336136,
+      "num_tokens": 1267837.0,
+      "mean_token_accuracy": 0.7322148531675339,
+      "epoch": 0.26515151515151514,
+      "step": 210
+    },
+    {
+      "loss": 1.2383,
+      "grad_norm": 0.6937864422798157,
+      "learning_rate": 8.88671875e-06,
+      "entropy": 1.2862246632575989,
+      "num_tokens": 1328436.0,
+      "mean_token_accuracy": 0.7358245223760604,
+      "epoch": 0.2777777777777778,
+      "step": 220
+    },
+    {
+      "loss": 1.2007,
+      "grad_norm": 0.6792387366294861,
+      "learning_rate": 8.821614583333334e-06,
+      "entropy": 1.2531811505556107,
+      "num_tokens": 1389877.0,
+      "mean_token_accuracy": 0.7360369265079498,
+      "epoch": 0.2904040404040404,
+      "step": 230
+    },
+    {
+      "loss": 1.2474,
+      "grad_norm": 0.6865427494049072,
+      "learning_rate": 8.756510416666666e-06,
+      "entropy": 1.2815489560365676,
+      "num_tokens": 1450927.0,
+      "mean_token_accuracy": 0.7304804190993309,
+      "epoch": 0.30303030303030304,
+      "step": 240
+    },
+    {
+      "loss": 1.2172,
+      "grad_norm": 0.669840395450592,
+      "learning_rate": 8.69140625e-06,
+      "entropy": 1.2568059146404267,
+      "num_tokens": 1511174.0,
+      "mean_token_accuracy": 0.7385278165340423,
+      "epoch": 0.31565656565656564,
+      "step": 250
+    },
+    {
+      "loss": 1.213,
+      "grad_norm": 0.6434893012046814,
+      "learning_rate": 8.626302083333334e-06,
+      "entropy": 1.254868358373642,
+      "num_tokens": 1570570.0,
+      "mean_token_accuracy": 0.7380544006824493,
+      "epoch": 0.3282828282828283,
+      "step": 260
+    },
+    {
+      "loss": 1.2116,
+      "grad_norm": 0.6034978032112122,
+      "learning_rate": 8.561197916666667e-06,
+      "entropy": 1.2532441645860672,
+      "num_tokens": 1630930.0,
+      "mean_token_accuracy": 0.7382062628865242,
+      "epoch": 0.3409090909090909,
+      "step": 270
+    },
+    {
+      "loss": 1.2313,
+      "grad_norm": 0.6371450424194336,
+      "learning_rate": 8.496093750000001e-06,
+      "entropy": 1.2605280816555022,
+      "num_tokens": 1692624.0,
+      "mean_token_accuracy": 0.7328852489590645,
+      "epoch": 0.35353535353535354,
+      "step": 280
+    },
+    {
+      "loss": 1.2195,
+      "grad_norm": 0.6300278306007385,
+      "learning_rate": 8.430989583333335e-06,
+      "entropy": 1.248606452345848,
+      "num_tokens": 1754213.0,
+      "mean_token_accuracy": 0.7370449885725975,
+      "epoch": 0.3661616161616162,
+      "step": 290
+    },
+    {
+      "loss": 1.1845,
+      "grad_norm": 0.6430155634880066,
+      "learning_rate": 8.365885416666667e-06,
+      "entropy": 1.2246102809906005,
+      "num_tokens": 1813841.0,
+      "mean_token_accuracy": 0.7439196646213532,
+      "epoch": 0.3787878787878788,
+      "step": 300
+    },
+    {
+      "loss": 1.1856,
+      "grad_norm": 0.6395701766014099,
+      "learning_rate": 8.30078125e-06,
+      "entropy": 1.2188815206289292,
+      "num_tokens": 1873568.0,
+      "mean_token_accuracy": 0.7422183871269226,
+      "epoch": 0.39141414141414144,
+      "step": 310
+    },
+    {
+      "loss": 1.1951,
+      "grad_norm": 0.6168740391731262,
+      "learning_rate": 8.235677083333334e-06,
+      "entropy": 1.234528934955597,
+      "num_tokens": 1935518.0,
+      "mean_token_accuracy": 0.7380509555339814,
+      "epoch": 0.40404040404040403,
+      "step": 320
+    },
+    {
+      "loss": 1.2078,
+      "grad_norm": 0.611132800579071,
+      "learning_rate": 8.170572916666666e-06,
+      "entropy": 1.226365676522255,
+      "num_tokens": 1995604.0,
+      "mean_token_accuracy": 0.739974245429039,
+      "epoch": 0.4166666666666667,
+      "step": 330
+    },
+    {
+      "loss": 1.1642,
+      "grad_norm": 0.6103131771087646,
+      "learning_rate": 8.10546875e-06,
+      "entropy": 1.2132183194160462,
+      "num_tokens": 2056484.0,
+      "mean_token_accuracy": 0.7444137379527092,
+      "epoch": 0.4292929292929293,
+      "step": 340
+    },
+    {
+      "loss": 1.2001,
+      "grad_norm": 0.6188805103302002,
+      "learning_rate": 8.040364583333334e-06,
+      "entropy": 1.223158246278763,
+      "num_tokens": 2118437.0,
+      "mean_token_accuracy": 0.7385613292455673,
+      "epoch": 0.44191919191919193,
+      "step": 350
+    },
+    {
+      "loss": 1.1848,
+      "grad_norm": 0.6238694190979004,
+      "learning_rate": 7.975260416666668e-06,
+      "entropy": 1.2154075980186463,
+      "num_tokens": 2179333.0,
+      "mean_token_accuracy": 0.7404153689742088,
+      "epoch": 0.45454545454545453,
+      "step": 360
+    },
+    {
+      "loss": 1.1597,
+      "grad_norm": 0.6028566956520081,
+      "learning_rate": 7.910156250000001e-06,
+      "entropy": 1.197928261756897,
+      "num_tokens": 2239604.0,
+      "mean_token_accuracy": 0.7475010469555855,
+      "epoch": 0.4671717171717172,
+      "step": 370
+    },
+    {
+      "loss": 1.1805,
+      "grad_norm": 0.6569434404373169,
+      "learning_rate": 7.845052083333335e-06,
+      "entropy": 1.189855706691742,
+      "num_tokens": 2300936.0,
+      "mean_token_accuracy": 0.7427218139171601,
+      "epoch": 0.4797979797979798,
+      "step": 380
+    },
+    {
+      "loss": 1.1759,
+      "grad_norm": 0.6351733207702637,
+      "learning_rate": 7.779947916666667e-06,
+      "entropy": 1.2076493889093398,
+      "num_tokens": 2361539.0,
+      "mean_token_accuracy": 0.7408407002687454,
+      "epoch": 0.49242424242424243,
+      "step": 390
+    },
+    {
+      "loss": 1.1874,
+      "grad_norm": 0.6327986121177673,
+      "learning_rate": 7.71484375e-06,
+      "entropy": 1.2223081022500992,
+      "num_tokens": 2422162.0,
+      "mean_token_accuracy": 0.7407250568270684,
+      "epoch": 0.5050505050505051,
+      "step": 400
+    },
+    {
+      "loss": 1.1726,
+      "grad_norm": 0.622104823589325,
+      "learning_rate": 7.649739583333334e-06,
+      "entropy": 1.2013412863016129,
+      "num_tokens": 2483447.0,
+      "mean_token_accuracy": 0.744242025911808,
+      "epoch": 0.5176767676767676,
+      "step": 410
+    },
+    {
+      "loss": 1.1838,
+      "grad_norm": 0.637651264667511,
+      "learning_rate": 7.5846354166666665e-06,
+      "entropy": 1.2116613179445266,
+      "num_tokens": 2544848.0,
+      "mean_token_accuracy": 0.7405030101537704,
+      "epoch": 0.5303030303030303,
+      "step": 420
+    },
+    {
+      "loss": 1.1681,
+      "grad_norm": 0.6252374649047852,
+      "learning_rate": 7.51953125e-06,
+      "entropy": 1.2024697184562683,
+      "num_tokens": 2605232.0,
+      "mean_token_accuracy": 0.7458183988928795,
+      "epoch": 0.5429292929292929,
+      "step": 430
+    },
+    {
+      "loss": 1.1452,
+      "grad_norm": 0.6502755284309387,
+      "learning_rate": 7.454427083333334e-06,
+      "entropy": 1.1797083109617232,
+      "num_tokens": 2664276.0,
+      "mean_token_accuracy": 0.7477341219782829,
+      "epoch": 0.5555555555555556,
+      "step": 440
+    },
+    {
+      "loss": 1.1665,
+      "grad_norm": 0.639979362487793,
+      "learning_rate": 7.389322916666667e-06,
+      "entropy": 1.1964200481772422,
+      "num_tokens": 2724073.0,
+      "mean_token_accuracy": 0.7431837096810341,
+      "epoch": 0.5681818181818182,
+      "step": 450
+    },
+    {
+      "loss": 1.1529,
+      "grad_norm": 0.6212354302406311,
+      "learning_rate": 7.3242187500000006e-06,
+      "entropy": 1.1795272737741471,
+      "num_tokens": 2784262.0,
+      "mean_token_accuracy": 0.7479098170995713,
+      "epoch": 0.5808080808080808,
+      "step": 460
+    },
+    {
+      "loss": 1.1678,
+      "grad_norm": 0.6528693437576294,
+      "learning_rate": 7.259114583333334e-06,
+      "entropy": 1.1908618807792664,
+      "num_tokens": 2843804.0,
+      "mean_token_accuracy": 0.745174677670002,
+      "epoch": 0.5934343434343434,
+      "step": 470
+    },
+    {
+      "loss": 1.1565,
+      "grad_norm": 0.639481246471405,
+      "learning_rate": 7.194010416666667e-06,
+      "entropy": 1.1862946093082427,
+      "num_tokens": 2903408.0,
+      "mean_token_accuracy": 0.7461866185069084,
+      "epoch": 0.6060606060606061,
+      "step": 480
+    },
+    {
+      "loss": 1.1251,
+      "grad_norm": 0.6332777142524719,
+      "learning_rate": 7.128906250000001e-06,
+      "entropy": 1.151743534207344,
+      "num_tokens": 2963401.0,
+      "mean_token_accuracy": 0.7535199671983719,
+      "epoch": 0.6186868686868687,
+      "step": 490
+    },
+    {
+      "loss": 1.1407,
+      "grad_norm": 0.5991836190223694,
+      "learning_rate": 7.063802083333335e-06,
+      "entropy": 1.1778477430343628,
+      "num_tokens": 3023530.0,
+      "mean_token_accuracy": 0.7491110354661942,
+      "epoch": 0.6313131313131313,
+      "step": 500
+    },
+    {
+      "loss": 1.1724,
+      "grad_norm": 0.6293458938598633,
+      "learning_rate": 6.998697916666667e-06,
+      "entropy": 1.2023035794496537,
+      "num_tokens": 3085225.0,
+      "mean_token_accuracy": 0.7405256554484367,
+      "epoch": 0.6439393939393939,
+      "step": 510
+    },
+    {
+      "loss": 1.1604,
+      "grad_norm": 0.6213802695274353,
+      "learning_rate": 6.93359375e-06,
+      "entropy": 1.1997334092855454,
+      "num_tokens": 3145749.0,
+      "mean_token_accuracy": 0.7435309410095214,
+      "epoch": 0.6565656565656566,
+      "step": 520
+    },
+    {
+      "loss": 1.1381,
+      "grad_norm": 0.6495156288146973,
+      "learning_rate": 6.868489583333334e-06,
+      "entropy": 1.1643184214830398,
+      "num_tokens": 3205761.0,
+      "mean_token_accuracy": 0.7496751576662064,
+      "epoch": 0.6691919191919192,
+      "step": 530
+    },
+    {
+      "loss": 1.1563,
+      "grad_norm": 0.6004510521888733,
+      "learning_rate": 6.803385416666667e-06,
+      "entropy": 1.1837532848119736,
+      "num_tokens": 3267024.0,
+      "mean_token_accuracy": 0.7447851061820984,
+      "epoch": 0.6818181818181818,
+      "step": 540
+    },
+    {
+      "loss": 1.1776,
+      "grad_norm": 0.607467532157898,
+      "learning_rate": 6.738281250000001e-06,
+      "entropy": 1.201677542924881,
+      "num_tokens": 3329070.0,
+      "mean_token_accuracy": 0.7406648993492126,
+      "epoch": 0.6944444444444444,
+      "step": 550
+    },
+    {
+      "loss": 1.1298,
+      "grad_norm": 0.6079947352409363,
+      "learning_rate": 6.6731770833333345e-06,
+      "entropy": 1.1659786373376846,
+      "num_tokens": 3389505.0,
+      "mean_token_accuracy": 0.7505511298775673,
+      "epoch": 0.7070707070707071,
+      "step": 560
+    },
+    {
+      "loss": 1.144,
+      "grad_norm": 0.6534572839736938,
+      "learning_rate": 6.6080729166666665e-06,
+      "entropy": 1.1714913487434386,
+      "num_tokens": 3449834.0,
+      "mean_token_accuracy": 0.7481625184416771,
+      "epoch": 0.7196969696969697,
+      "step": 570
+    },
+    {
+      "loss": 1.139,
+      "grad_norm": 0.5903164744377136,
+      "learning_rate": 6.54296875e-06,
+      "entropy": 1.1694963037967683,
+      "num_tokens": 3510701.0,
+      "mean_token_accuracy": 0.7476441130042076,
+      "epoch": 0.7323232323232324,
+      "step": 580
+    },
+    {
+      "loss": 1.1422,
+      "grad_norm": 0.6284182071685791,
+      "learning_rate": 6.477864583333334e-06,
+      "entropy": 1.1777653217315673,
+      "num_tokens": 3570992.0,
+      "mean_token_accuracy": 0.7490358456969262,
+      "epoch": 0.7449494949494949,
+      "step": 590
+    },
+    {
+      "loss": 1.1336,
+      "grad_norm": 0.6250146627426147,
+      "learning_rate": 6.412760416666667e-06,
+      "entropy": 1.1588268011808396,
+      "num_tokens": 3631087.0,
+      "mean_token_accuracy": 0.7504108369350433,
+      "epoch": 0.7575757575757576,
+      "step": 600
+    },
+    {
+      "loss": 1.1534,
+      "grad_norm": 0.6420578956604004,
+      "learning_rate": 6.3476562500000006e-06,
+      "entropy": 1.1890273630619048,
+      "num_tokens": 3692871.0,
+      "mean_token_accuracy": 0.7442372158169747,
+      "epoch": 0.7702020202020202,
+      "step": 610
+    },
+    {
+      "loss": 1.1477,
+      "grad_norm": 0.6156490445137024,
+      "learning_rate": 6.282552083333334e-06,
+      "entropy": 1.1817798465490341,
+      "num_tokens": 3753671.0,
+      "mean_token_accuracy": 0.7468263059854507,
+      "epoch": 0.7828282828282829,
+      "step": 620
+    },
+    {
+      "loss": 1.139,
+      "grad_norm": 0.6248748898506165,
+      "learning_rate": 6.217447916666667e-06,
+      "entropy": 1.1686612635850906,
+      "num_tokens": 3813110.0,
+      "mean_token_accuracy": 0.7478924334049225,
+      "epoch": 0.7954545454545454,
+      "step": 630
+    },
+    {
+      "loss": 1.1118,
+      "grad_norm": 0.6052266359329224,
+      "learning_rate": 6.152343750000001e-06,
+      "entropy": 1.1387953266501427,
+      "num_tokens": 3873477.0,
+      "mean_token_accuracy": 0.7538487210869789,
+      "epoch": 0.8080808080808081,
+      "step": 640
+    },
+    {
+      "loss": 1.1108,
+      "grad_norm": 0.6769536137580872,
+      "learning_rate": 6.087239583333335e-06,
+      "entropy": 1.1343814879655838,
+      "num_tokens": 3932873.0,
+      "mean_token_accuracy": 0.7548464313149452,
+      "epoch": 0.8207070707070707,
+      "step": 650
+    },
+    {
+      "loss": 1.134,
+      "grad_norm": 0.6545736789703369,
+      "learning_rate": 6.022135416666667e-06,
+      "entropy": 1.1686519652605056,
+      "num_tokens": 3992754.0,
+      "mean_token_accuracy": 0.7495195478200912,
+      "epoch": 0.8333333333333334,
+      "step": 660
+    },
+    {
+      "loss": 1.1213,
+      "grad_norm": 0.6192017793655396,
+      "learning_rate": 5.95703125e-06,
+      "entropy": 1.1596283346414566,
+      "num_tokens": 4053540.0,
+      "mean_token_accuracy": 0.7510297149419785,
+      "epoch": 0.8459595959595959,
+      "step": 670
+    },
+    {
+      "loss": 1.1181,
+      "grad_norm": 0.6520631909370422,
+      "learning_rate": 5.891927083333334e-06,
+      "entropy": 1.1484344542026519,
+      "num_tokens": 4113582.0,
+      "mean_token_accuracy": 0.751795919239521,
+      "epoch": 0.8585858585858586,
+      "step": 680
+    },
+    {
+      "loss": 1.1204,
+      "grad_norm": 0.6247655153274536,
+      "learning_rate": 5.826822916666667e-06,
+      "entropy": 1.1519750133156776,
+      "num_tokens": 4174983.0,
+      "mean_token_accuracy": 0.7506368085741997,
+      "epoch": 0.8712121212121212,
+      "step": 690
+    },
+    {
+      "loss": 1.1191,
+      "grad_norm": 0.620272159576416,
+      "learning_rate": 5.761718750000001e-06,
+      "entropy": 1.1486561581492425,
+      "num_tokens": 4234465.0,
+      "mean_token_accuracy": 0.7536391675472259,
+      "epoch": 0.8838383838383839,
+      "step": 700
+    },
+    {
+      "loss": 1.1224,
+      "grad_norm": 0.6308649182319641,
+      "learning_rate": 5.6966145833333344e-06,
+      "entropy": 1.1504923462867738,
+      "num_tokens": 4295955.0,
+      "mean_token_accuracy": 0.750646598637104,
+      "epoch": 0.8964646464646465,
+      "step": 710
+    },
+    {
+      "loss": 1.1238,
+      "grad_norm": 0.6629899740219116,
+      "learning_rate": 5.6315104166666665e-06,
+      "entropy": 1.1561898440122604,
+      "num_tokens": 4357171.0,
+      "mean_token_accuracy": 0.7507557719945908,
+      "epoch": 0.9090909090909091,
+      "step": 720
+    },
+    {
+      "loss": 1.1038,
+      "grad_norm": 0.5972346067428589,
+      "learning_rate": 5.56640625e-06,
+      "entropy": 1.1413449853658677,
+      "num_tokens": 4417636.0,
+      "mean_token_accuracy": 0.7554104939103127,
+      "epoch": 0.9217171717171717,
+      "step": 730
+    },
+    {
+      "loss": 1.1005,
+      "grad_norm": 0.6356479525566101,
+      "learning_rate": 5.501302083333334e-06,
+      "entropy": 1.1356119453907012,
+      "num_tokens": 4477294.0,
+      "mean_token_accuracy": 0.7565629109740257,
+      "epoch": 0.9343434343434344,
+      "step": 740
+    },
+    {
+      "loss": 1.1225,
+      "grad_norm": 0.6416464447975159,
+      "learning_rate": 5.436197916666667e-06,
+      "entropy": 1.1633600294589996,
+      "num_tokens": 4537503.0,
+      "mean_token_accuracy": 0.7515855401754379,
+      "epoch": 0.946969696969697,
+      "step": 750
+    },
+    {
+      "loss": 1.1184,
+      "grad_norm": 0.6126084327697754,
+      "learning_rate": 5.3710937500000005e-06,
+      "entropy": 1.1527763932943345,
+      "num_tokens": 4598778.0,
+      "mean_token_accuracy": 0.7526160582900048,
+      "epoch": 0.9595959595959596,
+      "step": 760
+    },
+    {
+      "loss": 1.1144,
+      "grad_norm": 0.6359922289848328,
+      "learning_rate": 5.305989583333334e-06,
+      "entropy": 1.1397768080234527,
+      "num_tokens": 4658978.0,
+      "mean_token_accuracy": 0.7548302739858628,
+      "epoch": 0.9722222222222222,
+      "step": 770
+    },
+    {
+      "loss": 1.1213,
+      "grad_norm": 0.6260409951210022,
+      "learning_rate": 5.240885416666667e-06,
+      "entropy": 1.1569962561130525,
+      "num_tokens": 4720500.0,
+      "mean_token_accuracy": 0.7512885302305221,
+      "epoch": 0.9848484848484849,
+      "step": 780
+    },
+    {
+      "loss": 1.1227,
+      "grad_norm": 0.6293452978134155,
+      "learning_rate": 5.17578125e-06,
+      "entropy": 1.1509152203798294,
+      "num_tokens": 4781612.0,
+      "mean_token_accuracy": 0.7519564241170883,
+      "epoch": 0.9974747474747475,
+      "step": 790
+    },
+    {
+      "loss": 1.1034,
+      "grad_norm": 0.664761483669281,
+      "learning_rate": 5.110677083333334e-06,
+      "entropy": 1.1399024561047555,
+      "num_tokens": 4841359.0,
+      "mean_token_accuracy": 0.7526706486940384,
+      "epoch": 1.0101010101010102,
+      "step": 800
+    },
+    {
+      "loss": 1.0857,
+      "grad_norm": 0.5934865474700928,
+      "learning_rate": 5.045572916666667e-06,
+      "entropy": 1.120214229822159,
+      "num_tokens": 4901016.0,
+      "mean_token_accuracy": 0.7593594208359719,
+      "epoch": 1.0227272727272727,
+      "step": 810
+    },
+    {
+      "loss": 1.1165,
+      "grad_norm": 0.6040735840797424,
+      "learning_rate": 4.98046875e-06,
+      "entropy": 1.1430341199040412,
+      "num_tokens": 4961646.0,
+      "mean_token_accuracy": 0.7525988414883613,
+      "epoch": 1.0353535353535352,
+      "step": 820
+    },
+    {
+      "loss": 1.0851,
+      "grad_norm": 0.6277610063552856,
+      "learning_rate": 4.915364583333333e-06,
+      "entropy": 1.1245349109172822,
+      "num_tokens": 5022365.0,
+      "mean_token_accuracy": 0.7577921718358993,
+      "epoch": 1.047979797979798,
+      "step": 830
+    },
+    {
+      "loss": 1.0813,
+      "grad_norm": 0.6260582804679871,
+      "learning_rate": 4.850260416666667e-06,
+      "entropy": 1.1204675793647767,
+      "num_tokens": 5081972.0,
+      "mean_token_accuracy": 0.7580071151256561,
+      "epoch": 1.0606060606060606,
+      "step": 840
+    },
+    {
+      "loss": 1.0922,
+      "grad_norm": 0.6023226976394653,
+      "learning_rate": 4.785156250000001e-06,
+      "entropy": 1.1222782507538795,
+      "num_tokens": 5142184.0,
+      "mean_token_accuracy": 0.7566258609294891,
+      "epoch": 1.0732323232323233,
+      "step": 850
+    },
+    {
+      "loss": 1.0994,
+      "grad_norm": 0.6206791996955872,
+      "learning_rate": 4.7200520833333336e-06,
+      "entropy": 1.1227335944771766,
+      "num_tokens": 5203020.0,
+      "mean_token_accuracy": 0.7540625646710396,
+      "epoch": 1.0858585858585859,
+      "step": 860
+    },
+    {
+      "loss": 1.0958,
+      "grad_norm": 0.6301055550575256,
+      "learning_rate": 4.654947916666667e-06,
+      "entropy": 1.1352888554334641,
+      "num_tokens": 5263291.0,
+      "mean_token_accuracy": 0.7562039017677307,
+      "epoch": 1.0984848484848484,
+      "step": 870
+    },
+    {
+      "loss": 1.0793,
+      "grad_norm": 0.6210020780563354,
+      "learning_rate": 4.58984375e-06,
+      "entropy": 1.1120088309049607,
+      "num_tokens": 5323972.0,
+      "mean_token_accuracy": 0.7590557768940925,
+      "epoch": 1.1111111111111112,
+      "step": 880
+    },
+    {
+      "loss": 1.0717,
+      "grad_norm": 0.6332690715789795,
+      "learning_rate": 4.524739583333334e-06,
+      "entropy": 1.0996666207909584,
+      "num_tokens": 5383911.0,
+      "mean_token_accuracy": 0.7615471586585045,
+      "epoch": 1.1237373737373737,
+      "step": 890
+    },
+    {
+      "loss": 1.1027,
+      "grad_norm": 0.6505516767501831,
+      "learning_rate": 4.459635416666668e-06,
+      "entropy": 1.127107810974121,
+      "num_tokens": 5445417.0,
+      "mean_token_accuracy": 0.7562421515583992,
+      "epoch": 1.1363636363636362,
+      "step": 900
+    },
+    {
+      "loss": 1.0879,
+      "grad_norm": 0.6406158804893494,
+      "learning_rate": 4.3945312500000005e-06,
+      "entropy": 1.129740473628044,
+      "num_tokens": 5505455.0,
+      "mean_token_accuracy": 0.7587148532271385,
+      "epoch": 1.148989898989899,
+      "step": 910
+    },
+    {
+      "loss": 1.0752,
+      "grad_norm": 0.6297397613525391,
+      "learning_rate": 4.329427083333333e-06,
+      "entropy": 1.1167259424924851,
+      "num_tokens": 5565311.0,
+      "mean_token_accuracy": 0.7604142814874649,
+      "epoch": 1.1616161616161615,
+      "step": 920
+    },
+    {
+      "loss": 1.0686,
+      "grad_norm": 0.6490073204040527,
+      "learning_rate": 4.264322916666667e-06,
+      "entropy": 1.1037891641259194,
+      "num_tokens": 5625358.0,
+      "mean_token_accuracy": 0.7610052570700645,
+      "epoch": 1.1742424242424243,
+      "step": 930
+    },
+    {
+      "loss": 1.0868,
+      "grad_norm": 0.6366387009620667,
+      "learning_rate": 4.19921875e-06,
+      "entropy": 1.1061881184577942,
+      "num_tokens": 5686421.0,
+      "mean_token_accuracy": 0.7574937298893929,
+      "epoch": 1.1868686868686869,
+      "step": 940
+    },
+    {
+      "loss": 1.0694,
+      "grad_norm": 0.6556055545806885,
+      "learning_rate": 4.134114583333334e-06,
+      "entropy": 1.1124324068427085,
+      "num_tokens": 5745891.0,
+      "mean_token_accuracy": 0.7602224007248879,
+      "epoch": 1.1994949494949494,
+      "step": 950
+    },
+    {
+      "loss": 1.081,
+      "grad_norm": 0.6404849886894226,
+      "learning_rate": 4.0690104166666675e-06,
+      "entropy": 1.1175200879573821,
+      "num_tokens": 5806078.0,
+      "mean_token_accuracy": 0.7568994402885437,
+      "epoch": 1.2121212121212122,
+      "step": 960
+    },
+    {
+      "loss": 1.0791,
+      "grad_norm": 0.6227584481239319,
+      "learning_rate": 4.00390625e-06,
+      "entropy": 1.1186978340148925,
+      "num_tokens": 5866369.0,
+      "mean_token_accuracy": 0.759756401181221,
+      "epoch": 1.2247474747474747,
+      "step": 970
+    },
+    {
+      "loss": 1.0937,
+      "grad_norm": 0.6616361141204834,
+      "learning_rate": 3.938802083333333e-06,
+      "entropy": 1.122128139436245,
+      "num_tokens": 5926217.0,
+      "mean_token_accuracy": 0.7582718566060066,
+      "epoch": 1.2373737373737375,
+      "step": 980
+    },
+    {
+      "loss": 1.0978,
+      "grad_norm": 0.6384168267250061,
+      "learning_rate": 3.873697916666667e-06,
+      "entropy": 1.122861033678055,
+      "num_tokens": 5987666.0,
+      "mean_token_accuracy": 0.7549964562058449,
+      "epoch": 1.25,
+      "step": 990
+    },
+    {
+      "loss": 1.0952,
+      "grad_norm": 0.6038117408752441,
+      "learning_rate": 3.8085937500000002e-06,
+      "entropy": 1.1277505576610565,
+      "num_tokens": 6048708.0,
+      "mean_token_accuracy": 0.755272176861763,
+      "epoch": 1.2626262626262625,
+      "step": 1000
+    },
+    {
+      "loss": 1.078,
+      "grad_norm": 0.6418159604072571,
+      "learning_rate": 3.7434895833333336e-06,
+      "entropy": 1.1120157346129418,
+      "num_tokens": 6109652.0,
+      "mean_token_accuracy": 0.7594122514128685,
+      "epoch": 1.2752525252525253,
+      "step": 1010
+    },
+    {
+      "loss": 1.0688,
+      "grad_norm": 0.6218425035476685,
+      "learning_rate": 3.6783854166666673e-06,
+      "entropy": 1.101425115764141,
+      "num_tokens": 6169125.0,
+      "mean_token_accuracy": 0.7604865297675133,
+      "epoch": 1.2878787878787878,
+      "step": 1020
+    },
+    {
+      "loss": 1.0581,
+      "grad_norm": 0.6429149508476257,
+      "learning_rate": 3.61328125e-06,
+      "entropy": 1.1007713869214057,
+      "num_tokens": 6230303.0,
+      "mean_token_accuracy": 0.7621071562170982,
+      "epoch": 1.3005050505050506,
+      "step": 1030
+    },
+    {
+      "loss": 1.0715,
+      "grad_norm": 0.6489748358726501,
+      "learning_rate": 3.5481770833333335e-06,
+      "entropy": 1.1094096556305886,
+      "num_tokens": 6291396.0,
+      "mean_token_accuracy": 0.7599423810839653,
+      "epoch": 1.3131313131313131,
+      "step": 1040
+    },
+    {
+      "loss": 1.0584,
+      "grad_norm": 0.6485461592674255,
+      "learning_rate": 3.483072916666667e-06,
+      "entropy": 1.0827289715409278,
+      "num_tokens": 6351579.0,
+      "mean_token_accuracy": 0.7630694910883904,
+      "epoch": 1.3257575757575757,
+      "step": 1050
+    },
+    {
+      "loss": 1.0764,
+      "grad_norm": 0.6261104941368103,
+      "learning_rate": 3.41796875e-06,
+      "entropy": 1.114325873553753,
+      "num_tokens": 6411662.0,
+      "mean_token_accuracy": 0.7585488513112069,
+      "epoch": 1.3383838383838385,
+      "step": 1060
+    },
+    {
+      "loss": 1.0902,
+      "grad_norm": 0.6522034406661987,
+      "learning_rate": 3.3528645833333334e-06,
+      "entropy": 1.1271554425358772,
+      "num_tokens": 6473505.0,
+      "mean_token_accuracy": 0.7562535598874092,
+      "epoch": 1.351010101010101,
+      "step": 1070
+    },
+    {
+      "loss": 1.065,
+      "grad_norm": 0.6176674962043762,
+      "learning_rate": 3.287760416666667e-06,
+      "entropy": 1.1013643085956573,
+      "num_tokens": 6533580.0,
+      "mean_token_accuracy": 0.763075165450573,
+      "epoch": 1.3636363636363638,
+      "step": 1080
+    },
+    {
+      "loss": 1.0596,
+      "grad_norm": 0.6540253758430481,
+      "learning_rate": 3.2226562500000004e-06,
+      "entropy": 1.098090337216854,
+      "num_tokens": 6593481.0,
+      "mean_token_accuracy": 0.7616770043969154,
+      "epoch": 1.3762626262626263,
+      "step": 1090
+    },
+    {
+      "loss": 1.0861,
+      "grad_norm": 0.6754550933837891,
+      "learning_rate": 3.1575520833333333e-06,
+      "entropy": 1.1176372200250626,
+      "num_tokens": 6653967.0,
+      "mean_token_accuracy": 0.7573029339313507,
+      "epoch": 1.3888888888888888,
+      "step": 1100
+    },
+    {
+      "loss": 1.0573,
+      "grad_norm": 0.6022531986236572,
+      "learning_rate": 3.092447916666667e-06,
+      "entropy": 1.1040414482355119,
+      "num_tokens": 6714685.0,
+      "mean_token_accuracy": 0.7612267225980759,
+      "epoch": 1.4015151515151514,
+      "step": 1110
+    },
+    {
+      "loss": 1.0637,
+      "grad_norm": 0.6621010303497314,
+      "learning_rate": 3.0273437500000003e-06,
+      "entropy": 1.0926991075277328,
+      "num_tokens": 6774659.0,
+      "mean_token_accuracy": 0.7612977519631385,
+      "epoch": 1.4141414141414141,
+      "step": 1120
+    },
+    {
+      "loss": 1.0701,
+      "grad_norm": 0.62503981590271,
+      "learning_rate": 2.962239583333333e-06,
+      "entropy": 1.1095534324645997,
+      "num_tokens": 6834579.0,
+      "mean_token_accuracy": 0.7618053883314133,
+      "epoch": 1.4267676767676767,
+      "step": 1130
+    },
+    {
+      "loss": 1.0747,
+      "grad_norm": 0.6527109742164612,
+      "learning_rate": 2.897135416666667e-06,
+      "entropy": 1.117809349298477,
+      "num_tokens": 6894074.0,
+      "mean_token_accuracy": 0.759482853114605,
+      "epoch": 1.4393939393939394,
+      "step": 1140
+    },
+    {
+      "loss": 1.0607,
+      "grad_norm": 0.6720954775810242,
+      "learning_rate": 2.8320312500000002e-06,
+      "entropy": 1.1005077749490737,
+      "num_tokens": 6953870.0,
+      "mean_token_accuracy": 0.7621246844530105,
+      "epoch": 1.452020202020202,
+      "step": 1150
+    },
+    {
+      "loss": 1.0884,
+      "grad_norm": 0.658524215221405,
+      "learning_rate": 2.7669270833333335e-06,
+      "entropy": 1.1236482918262483,
+      "num_tokens": 7014553.0,
+      "mean_token_accuracy": 0.7560836613178253,
+      "epoch": 1.4646464646464645,
+      "step": 1160
+    },
+    {
+      "loss": 1.0659,
+      "grad_norm": 0.6261802911758423,
+      "learning_rate": 2.7018229166666673e-06,
+      "entropy": 1.1116504594683647,
+      "num_tokens": 7076291.0,
+      "mean_token_accuracy": 0.7597616642713547,
+      "epoch": 1.4772727272727273,
+      "step": 1170
+    },
+    {
+      "loss": 1.0524,
+      "grad_norm": 0.6310375332832336,
+      "learning_rate": 2.63671875e-06,
+      "entropy": 1.073892480134964,
+      "num_tokens": 7137305.0,
+      "mean_token_accuracy": 0.7628733053803444,
+      "epoch": 1.4898989898989898,
+      "step": 1180
+    },
+    {
+      "loss": 1.0679,
+      "grad_norm": 0.638482391834259,
+      "learning_rate": 2.5716145833333334e-06,
+      "entropy": 1.0975843235850333,
+      "num_tokens": 7198239.0,
+      "mean_token_accuracy": 0.7603248566389084,
+      "epoch": 1.5025252525252526,
+      "step": 1190
+    },
+    {
+      "loss": 1.0666,
+      "grad_norm": 0.640065610408783,
+      "learning_rate": 2.506510416666667e-06,
+      "entropy": 1.0986508697271347,
+      "num_tokens": 7257847.0,
+      "mean_token_accuracy": 0.7622530281543731,
+      "epoch": 1.5151515151515151,
+      "step": 1200
+    },
+    {
+      "loss": 1.0587,
+      "grad_norm": 0.6437165141105652,
+      "learning_rate": 2.44140625e-06,
+      "entropy": 1.0971406906843186,
+      "num_tokens": 7317615.0,
+      "mean_token_accuracy": 0.7623877748847008,
+      "epoch": 1.5277777777777777,
+      "step": 1210
+    },
+    {
+      "loss": 1.0569,
+      "grad_norm": 0.6590547561645508,
+      "learning_rate": 2.3763020833333338e-06,
+      "entropy": 1.1032136514782906,
+      "num_tokens": 7377946.0,
+      "mean_token_accuracy": 0.7616146191954613,
+      "epoch": 1.5404040404040404,
+      "step": 1220
+    },
+    {
+      "loss": 1.0616,
+      "grad_norm": 0.6317723989486694,
+      "learning_rate": 2.3111979166666667e-06,
+      "entropy": 1.0922824308276176,
+      "num_tokens": 7438666.0,
+      "mean_token_accuracy": 0.7611089378595353,
+      "epoch": 1.553030303030303,
+      "step": 1230
+    },
+    {
+      "loss": 1.0725,
+      "grad_norm": 0.66637122631073,
+      "learning_rate": 2.2460937500000004e-06,
+      "entropy": 1.1092566132545472,
+      "num_tokens": 7499747.0,
+      "mean_token_accuracy": 0.7594954133033752,
+      "epoch": 1.5656565656565657,
+      "step": 1240
+    },
+    {
+      "loss": 1.074,
+      "grad_norm": 0.6520881652832031,
+      "learning_rate": 2.1809895833333337e-06,
+      "entropy": 1.111878876388073,
+      "num_tokens": 7561004.0,
+      "mean_token_accuracy": 0.7558068126440048,
+      "epoch": 1.5782828282828283,
+      "step": 1250
+    },
+    {
+      "loss": 1.0844,
+      "grad_norm": 0.6495437622070312,
+      "learning_rate": 2.1158854166666666e-06,
+      "entropy": 1.1105972841382026,
+      "num_tokens": 7622115.0,
+      "mean_token_accuracy": 0.7563070356845856,
+      "epoch": 1.5909090909090908,
+      "step": 1260
+    },
+    {
+      "loss": 1.0649,
+      "grad_norm": 0.6420316696166992,
+      "learning_rate": 2.0507812500000003e-06,
+      "entropy": 1.0932338371872903,
+      "num_tokens": 7682210.0,
+      "mean_token_accuracy": 0.7622412323951722,
+      "epoch": 1.6035353535353534,
+      "step": 1270
+    },
+    {
+      "loss": 1.0427,
+      "grad_norm": 0.6192623972892761,
+      "learning_rate": 1.9856770833333336e-06,
+      "entropy": 1.0919055327773095,
+      "num_tokens": 7742745.0,
+      "mean_token_accuracy": 0.7657591253519058,
+      "epoch": 1.6161616161616161,
+      "step": 1280
+    },
+    {
+      "loss": 1.0701,
+      "grad_norm": 0.6355161666870117,
+      "learning_rate": 1.920572916666667e-06,
+      "entropy": 1.1026942864060403,
+      "num_tokens": 7802902.0,
+      "mean_token_accuracy": 0.7609635755419731,
+      "epoch": 1.628787878787879,
+      "step": 1290
+    },
+    {
+      "loss": 1.0789,
+      "grad_norm": 0.6254522800445557,
+      "learning_rate": 1.8554687500000002e-06,
+      "entropy": 1.1140723824501038,
+      "num_tokens": 7863576.0,
+      "mean_token_accuracy": 0.7592580512166023,
+      "epoch": 1.6414141414141414,
+      "step": 1300
+    },
+    {
+      "loss": 1.0714,
+      "grad_norm": 0.633172333240509,
+      "learning_rate": 1.7903645833333335e-06,
+      "entropy": 1.108394268155098,
+      "num_tokens": 7925518.0,
+      "mean_token_accuracy": 0.7598690986633301,
+      "epoch": 1.654040404040404,
+      "step": 1310
+    },
+    {
+      "loss": 1.0701,
+      "grad_norm": 0.6279735565185547,
+      "learning_rate": 1.7252604166666668e-06,
+      "entropy": 1.1097407966852189,
+      "num_tokens": 7987388.0,
+      "mean_token_accuracy": 0.7611567705869675,
+      "epoch": 1.6666666666666665,
+      "step": 1320
+    },
+    {
+      "loss": 1.0786,
+      "grad_norm": 0.6425778269767761,
+      "learning_rate": 1.6601562500000001e-06,
+      "entropy": 1.1071213275194167,
+      "num_tokens": 8048853.0,
+      "mean_token_accuracy": 0.7576610520482063,
+      "epoch": 1.6792929292929293,
+      "step": 1330
+    },
+    {
+      "loss": 1.0604,
+      "grad_norm": 0.666192889213562,
+      "learning_rate": 1.5950520833333336e-06,
+      "entropy": 1.0931925728917122,
+      "num_tokens": 8108967.0,
+      "mean_token_accuracy": 0.7616572439670563,
+      "epoch": 1.691919191919192,
+      "step": 1340
+    },
+    {
+      "loss": 1.0769,
+      "grad_norm": 0.6348255276679993,
+      "learning_rate": 1.5299479166666667e-06,
+      "entropy": 1.0973364993929864,
+      "num_tokens": 8169700.0,
+      "mean_token_accuracy": 0.7596119627356529,
+      "epoch": 1.7045454545454546,
+      "step": 1350
+    },
+    {
+      "loss": 1.0731,
+      "grad_norm": 0.6510699391365051,
+      "learning_rate": 1.46484375e-06,
+      "entropy": 1.1137071400880814,
+      "num_tokens": 8229676.0,
+      "mean_token_accuracy": 0.7593250289559365,
+      "epoch": 1.7171717171717171,
+      "step": 1360
+    },
+    {
+      "loss": 1.069,
+      "grad_norm": 0.6622318625450134,
+      "learning_rate": 1.3997395833333335e-06,
+      "entropy": 1.1052428260445595,
+      "num_tokens": 8289396.0,
+      "mean_token_accuracy": 0.7627501472830772,
+      "epoch": 1.7297979797979797,
+      "step": 1370
+    },
+    {
+      "loss": 1.0506,
+      "grad_norm": 0.6430277824401855,
+      "learning_rate": 1.3346354166666666e-06,
+      "entropy": 1.0989198789000512,
+      "num_tokens": 8351152.0,
+      "mean_token_accuracy": 0.7634323209524154,
+      "epoch": 1.7424242424242424,
+      "step": 1380
+    },
+    {
+      "loss": 1.0505,
+      "grad_norm": 0.639707088470459,
+      "learning_rate": 1.2695312500000002e-06,
+      "entropy": 1.0913211867213248,
+      "num_tokens": 8411357.0,
+      "mean_token_accuracy": 0.763850274682045,
+      "epoch": 1.7550505050505052,
+      "step": 1390
+    },
+    {
+      "loss": 1.0756,
+      "grad_norm": 0.680479109287262,
+      "learning_rate": 1.2044270833333335e-06,
+      "entropy": 1.1044820442795753,
+      "num_tokens": 8471975.0,
+      "mean_token_accuracy": 0.7598197475075722,
+      "epoch": 1.7676767676767677,
+      "step": 1400
+    },
+    {
+      "loss": 1.048,
+      "grad_norm": 0.651622474193573,
+      "learning_rate": 1.1393229166666668e-06,
+      "entropy": 1.0821994885802269,
+      "num_tokens": 8532480.0,
+      "mean_token_accuracy": 0.7638402819633484,
+      "epoch": 1.7803030303030303,
+      "step": 1410
+    },
+    {
+      "loss": 1.0545,
+      "grad_norm": 0.6294305920600891,
+      "learning_rate": 1.07421875e-06,
+      "entropy": 1.0933044001460075,
+      "num_tokens": 8593538.0,
+      "mean_token_accuracy": 0.7633207753300667,
+      "epoch": 1.7929292929292928,
+      "step": 1420
+    },
+    {
+      "loss": 1.065,
+      "grad_norm": 0.6396600008010864,
+      "learning_rate": 1.0091145833333334e-06,
+      "entropy": 1.1002878457307816,
+      "num_tokens": 8654446.0,
+      "mean_token_accuracy": 0.7610213488340378,
+      "epoch": 1.8055555555555556,
+      "step": 1430
+    },
+    {
+      "loss": 1.079,
+      "grad_norm": 0.6585692167282104,
+      "learning_rate": 9.440104166666668e-07,
+      "entropy": 1.1098253890872,
+      "num_tokens": 8715305.0,
+      "mean_token_accuracy": 0.7590781077742577,
+      "epoch": 1.8181818181818183,
+      "step": 1440
+    },
+    {
+      "loss": 1.0555,
+      "grad_norm": 0.6637106537818909,
+      "learning_rate": 8.789062500000001e-07,
+      "entropy": 1.0894212126731873,
+      "num_tokens": 8775351.0,
+      "mean_token_accuracy": 0.7621971383690834,
+      "epoch": 1.8308080808080809,
+      "step": 1450
+    },
+    {
+      "loss": 1.0464,
+      "grad_norm": 0.6491685509681702,
+      "learning_rate": 8.138020833333334e-07,
+      "entropy": 1.0821923539042473,
+      "num_tokens": 8836131.0,
+      "mean_token_accuracy": 0.7639922067523003,
+      "epoch": 1.8434343434343434,
+      "step": 1460
+    },
+    {
+      "loss": 1.0682,
+      "grad_norm": 0.6781795024871826,
+      "learning_rate": 7.486979166666668e-07,
+      "entropy": 1.1058385655283929,
+      "num_tokens": 8896334.0,
+      "mean_token_accuracy": 0.7595004603266716,
+      "epoch": 1.856060606060606,
+      "step": 1470
+    },
+    {
+      "loss": 1.0625,
+      "grad_norm": 0.652746319770813,
+      "learning_rate": 6.835937500000001e-07,
+      "entropy": 1.0934822604060173,
+      "num_tokens": 8956948.0,
+      "mean_token_accuracy": 0.7614723727107048,
+      "epoch": 1.8686868686868687,
+      "step": 1480
+    },
+    {
+      "loss": 1.0677,
+      "grad_norm": 0.6350075006484985,
+      "learning_rate": 6.184895833333334e-07,
+      "entropy": 1.0955146595835685,
+      "num_tokens": 9017330.0,
+      "mean_token_accuracy": 0.75977371186018,
+      "epoch": 1.8813131313131313,
+      "step": 1490
+    },
+    {
+      "loss": 1.0778,
+      "grad_norm": 0.6651970744132996,
+      "learning_rate": 5.533854166666667e-07,
+      "entropy": 1.1074895232915878,
+      "num_tokens": 9077561.0,
+      "mean_token_accuracy": 0.7592803448438644,
+      "epoch": 1.893939393939394,
+      "step": 1500
+    },
+    {
+      "loss": 1.0517,
+      "grad_norm": 0.6216638684272766,
+      "learning_rate": 4.8828125e-07,
+      "entropy": 1.088778705894947,
+      "num_tokens": 9137424.0,
+      "mean_token_accuracy": 0.7628024965524673,
+      "epoch": 1.9065656565656566,
+      "step": 1510
+    },
+    {
+      "loss": 1.0657,
+      "grad_norm": 0.6801443099975586,
+      "learning_rate": 4.2317708333333337e-07,
+      "entropy": 1.0963940545916557,
+      "num_tokens": 9198465.0,
+      "mean_token_accuracy": 0.7611119478940964,
+      "epoch": 1.9191919191919191,
+      "step": 1520
+    },
+    {
+      "loss": 1.0657,
+      "grad_norm": 0.6482690572738647,
+      "learning_rate": 3.5807291666666667e-07,
+      "entropy": 1.107513566315174,
+      "num_tokens": 9258486.0,
+      "mean_token_accuracy": 0.7620691254734993,
+      "epoch": 1.9318181818181817,
+      "step": 1530
+    },
+    {
+      "loss": 1.0758,
+      "grad_norm": 0.6314805746078491,
+      "learning_rate": 2.9296875000000003e-07,
+      "entropy": 1.1056584566831589,
+      "num_tokens": 9319619.0,
+      "mean_token_accuracy": 0.7584919854998589,
+      "epoch": 1.9444444444444444,
+      "step": 1540
+    },
+    {
+      "loss": 1.0745,
+      "grad_norm": 0.6383644938468933,
+      "learning_rate": 2.2786458333333333e-07,
+      "entropy": 1.1117310538887977,
+      "num_tokens": 9380350.0,
+      "mean_token_accuracy": 0.7591810420155525,
+      "epoch": 1.9570707070707072,
+      "step": 1550
+    },
+    {
+      "loss": 1.0557,
+      "grad_norm": 0.6331989169120789,
+      "learning_rate": 1.627604166666667e-07,
+      "entropy": 1.0970366701483727,
+      "num_tokens": 9441627.0,
+      "mean_token_accuracy": 0.7604442983865738,
+      "epoch": 1.9696969696969697,
+      "step": 1560
+    },
+    {
+      "loss": 1.0546,
+      "grad_norm": 0.6618102192878723,
+      "learning_rate": 9.765625e-08,
+      "entropy": 1.0917240902781487,
+      "num_tokens": 9501728.0,
+      "mean_token_accuracy": 0.7638352930545806,
+      "epoch": 1.9823232323232323,
+      "step": 1570
+    },
+    {
+      "loss": 1.0615,
+      "grad_norm": 0.6413847804069519,
+      "learning_rate": 3.2552083333333335e-08,
+      "entropy": 1.096452857553959,
+      "num_tokens": 9562238.0,
+      "mean_token_accuracy": 0.7619210347533226,
+      "epoch": 1.9949494949494948,
+      "step": 1580
+    },
+    {
+      "train_runtime": 50353.3546,
+      "train_samples_per_second": 1.006,
+      "train_steps_per_second": 0.031,
+      "total_flos": 5.4295385739381965e+17,
+      "train_loss": 1.1876104943680041,
+      "entropy": 1.1173381134867668,
+      "num_tokens": 9585534.0,
+      "mean_token_accuracy": 0.7577789686620235,
+      "epoch": 2.0,
+      "step": 1584
+    }
+  ],
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "global_step": 1584,
+  "num_train_epochs": 2
+}