Upload folder using huggingface_hub

README.md

@@ -0,0 +1,207 @@
+---
+base_model: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.16.0

adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b5e253cf1ec51c4624e2f5cc07545a96ccf4aa952439652f31d6d4cbfb33d5e
+size 11083960

checkpoint-800/README.md

@@ -0,0 +1,207 @@
+---
+base_model: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.16.0

checkpoint-800/adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-800/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b5e253cf1ec51c4624e2f5cc07545a96ccf4aa952439652f31d6d4cbfb33d5e
+size 11083960

checkpoint-800/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ebe00cdfff0797a7038c43ffa0ffa88cd4638feae628d1f71d79b10ef2d7dfb
+size 22270027

checkpoint-800/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25450de4bf098d33160cadc1eca1855d20fbf24a47c75d1abb2faa2e936baa39
+size 14645

checkpoint-800/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e748c5aaf9fec0191428bfcf3e6e7726b6330b26ea79785616afaf6d08ed577d
+size 1465

checkpoint-800/trainer_state.json

@@ -0,0 +1,1179 @@
+{
+  "best_global_step": 800,
+  "best_metric": 0.5232270359992981,
+  "best_model_checkpoint": "./exaone-lora-results-system-custom/checkpoint-800",
+  "epoch": 2.0,
+  "eval_steps": 500,
+  "global_step": 800,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0125,
+      "grad_norm": 1.4204977750778198,
+      "learning_rate": 8e-05,
+      "loss": 3.1159,
+      "step": 5
+    },
+    {
+      "epoch": 0.025,
+      "grad_norm": 1.3447633981704712,
+      "learning_rate": 0.00018,
+      "loss": 2.9296,
+      "step": 10
+    },
+    {
+      "epoch": 0.0375,
+      "grad_norm": 1.305059790611267,
+      "learning_rate": 0.00019959798994974876,
+      "loss": 2.5285,
+      "step": 15
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 1.3643401861190796,
+      "learning_rate": 0.0001990954773869347,
+      "loss": 2.1402,
+      "step": 20
+    },
+    {
+      "epoch": 0.0625,
+      "grad_norm": 1.3797638416290283,
+      "learning_rate": 0.00019859296482412062,
+      "loss": 1.773,
+      "step": 25
+    },
+    {
+      "epoch": 0.075,
+      "grad_norm": 1.8258603811264038,
+      "learning_rate": 0.00019809045226130653,
+      "loss": 1.37,
+      "step": 30
+    },
+    {
+      "epoch": 0.0875,
+      "grad_norm": 1.6069886684417725,
+      "learning_rate": 0.00019758793969849249,
+      "loss": 1.0568,
+      "step": 35
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 1.1830629110336304,
+      "learning_rate": 0.0001970854271356784,
+      "loss": 0.76,
+      "step": 40
+    },
+    {
+      "epoch": 0.1125,
+      "grad_norm": 0.8522692322731018,
+      "learning_rate": 0.00019658291457286432,
+      "loss": 0.6618,
+      "step": 45
+    },
+    {
+      "epoch": 0.125,
+      "grad_norm": 0.6791490316390991,
+      "learning_rate": 0.00019608040201005026,
+      "loss": 0.6323,
+      "step": 50
+    },
+    {
+      "epoch": 0.1375,
+      "grad_norm": 0.6827747821807861,
+      "learning_rate": 0.0001955778894472362,
+      "loss": 0.6331,
+      "step": 55
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.6210057139396667,
+      "learning_rate": 0.00019507537688442212,
+      "loss": 0.551,
+      "step": 60
+    },
+    {
+      "epoch": 0.1625,
+      "grad_norm": 0.6427358388900757,
+      "learning_rate": 0.00019457286432160805,
+      "loss": 0.5756,
+      "step": 65
+    },
+    {
+      "epoch": 0.175,
+      "grad_norm": 0.8321182131767273,
+      "learning_rate": 0.00019407035175879398,
+      "loss": 0.5466,
+      "step": 70
+    },
+    {
+      "epoch": 0.1875,
+      "grad_norm": 0.689738929271698,
+      "learning_rate": 0.00019356783919597992,
+      "loss": 0.5025,
+      "step": 75
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.6590372323989868,
+      "learning_rate": 0.00019306532663316582,
+      "loss": 0.5308,
+      "step": 80
+    },
+    {
+      "epoch": 0.2125,
+      "grad_norm": 0.7442909479141235,
+      "learning_rate": 0.00019256281407035178,
+      "loss": 0.4907,
+      "step": 85
+    },
+    {
+      "epoch": 0.225,
+      "grad_norm": 0.6165861487388611,
+      "learning_rate": 0.0001920603015075377,
+      "loss": 0.519,
+      "step": 90
+    },
+    {
+      "epoch": 0.2375,
+      "grad_norm": 0.4800119996070862,
+      "learning_rate": 0.00019155778894472362,
+      "loss": 0.4944,
+      "step": 95
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.5565855503082275,
+      "learning_rate": 0.00019105527638190955,
+      "loss": 0.4891,
+      "step": 100
+    },
+    {
+      "epoch": 0.2625,
+      "grad_norm": 0.5599360466003418,
+      "learning_rate": 0.00019055276381909548,
+      "loss": 0.4887,
+      "step": 105
+    },
+    {
+      "epoch": 0.275,
+      "grad_norm": 0.5745482444763184,
+      "learning_rate": 0.00019005025125628142,
+      "loss": 0.4915,
+      "step": 110
+    },
+    {
+      "epoch": 0.2875,
+      "grad_norm": 0.5511234998703003,
+      "learning_rate": 0.00018954773869346732,
+      "loss": 0.5147,
+      "step": 115
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.579424262046814,
+      "learning_rate": 0.00018904522613065328,
+      "loss": 0.4726,
+      "step": 120
+    },
+    {
+      "epoch": 0.3125,
+      "grad_norm": 0.604240357875824,
+      "learning_rate": 0.00018854271356783921,
+      "loss": 0.5083,
+      "step": 125
+    },
+    {
+      "epoch": 0.325,
+      "grad_norm": 0.631528913974762,
+      "learning_rate": 0.00018804020100502512,
+      "loss": 0.4827,
+      "step": 130
+    },
+    {
+      "epoch": 0.3375,
+      "grad_norm": 0.5906082391738892,
+      "learning_rate": 0.00018753768844221108,
+      "loss": 0.4704,
+      "step": 135
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.5601592659950256,
+      "learning_rate": 0.00018703517587939698,
+      "loss": 0.4645,
+      "step": 140
+    },
+    {
+      "epoch": 0.3625,
+      "grad_norm": 0.5383768081665039,
+      "learning_rate": 0.00018653266331658292,
+      "loss": 0.4808,
+      "step": 145
+    },
+    {
+      "epoch": 0.375,
+      "grad_norm": 0.6543328166007996,
+      "learning_rate": 0.00018603015075376885,
+      "loss": 0.4504,
+      "step": 150
+    },
+    {
+      "epoch": 0.3875,
+      "grad_norm": 0.6863328814506531,
+      "learning_rate": 0.00018552763819095478,
+      "loss": 0.4411,
+      "step": 155
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.6257234811782837,
+      "learning_rate": 0.00018502512562814071,
+      "loss": 0.4852,
+      "step": 160
+    },
+    {
+      "epoch": 0.4125,
+      "grad_norm": 0.5948039293289185,
+      "learning_rate": 0.00018452261306532662,
+      "loss": 0.4746,
+      "step": 165
+    },
+    {
+      "epoch": 0.425,
+      "grad_norm": 0.6108356714248657,
+      "learning_rate": 0.00018402010050251258,
+      "loss": 0.4325,
+      "step": 170
+    },
+    {
+      "epoch": 0.4375,
+      "grad_norm": 0.6409158706665039,
+      "learning_rate": 0.0001835175879396985,
+      "loss": 0.4792,
+      "step": 175
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.5427626371383667,
+      "learning_rate": 0.00018301507537688442,
+      "loss": 0.4591,
+      "step": 180
+    },
+    {
+      "epoch": 0.4625,
+      "grad_norm": 0.5712430477142334,
+      "learning_rate": 0.00018251256281407038,
+      "loss": 0.4663,
+      "step": 185
+    },
+    {
+      "epoch": 0.475,
+      "grad_norm": 0.5315428376197815,
+      "learning_rate": 0.00018201005025125628,
+      "loss": 0.4662,
+      "step": 190
+    },
+    {
+      "epoch": 0.4875,
+      "grad_norm": 0.6806014180183411,
+      "learning_rate": 0.00018150753768844221,
+      "loss": 0.4589,
+      "step": 195
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.4812653362751007,
+      "learning_rate": 0.00018100502512562815,
+      "loss": 0.4747,
+      "step": 200
+    },
+    {
+      "epoch": 0.5125,
+      "grad_norm": 0.620926558971405,
+      "learning_rate": 0.00018050251256281408,
+      "loss": 0.4689,
+      "step": 205
+    },
+    {
+      "epoch": 0.525,
+      "grad_norm": 0.5241305828094482,
+      "learning_rate": 0.00018,
+      "loss": 0.412,
+      "step": 210
+    },
+    {
+      "epoch": 0.5375,
+      "grad_norm": 0.5892027020454407,
+      "learning_rate": 0.00017949748743718592,
+      "loss": 0.4878,
+      "step": 215
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.5710367560386658,
+      "learning_rate": 0.00017899497487437188,
+      "loss": 0.4329,
+      "step": 220
+    },
+    {
+      "epoch": 0.5625,
+      "grad_norm": 0.5636909604072571,
+      "learning_rate": 0.0001784924623115578,
+      "loss": 0.4335,
+      "step": 225
+    },
+    {
+      "epoch": 0.575,
+      "grad_norm": 0.5712817311286926,
+      "learning_rate": 0.00017798994974874371,
+      "loss": 0.4728,
+      "step": 230
+    },
+    {
+      "epoch": 0.5875,
+      "grad_norm": 0.5080311298370361,
+      "learning_rate": 0.00017748743718592967,
+      "loss": 0.4576,
+      "step": 235
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.5396429896354675,
+      "learning_rate": 0.00017698492462311558,
+      "loss": 0.4731,
+      "step": 240
+    },
+    {
+      "epoch": 0.6125,
+      "grad_norm": 0.594582736492157,
+      "learning_rate": 0.0001764824120603015,
+      "loss": 0.4421,
+      "step": 245
+    },
+    {
+      "epoch": 0.625,
+      "grad_norm": 0.5550448298454285,
+      "learning_rate": 0.00017597989949748744,
+      "loss": 0.4473,
+      "step": 250
+    },
+    {
+      "epoch": 0.6375,
+      "grad_norm": 0.5841835737228394,
+      "learning_rate": 0.00017547738693467338,
+      "loss": 0.4141,
+      "step": 255
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 0.5154719352722168,
+      "learning_rate": 0.0001749748743718593,
+      "loss": 0.4178,
+      "step": 260
+    },
+    {
+      "epoch": 0.6625,
+      "grad_norm": 0.6086138486862183,
+      "learning_rate": 0.00017447236180904521,
+      "loss": 0.4309,
+      "step": 265
+    },
+    {
+      "epoch": 0.675,
+      "grad_norm": 0.5762608051300049,
+      "learning_rate": 0.00017396984924623117,
+      "loss": 0.4395,
+      "step": 270
+    },
+    {
+      "epoch": 0.6875,
+      "grad_norm": 0.6579628586769104,
+      "learning_rate": 0.0001734673366834171,
+      "loss": 0.4308,
+      "step": 275
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.5988358855247498,
+      "learning_rate": 0.000172964824120603,
+      "loss": 0.4157,
+      "step": 280
+    },
+    {
+      "epoch": 0.7125,
+      "grad_norm": 0.5210469365119934,
+      "learning_rate": 0.00017246231155778897,
+      "loss": 0.4433,
+      "step": 285
+    },
+    {
+      "epoch": 0.725,
+      "grad_norm": 0.5271325707435608,
+      "learning_rate": 0.00017195979899497488,
+      "loss": 0.429,
+      "step": 290
+    },
+    {
+      "epoch": 0.7375,
+      "grad_norm": 0.5668110847473145,
+      "learning_rate": 0.0001714572864321608,
+      "loss": 0.4789,
+      "step": 295
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.5577985644340515,
+      "learning_rate": 0.00017095477386934674,
+      "loss": 0.4278,
+      "step": 300
+    },
+    {
+      "epoch": 0.7625,
+      "grad_norm": 0.53072190284729,
+      "learning_rate": 0.00017045226130653267,
+      "loss": 0.4445,
+      "step": 305
+    },
+    {
+      "epoch": 0.775,
+      "grad_norm": 0.5304411053657532,
+      "learning_rate": 0.0001699497487437186,
+      "loss": 0.4329,
+      "step": 310
+    },
+    {
+      "epoch": 0.7875,
+      "grad_norm": 0.5181726813316345,
+      "learning_rate": 0.0001694472361809045,
+      "loss": 0.429,
+      "step": 315
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.5597214698791504,
+      "learning_rate": 0.00016894472361809047,
+      "loss": 0.4225,
+      "step": 320
+    },
+    {
+      "epoch": 0.8125,
+      "grad_norm": 0.5509529709815979,
+      "learning_rate": 0.0001684422110552764,
+      "loss": 0.4209,
+      "step": 325
+    },
+    {
+      "epoch": 0.825,
+      "grad_norm": 0.600302517414093,
+      "learning_rate": 0.0001679396984924623,
+      "loss": 0.4367,
+      "step": 330
+    },
+    {
+      "epoch": 0.8375,
+      "grad_norm": 0.5333036184310913,
+      "learning_rate": 0.00016743718592964827,
+      "loss": 0.4362,
+      "step": 335
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.5863168835639954,
+      "learning_rate": 0.00016693467336683417,
+      "loss": 0.4548,
+      "step": 340
+    },
+    {
+      "epoch": 0.8625,
+      "grad_norm": 0.526488184928894,
+      "learning_rate": 0.0001664321608040201,
+      "loss": 0.4399,
+      "step": 345
+    },
+    {
+      "epoch": 0.875,
+      "grad_norm": 0.5802232623100281,
+      "learning_rate": 0.00016592964824120604,
+      "loss": 0.4961,
+      "step": 350
+    },
+    {
+      "epoch": 0.8875,
+      "grad_norm": 0.5905202031135559,
+      "learning_rate": 0.00016542713567839197,
+      "loss": 0.4453,
+      "step": 355
+    },
+    {
+      "epoch": 0.9,
+      "grad_norm": 0.5512250065803528,
+      "learning_rate": 0.0001649246231155779,
+      "loss": 0.4135,
+      "step": 360
+    },
+    {
+      "epoch": 0.9125,
+      "grad_norm": 0.5865863561630249,
+      "learning_rate": 0.0001644221105527638,
+      "loss": 0.4645,
+      "step": 365
+    },
+    {
+      "epoch": 0.925,
+      "grad_norm": 0.538692831993103,
+      "learning_rate": 0.00016391959798994977,
+      "loss": 0.4821,
+      "step": 370
+    },
+    {
+      "epoch": 0.9375,
+      "grad_norm": 0.5884225368499756,
+      "learning_rate": 0.0001634170854271357,
+      "loss": 0.4484,
+      "step": 375
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.5514518618583679,
+      "learning_rate": 0.0001629145728643216,
+      "loss": 0.4529,
+      "step": 380
+    },
+    {
+      "epoch": 0.9625,
+      "grad_norm": 0.577700674533844,
+      "learning_rate": 0.00016241206030150756,
+      "loss": 0.4651,
+      "step": 385
+    },
+    {
+      "epoch": 0.975,
+      "grad_norm": 0.5781076550483704,
+      "learning_rate": 0.00016190954773869347,
+      "loss": 0.4349,
+      "step": 390
+    },
+    {
+      "epoch": 0.9875,
+      "grad_norm": 0.47729212045669556,
+      "learning_rate": 0.0001614070351758794,
+      "loss": 0.4509,
+      "step": 395
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.5096570253372192,
+      "learning_rate": 0.00016090452261306533,
+      "loss": 0.4153,
+      "step": 400
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 0.5257100462913513,
+      "eval_runtime": 17.864,
+      "eval_samples_per_second": 22.391,
+      "eval_steps_per_second": 2.799,
+      "step": 400
+    },
+    {
+      "epoch": 1.0125,
+      "grad_norm": 0.49858036637306213,
+      "learning_rate": 0.00016040201005025127,
+      "loss": 0.427,
+      "step": 405
+    },
+    {
+      "epoch": 1.025,
+      "grad_norm": 0.5483050346374512,
+      "learning_rate": 0.0001598994974874372,
+      "loss": 0.4414,
+      "step": 410
+    },
+    {
+      "epoch": 1.0375,
+      "grad_norm": 0.5298231244087219,
+      "learning_rate": 0.0001593969849246231,
+      "loss": 0.399,
+      "step": 415
+    },
+    {
+      "epoch": 1.05,
+      "grad_norm": 0.5137779712677002,
+      "learning_rate": 0.00015889447236180906,
+      "loss": 0.4153,
+      "step": 420
+    },
+    {
+      "epoch": 1.0625,
+      "grad_norm": 0.5221366882324219,
+      "learning_rate": 0.000158391959798995,
+      "loss": 0.4176,
+      "step": 425
+    },
+    {
+      "epoch": 1.075,
+      "grad_norm": 0.6192128658294678,
+      "learning_rate": 0.0001578894472361809,
+      "loss": 0.4107,
+      "step": 430
+    },
+    {
+      "epoch": 1.0875,
+      "grad_norm": 0.5545984506607056,
+      "learning_rate": 0.00015738693467336686,
+      "loss": 0.4102,
+      "step": 435
+    },
+    {
+      "epoch": 1.1,
+      "grad_norm": 0.5551674365997314,
+      "learning_rate": 0.00015688442211055277,
+      "loss": 0.4129,
+      "step": 440
+    },
+    {
+      "epoch": 1.1125,
+      "grad_norm": 0.505490779876709,
+      "learning_rate": 0.0001563819095477387,
+      "loss": 0.4172,
+      "step": 445
+    },
+    {
+      "epoch": 1.125,
+      "grad_norm": 0.5239485502243042,
+      "learning_rate": 0.00015587939698492463,
+      "loss": 0.4188,
+      "step": 450
+    },
+    {
+      "epoch": 1.1375,
+      "grad_norm": 0.5464276671409607,
+      "learning_rate": 0.00015537688442211056,
+      "loss": 0.4155,
+      "step": 455
+    },
+    {
+      "epoch": 1.15,
+      "grad_norm": 0.496952623128891,
+      "learning_rate": 0.0001548743718592965,
+      "loss": 0.4166,
+      "step": 460
+    },
+    {
+      "epoch": 1.1625,
+      "grad_norm": 0.5454787015914917,
+      "learning_rate": 0.0001543718592964824,
+      "loss": 0.4146,
+      "step": 465
+    },
+    {
+      "epoch": 1.175,
+      "grad_norm": 0.5375080704689026,
+      "learning_rate": 0.00015386934673366836,
+      "loss": 0.4166,
+      "step": 470
+    },
+    {
+      "epoch": 1.1875,
+      "grad_norm": 0.5614022612571716,
+      "learning_rate": 0.00015336683417085427,
+      "loss": 0.3995,
+      "step": 475
+    },
+    {
+      "epoch": 1.2,
+      "grad_norm": 0.530121386051178,
+      "learning_rate": 0.0001528643216080402,
+      "loss": 0.4187,
+      "step": 480
+    },
+    {
+      "epoch": 1.2125,
+      "grad_norm": 0.4971998333930969,
+      "learning_rate": 0.00015236180904522613,
+      "loss": 0.3899,
+      "step": 485
+    },
+    {
+      "epoch": 1.225,
+      "grad_norm": 0.5462106466293335,
+      "learning_rate": 0.00015185929648241206,
+      "loss": 0.4359,
+      "step": 490
+    },
+    {
+      "epoch": 1.2375,
+      "grad_norm": 0.5236663222312927,
+      "learning_rate": 0.000151356783919598,
+      "loss": 0.4127,
+      "step": 495
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.5054829716682434,
+      "learning_rate": 0.00015085427135678393,
+      "loss": 0.4064,
+      "step": 500
+    },
+    {
+      "epoch": 1.2625,
+      "grad_norm": 0.5926477909088135,
+      "learning_rate": 0.00015035175879396986,
+      "loss": 0.4085,
+      "step": 505
+    },
+    {
+      "epoch": 1.275,
+      "grad_norm": 0.5047740936279297,
+      "learning_rate": 0.0001498492462311558,
+      "loss": 0.398,
+      "step": 510
+    },
+    {
+      "epoch": 1.2875,
+      "grad_norm": 0.5045680403709412,
+      "learning_rate": 0.0001493467336683417,
+      "loss": 0.4198,
+      "step": 515
+    },
+    {
+      "epoch": 1.3,
+      "grad_norm": 0.6037045121192932,
+      "learning_rate": 0.00014884422110552766,
+      "loss": 0.4625,
+      "step": 520
+    },
+    {
+      "epoch": 1.3125,
+      "grad_norm": 0.5416387915611267,
+      "learning_rate": 0.00014834170854271356,
+      "loss": 0.437,
+      "step": 525
+    },
+    {
+      "epoch": 1.325,
+      "grad_norm": 0.5217929482460022,
+      "learning_rate": 0.0001478391959798995,
+      "loss": 0.4255,
+      "step": 530
+    },
+    {
+      "epoch": 1.3375,
+      "grad_norm": 0.6118584275245667,
+      "learning_rate": 0.00014733668341708543,
+      "loss": 0.4395,
+      "step": 535
+    },
+    {
+      "epoch": 1.35,
+      "grad_norm": 0.5826886892318726,
+      "learning_rate": 0.00014683417085427136,
+      "loss": 0.4571,
+      "step": 540
+    },
+    {
+      "epoch": 1.3625,
+      "grad_norm": 0.5662903785705566,
+      "learning_rate": 0.0001463316582914573,
+      "loss": 0.4338,
+      "step": 545
+    },
+    {
+      "epoch": 1.375,
+      "grad_norm": 0.5721105933189392,
+      "learning_rate": 0.00014582914572864323,
+      "loss": 0.4413,
+      "step": 550
+    },
+    {
+      "epoch": 1.3875,
+      "grad_norm": 0.5065754055976868,
+      "learning_rate": 0.00014532663316582916,
+      "loss": 0.3901,
+      "step": 555
+    },
+    {
+      "epoch": 1.4,
+      "grad_norm": 0.5614690780639648,
+      "learning_rate": 0.0001448241206030151,
+      "loss": 0.4349,
+      "step": 560
+    },
+    {
+      "epoch": 1.4125,
+      "grad_norm": 0.5476651787757874,
+      "learning_rate": 0.000144321608040201,
+      "loss": 0.4219,
+      "step": 565
+    },
+    {
+      "epoch": 1.425,
+      "grad_norm": 0.6010956764221191,
+      "learning_rate": 0.00014381909547738696,
+      "loss": 0.4107,
+      "step": 570
+    },
+    {
+      "epoch": 1.4375,
+      "grad_norm": 0.5348302721977234,
+      "learning_rate": 0.00014331658291457286,
+      "loss": 0.406,
+      "step": 575
+    },
+    {
+      "epoch": 1.45,
+      "grad_norm": 0.5349542498588562,
+      "learning_rate": 0.0001428140703517588,
+      "loss": 0.4016,
+      "step": 580
+    },
+    {
+      "epoch": 1.4625,
+      "grad_norm": 0.5250505208969116,
+      "learning_rate": 0.00014231155778894473,
+      "loss": 0.4373,
+      "step": 585
+    },
+    {
+      "epoch": 1.475,
+      "grad_norm": 0.5296792984008789,
+      "learning_rate": 0.00014180904522613066,
+      "loss": 0.4394,
+      "step": 590
+    },
+    {
+      "epoch": 1.4875,
+      "grad_norm": 0.5652737021446228,
+      "learning_rate": 0.0001413065326633166,
+      "loss": 0.3933,
+      "step": 595
+    },
+    {
+      "epoch": 1.5,
+      "grad_norm": 0.5297503471374512,
+      "learning_rate": 0.00014080402010050252,
+      "loss": 0.4161,
+      "step": 600
+    },
+    {
+      "epoch": 1.5125,
+      "grad_norm": 0.5393604040145874,
+      "learning_rate": 0.00014030150753768846,
+      "loss": 0.4245,
+      "step": 605
+    },
+    {
+      "epoch": 1.525,
+      "grad_norm": 0.4794439375400543,
+      "learning_rate": 0.0001397989949748744,
+      "loss": 0.4124,
+      "step": 610
+    },
+    {
+      "epoch": 1.5375,
+      "grad_norm": 0.49562177062034607,
+      "learning_rate": 0.0001392964824120603,
+      "loss": 0.4092,
+      "step": 615
+    },
+    {
+      "epoch": 1.55,
+      "grad_norm": 0.562262237071991,
+      "learning_rate": 0.00013879396984924625,
+      "loss": 0.4007,
+      "step": 620
+    },
+    {
+      "epoch": 1.5625,
+      "grad_norm": 0.6310052871704102,
+      "learning_rate": 0.00013829145728643216,
+      "loss": 0.4124,
+      "step": 625
+    },
+    {
+      "epoch": 1.575,
+      "grad_norm": 0.5664736032485962,
+      "learning_rate": 0.0001377889447236181,
+      "loss": 0.3917,
+      "step": 630
+    },
+    {
+      "epoch": 1.5875,
+      "grad_norm": 0.5304264426231384,
+      "learning_rate": 0.00013728643216080402,
+      "loss": 0.4028,
+      "step": 635
+    },
+    {
+      "epoch": 1.6,
+      "grad_norm": 0.48095929622650146,
+      "learning_rate": 0.00013678391959798996,
+      "loss": 0.4264,
+      "step": 640
+    },
+    {
+      "epoch": 1.6125,
+      "grad_norm": 0.5145896077156067,
+      "learning_rate": 0.0001362814070351759,
+      "loss": 0.42,
+      "step": 645
+    },
+    {
+      "epoch": 1.625,
+      "grad_norm": 0.5223731398582458,
+      "learning_rate": 0.0001357788944723618,
+      "loss": 0.4678,
+      "step": 650
+    },
+    {
+      "epoch": 1.6375,
+      "grad_norm": 0.5224917531013489,
+      "learning_rate": 0.00013527638190954775,
+      "loss": 0.3959,
+      "step": 655
+    },
+    {
+      "epoch": 1.65,
+      "grad_norm": 0.542758047580719,
+      "learning_rate": 0.00013477386934673368,
+      "loss": 0.4055,
+      "step": 660
+    },
+    {
+      "epoch": 1.6625,
+      "grad_norm": 0.5222824215888977,
+      "learning_rate": 0.0001342713567839196,
+      "loss": 0.4452,
+      "step": 665
+    },
+    {
+      "epoch": 1.675,
+      "grad_norm": 0.5793522596359253,
+      "learning_rate": 0.00013376884422110555,
+      "loss": 0.4131,
+      "step": 670
+    },
+    {
+      "epoch": 1.6875,
+      "grad_norm": 0.5987710952758789,
+      "learning_rate": 0.00013326633165829146,
+      "loss": 0.4091,
+      "step": 675
+    },
+    {
+      "epoch": 1.7,
+      "grad_norm": 0.6028478741645813,
+      "learning_rate": 0.0001327638190954774,
+      "loss": 0.4376,
+      "step": 680
+    },
+    {
+      "epoch": 1.7125,
+      "grad_norm": 0.5682055950164795,
+      "learning_rate": 0.00013226130653266332,
+      "loss": 0.4102,
+      "step": 685
+    },
+    {
+      "epoch": 1.725,
+      "grad_norm": 0.5360214710235596,
+      "learning_rate": 0.00013175879396984925,
+      "loss": 0.4456,
+      "step": 690
+    },
+    {
+      "epoch": 1.7375,
+      "grad_norm": 0.5613261461257935,
+      "learning_rate": 0.00013125628140703518,
+      "loss": 0.4042,
+      "step": 695
+    },
+    {
+      "epoch": 1.75,
+      "grad_norm": 0.5854392647743225,
+      "learning_rate": 0.0001307537688442211,
+      "loss": 0.3754,
+      "step": 700
+    },
+    {
+      "epoch": 1.7625,
+      "grad_norm": 0.5785875916481018,
+      "learning_rate": 0.00013025125628140705,
+      "loss": 0.4128,
+      "step": 705
+    },
+    {
+      "epoch": 1.775,
+      "grad_norm": 0.5500577092170715,
+      "learning_rate": 0.00012974874371859298,
+      "loss": 0.4176,
+      "step": 710
+    },
+    {
+      "epoch": 1.7875,
+      "grad_norm": 0.5343199372291565,
+      "learning_rate": 0.0001292462311557789,
+      "loss": 0.4008,
+      "step": 715
+    },
+    {
+      "epoch": 1.8,
+      "grad_norm": 0.5914020538330078,
+      "learning_rate": 0.00012874371859296485,
+      "loss": 0.3919,
+      "step": 720
+    },
+    {
+      "epoch": 1.8125,
+      "grad_norm": 0.5501930117607117,
+      "learning_rate": 0.00012824120603015075,
+      "loss": 0.3797,
+      "step": 725
+    },
+    {
+      "epoch": 1.825,
+      "grad_norm": 0.5344274640083313,
+      "learning_rate": 0.00012773869346733668,
+      "loss": 0.4045,
+      "step": 730
+    },
+    {
+      "epoch": 1.8375,
+      "grad_norm": 0.5303798913955688,
+      "learning_rate": 0.00012723618090452262,
+      "loss": 0.4098,
+      "step": 735
+    },
+    {
+      "epoch": 1.85,
+      "grad_norm": 0.5791289806365967,
+      "learning_rate": 0.00012673366834170855,
+      "loss": 0.4166,
+      "step": 740
+    },
+    {
+      "epoch": 1.8625,
+      "grad_norm": 0.5153045058250427,
+      "learning_rate": 0.00012623115577889448,
+      "loss": 0.4011,
+      "step": 745
+    },
+    {
+      "epoch": 1.875,
+      "grad_norm": 0.5780820846557617,
+      "learning_rate": 0.0001257286432160804,
+      "loss": 0.3839,
+      "step": 750
+    },
+    {
+      "epoch": 1.8875,
+      "grad_norm": 0.5261033177375793,
+      "learning_rate": 0.00012522613065326635,
+      "loss": 0.402,
+      "step": 755
+    },
+    {
+      "epoch": 1.9,
+      "grad_norm": 0.5911995768547058,
+      "learning_rate": 0.00012472361809045228,
+      "loss": 0.4032,
+      "step": 760
+    },
+    {
+      "epoch": 1.9125,
+      "grad_norm": 0.519932746887207,
+      "learning_rate": 0.00012422110552763818,
+      "loss": 0.4091,
+      "step": 765
+    },
+    {
+      "epoch": 1.925,
+      "grad_norm": 0.5260487794876099,
+      "learning_rate": 0.00012371859296482414,
+      "loss": 0.3896,
+      "step": 770
+    },
+    {
+      "epoch": 1.9375,
+      "grad_norm": 0.5135574340820312,
+      "learning_rate": 0.00012321608040201005,
+      "loss": 0.4313,
+      "step": 775
+    },
+    {
+      "epoch": 1.95,
+      "grad_norm": 0.6207651495933533,
+      "learning_rate": 0.00012271356783919598,
+      "loss": 0.4357,
+      "step": 780
+    },
+    {
+      "epoch": 1.9625,
+      "grad_norm": 0.5082093477249146,
+      "learning_rate": 0.00012221105527638191,
+      "loss": 0.4082,
+      "step": 785
+    },
+    {
+      "epoch": 1.975,
+      "grad_norm": 0.5137083530426025,
+      "learning_rate": 0.00012170854271356785,
+      "loss": 0.402,
+      "step": 790
+    },
+    {
+      "epoch": 1.9875,
+      "grad_norm": 0.5387287139892578,
+      "learning_rate": 0.00012120603015075378,
+      "loss": 0.3952,
+      "step": 795
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 0.5328527092933655,
+      "learning_rate": 0.0001207035175879397,
+      "loss": 0.394,
+      "step": 800
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 0.5232270359992981,
+      "eval_runtime": 17.3583,
+      "eval_samples_per_second": 23.044,
+      "eval_steps_per_second": 2.88,
+      "step": 800
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 2000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 2,
+        "early_stopping_threshold": 0.01
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 1
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.747212965907456e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-800/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70074cd0f36e0bbd26ec8acd16a12c18f0120870f7f2a1624c8b0f6e58a06a26
+size 5777

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70074cd0f36e0bbd26ec8acd16a12c18f0120870f7f2a1624c8b0f6e58a06a26
+size 5777