Upload folder using huggingface_hub

README.md

@@ -0,0 +1,209 @@
+---
+base_model: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "q_proj",
+    "v_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:457ab064324fe56cc82f7fd2a253d24365a9740289c2cfdb6b84ff487e7b391c
+size 11084320

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{% if loop.first and message['role'] != 'system' %}{{ '[|system|][|endofturn|]
+' }}{% endif %}{{ '[|' + message['role'] + '|]' + message['content'] }}{% if message['role'] == 'user' %}{{ '
+' }}{% else %}{{ '[|endofturn|]
+' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '[|assistant|]' }}{% endif %}

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726b809e9b245b726dc6bf7b366f519a713eb7a5eaf2c1205509c0ea73a40f5b
+size 22269626

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2059937ef78b30562ecae8b5ce0fc4732393da3e33a64d6e0f947760553b676
+size 14244

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6d2e4ccf3f434eedd99c5624ed87df7ac7ee034200b2c67a1c6236ad775d2f5
+size 1064

tokenizer_config.json

@@ -0,0 +1,325 @@
+{
+  "add_prefix_space": false,
+  "additional_special_token": [
+    "[unused0]",
+    "[unused1]",
+    "[unused2]",
+    "[unused3]",
+    "[unused4]",
+    "[unused5]",
+    "[unused6]",
+    "[unused7]",
+    "[unused8]",
+    "[unused9]",
+    "[unused10]",
+    "[unused11]",
+    "[unused12]",
+    "[unused13]",
+    "[unused14]",
+    "[unused15]",
+    "[unused16]",
+    "[unused17]",
+    "[unused18]",
+    "[unused19]",
+    "[unused20]",
+    "[unused21]",
+    "[unused22]",
+    "[unused23]",
+    "[unused24]",
+    "[unused25]",
+    "[unused26]",
+    "[unused27]",
+    "[unused28]",
+    "[unused29]",
+    "[unused30]",
+    "[unused31]",
+    "[unused32]",
+    "[unused33]",
+    "[unused34]",
+    "[unused35]",
+    "[unused36]",
+    "[unused37]",
+    "[unused38]",
+    "[unused39]",
+    "[unused40]",
+    "[unused41]",
+    "[unused42]",
+    "[unused43]",
+    "[unused44]",
+    "[unused45]",
+    "[unused46]",
+    "[unused47]",
+    "[unused48]",
+    "[unused49]",
+    "[unused50]",
+    "[unused51]",
+    "[unused52]",
+    "[unused53]",
+    "[unused54]",
+    "[unused55]",
+    "[unused56]",
+    "[unused57]",
+    "[unused58]",
+    "[unused59]",
+    "[unused60]",
+    "[unused61]",
+    "[unused62]",
+    "[unused63]",
+    "[unused64]",
+    "[unused65]",
+    "[unused66]",
+    "[unused67]",
+    "[unused68]",
+    "[unused69]",
+    "[unused70]",
+    "[unused71]",
+    "[unused72]",
+    "[unused73]",
+    "[unused74]",
+    "[unused75]",
+    "[unused76]",
+    "[unused77]",
+    "[unused78]",
+    "[unused79]",
+    "[unused80]",
+    "[unused81]",
+    "[unused82]",
+    "[unused83]",
+    "[unused84]",
+    "[unused85]",
+    "[unused86]",
+    "[unused87]",
+    "[unused88]",
+    "[unused89]",
+    "[unused90]",
+    "[unused91]",
+    "[unused92]",
+    "[unused93]",
+    "[unused94]",
+    "[unused95]",
+    "[unused96]",
+    "[unused97]",
+    "[unused98]",
+    "[unused99]",
+    "[extra_id_0]",
+    "[extra_id_1]",
+    "[extra_id_2]",
+    "[extra_id_3]",
+    "[extra_id_4]",
+    "[extra_id_5]",
+    "[extra_id_6]",
+    "[extra_id_7]",
+    "[extra_id_8]",
+    "[extra_id_9]",
+    "[extra_id_10]",
+    "[extra_id_11]",
+    "[extra_id_12]",
+    "[extra_id_13]",
+    "[extra_id_14]",
+    "[extra_id_15]",
+    "[extra_id_16]",
+    "[extra_id_17]",
+    "[extra_id_18]",
+    "[extra_id_19]",
+    "[extra_id_20]",
+    "[extra_id_21]",
+    "[extra_id_22]",
+    "[extra_id_23]",
+    "[extra_id_24]",
+    "[extra_id_25]",
+    "[extra_id_26]",
+    "[extra_id_27]",
+    "[extra_id_28]",
+    "[extra_id_29]",
+    "[extra_id_30]",
+    "[extra_id_31]",
+    "[extra_id_32]",
+    "[extra_id_33]",
+    "[extra_id_34]",
+    "[extra_id_35]",
+    "[extra_id_36]",
+    "[extra_id_37]",
+    "[extra_id_38]",
+    "[extra_id_39]",
+    "[extra_id_40]",
+    "[extra_id_41]",
+    "[extra_id_42]",
+    "[extra_id_43]",
+    "[extra_id_44]",
+    "[extra_id_45]",
+    "[extra_id_46]",
+    "[extra_id_47]",
+    "[extra_id_48]",
+    "[extra_id_49]",
+    "[extra_id_50]",
+    "[extra_id_51]",
+    "[extra_id_52]",
+    "[extra_id_53]",
+    "[extra_id_54]",
+    "[extra_id_55]",
+    "[extra_id_56]",
+    "[extra_id_57]",
+    "[extra_id_58]",
+    "[extra_id_59]",
+    "[extra_id_60]",
+    "[extra_id_61]",
+    "[extra_id_62]",
+    "[extra_id_63]",
+    "[extra_id_64]",
+    "[extra_id_65]",
+    "[extra_id_66]",
+    "[extra_id_67]",
+    "[extra_id_68]",
+    "[extra_id_69]",
+    "[extra_id_70]",
+    "[extra_id_71]",
+    "[extra_id_72]",
+    "[extra_id_73]",
+    "[extra_id_74]",
+    "[extra_id_75]",
+    "[extra_id_76]",
+    "[extra_id_77]",
+    "[extra_id_78]",
+    "[extra_id_79]",
+    "[extra_id_80]",
+    "[extra_id_81]",
+    "[extra_id_82]",
+    "[extra_id_83]",
+    "[extra_id_84]",
+    "[extra_id_85]",
+    "[extra_id_86]",
+    "[extra_id_87]",
+    "[extra_id_88]",
+    "[extra_id_89]",
+    "[extra_id_90]",
+    "[extra_id_91]",
+    "[extra_id_92]",
+    "[extra_id_93]",
+    "[extra_id_94]",
+    "[extra_id_95]",
+    "[extra_id_96]",
+    "[extra_id_97]",
+    "[extra_id_98]",
+    "[extra_id_99]",
+    "[extra_id_100]",
+    "[extra_id_101]",
+    "[extra_id_102]",
+    "[extra_id_103]",
+    "[extra_id_104]",
+    "[extra_id_105]",
+    "[extra_id_106]",
+    "[extra_id_107]",
+    "[extra_id_108]",
+    "[extra_id_109]",
+    "[extra_id_110]",
+    "[extra_id_111]",
+    "[extra_id_112]",
+    "[extra_id_113]",
+    "[extra_id_114]",
+    "[extra_id_115]",
+    "[extra_id_116]",
+    "[extra_id_117]",
+    "[extra_id_118]",
+    "[extra_id_119]",
+    "[extra_id_120]",
+    "[extra_id_121]",
+    "[extra_id_122]",
+    "[extra_id_123]",
+    "[extra_id_124]",
+    "[extra_id_125]",
+    "[extra_id_126]",
+    "[extra_id_127]",
+    "[extra_id_128]",
+    "[extra_id_129]",
+    "[extra_id_130]",
+    "[extra_id_131]",
+    "[extra_id_132]",
+    "[extra_id_133]",
+    "[extra_id_134]",
+    "[extra_id_135]",
+    "[extra_id_136]",
+    "[extra_id_137]",
+    "[extra_id_138]",
+    "[extra_id_139]",
+    "[extra_id_140]",
+    "[extra_id_141]",
+    "[extra_id_142]",
+    "[extra_id_143]",
+    "[extra_id_144]",
+    "[extra_id_145]",
+    "[extra_id_146]",
+    "[extra_id_147]",
+    "[extra_id_148]",
+    "[extra_id_149]",
+    "[extra_id_150]",
+    "[extra_id_151]",
+    "[extra_id_152]",
+    "[extra_id_153]",
+    "[extra_id_154]",
+    "[extra_id_155]",
+    "[extra_id_156]",
+    "[extra_id_157]",
+    "[extra_id_158]",
+    "[extra_id_159]",
+    "[extra_id_160]",
+    "[extra_id_161]",
+    "[extra_id_162]",
+    "[extra_id_163]",
+    "[extra_id_164]",
+    "[extra_id_165]",
+    "[extra_id_166]",
+    "[extra_id_167]",
+    "[extra_id_168]",
+    "[extra_id_169]",
+    "[extra_id_170]",
+    "[extra_id_171]",
+    "[extra_id_172]",
+    "[extra_id_173]",
+    "[extra_id_174]",
+    "[extra_id_175]",
+    "[extra_id_176]",
+    "[extra_id_177]",
+    "[extra_id_178]",
+    "[extra_id_179]",
+    "[extra_id_180]",
+    "[extra_id_181]",
+    "[extra_id_182]",
+    "[extra_id_183]",
+    "[extra_id_184]",
+    "[extra_id_185]",
+    "[extra_id_186]",
+    "[extra_id_187]",
+    "[extra_id_188]",
+    "[extra_id_189]",
+    "[extra_id_190]",
+    "[extra_id_191]",
+    "[extra_id_192]",
+    "[extra_id_193]",
+    "[extra_id_194]",
+    "[extra_id_195]",
+    "[extra_id_196]",
+    "[extra_id_197]",
+    "[extra_id_198]",
+    "[|endofturn|]",
+    "PI:URL",
+    "PI:EMAIL",
+    "PI:ACCOUNT_NUM",
+    "PI:PHONE_NUM",
+    "PI:BUSINESS_NUM",
+    "PI:ANNON",
+    "PI:KEY",
+    "PI:ID",
+    "PI:IP_ADDRESS",
+    "PI:USER"
+  ],
+  "backend": "tokenizers",
+  "bos_token": "[BOS]",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "[|endofturn|]",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "[UNK]"
+}

trainer_state.json

@@ -0,0 +1,696 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 15.0,
+  "eval_steps": 200,
+  "global_step": 6210,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.5034248107671737,
+      "epoch": 0.4839685420447671,
+      "grad_norm": 0.609375,
+      "learning_rate": 1.9900000000000003e-05,
+      "loss": 3.8386904907226564,
+      "mean_token_accuracy": 0.4763091333582997,
+      "num_tokens": 258555.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.4839685420447671,
+      "eval_entropy": 1.89908932659724,
+      "eval_loss": 1.6288481950759888,
+      "eval_mean_token_accuracy": 0.7560858702006406,
+      "eval_num_tokens": 258555.0,
+      "eval_runtime": 48.9754,
+      "eval_samples_per_second": 5.921,
+      "eval_steps_per_second": 1.491,
+      "step": 200
+    },
+    {
+      "entropy": 0.9987735440582037,
+      "epoch": 0.9679370840895342,
+      "grad_norm": 0.1767578125,
+      "learning_rate": 3.99e-05,
+      "loss": 0.8650601196289063,
+      "mean_token_accuracy": 0.8588662607967854,
+      "num_tokens": 516030.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.9679370840895342,
+      "eval_entropy": 0.7666851747525881,
+      "eval_loss": 0.7402243614196777,
+      "eval_mean_token_accuracy": 0.8760081317326794,
+      "eval_num_tokens": 516030.0,
+      "eval_runtime": 46.9,
+      "eval_samples_per_second": 6.183,
+      "eval_steps_per_second": 1.557,
+      "step": 400
+    },
+    {
+      "entropy": 0.6914621311119539,
+      "epoch": 1.4500907441016335,
+      "grad_norm": 0.265625,
+      "learning_rate": 5.99e-05,
+      "loss": 0.6239362335205079,
+      "mean_token_accuracy": 0.8933641332455232,
+      "num_tokens": 773384.0,
+      "step": 600
+    },
+    {
+      "epoch": 1.4500907441016335,
+      "eval_entropy": 0.6780054618234503,
+      "eval_loss": 0.6326681971549988,
+      "eval_mean_token_accuracy": 0.8811007974898979,
+      "eval_num_tokens": 773384.0,
+      "eval_runtime": 47.2911,
+      "eval_samples_per_second": 6.132,
+      "eval_steps_per_second": 1.544,
+      "step": 600
+    },
+    {
+      "entropy": 0.5927861626073718,
+      "epoch": 1.9340592861464003,
+      "grad_norm": 0.240234375,
+      "learning_rate": 7.99e-05,
+      "loss": 0.5069254684448242,
+      "mean_token_accuracy": 0.9013223663717508,
+      "num_tokens": 1031174.0,
+      "step": 800
+    },
+    {
+      "epoch": 1.9340592861464003,
+      "eval_entropy": 0.6229167531614435,
+      "eval_loss": 0.5980483293533325,
+      "eval_mean_token_accuracy": 0.893642638644127,
+      "eval_num_tokens": 1031174.0,
+      "eval_runtime": 47.2124,
+      "eval_samples_per_second": 6.142,
+      "eval_steps_per_second": 1.546,
+      "step": 800
+    },
+    {
+      "entropy": 0.5391529030452857,
+      "epoch": 2.4162129461585,
+      "grad_norm": 0.2080078125,
+      "learning_rate": 9.99e-05,
+      "loss": 0.4552752685546875,
+      "mean_token_accuracy": 0.913724020927434,
+      "num_tokens": 1287528.0,
+      "step": 1000
+    },
+    {
+      "epoch": 2.4162129461585,
+      "eval_entropy": 0.5597588991465634,
+      "eval_loss": 0.5732213258743286,
+      "eval_mean_token_accuracy": 0.8946846454110864,
+      "eval_num_tokens": 1287528.0,
+      "eval_runtime": 46.718,
+      "eval_samples_per_second": 6.207,
+      "eval_steps_per_second": 1.563,
+      "step": 1000
+    },
+    {
+      "entropy": 0.48732462897896767,
+      "epoch": 2.900181488203267,
+      "grad_norm": 0.1865234375,
+      "learning_rate": 9.964045828736265e-05,
+      "loss": 0.43026100158691405,
+      "mean_token_accuracy": 0.9152925322949886,
+      "num_tokens": 1545979.0,
+      "step": 1200
+    },
+    {
+      "epoch": 2.900181488203267,
+      "eval_entropy": 0.5305672479818945,
+      "eval_loss": 0.571603000164032,
+      "eval_mean_token_accuracy": 0.8954706118531424,
+      "eval_num_tokens": 1545979.0,
+      "eval_runtime": 87.5618,
+      "eval_samples_per_second": 3.312,
+      "eval_steps_per_second": 0.834,
+      "step": 1200
+    },
+    {
+      "entropy": 0.4696483660297681,
+      "epoch": 3.382335148215366,
+      "grad_norm": 0.1953125,
+      "learning_rate": 9.855982874039193e-05,
+      "loss": 0.41300594329833984,
+      "mean_token_accuracy": 0.9177645573502353,
+      "num_tokens": 1802155.0,
+      "step": 1400
+    },
+    {
+      "epoch": 3.382335148215366,
+      "eval_entropy": 0.5139341966746604,
+      "eval_loss": 0.580710232257843,
+      "eval_mean_token_accuracy": 0.8955999376022652,
+      "eval_num_tokens": 1802155.0,
+      "eval_runtime": 87.6062,
+      "eval_samples_per_second": 3.31,
+      "eval_steps_per_second": 0.833,
+      "step": 1400
+    },
+    {
+      "entropy": 0.4651796716451645,
+      "epoch": 3.866303690260133,
+      "grad_norm": 0.2001953125,
+      "learning_rate": 9.677379992114693e-05,
+      "loss": 0.40818458557128906,
+      "mean_token_accuracy": 0.9180439539253712,
+      "num_tokens": 2060871.0,
+      "step": 1600
+    },
+    {
+      "epoch": 3.866303690260133,
+      "eval_entropy": 0.5178806251042509,
+      "eval_loss": 0.5665879845619202,
+      "eval_mean_token_accuracy": 0.8960714160579525,
+      "eval_num_tokens": 2060871.0,
+      "eval_runtime": 87.2236,
+      "eval_samples_per_second": 3.325,
+      "eval_steps_per_second": 0.837,
+      "step": 1600
+    },
+    {
+      "entropy": 0.44904655928593806,
+      "epoch": 4.348457350272232,
+      "grad_norm": 0.208984375,
+      "learning_rate": 9.430831639043096e-05,
+      "loss": 0.39284423828125,
+      "mean_token_accuracy": 0.9206457537420122,
+      "num_tokens": 2317399.0,
+      "step": 1800
+    },
+    {
+      "epoch": 4.348457350272232,
+      "eval_entropy": 0.5153208285161893,
+      "eval_loss": 0.5782919526100159,
+      "eval_mean_token_accuracy": 0.8959851983475359,
+      "eval_num_tokens": 2317399.0,
+      "eval_runtime": 47.2746,
+      "eval_samples_per_second": 6.134,
+      "eval_steps_per_second": 1.544,
+      "step": 1800
+    },
+    {
+      "entropy": 0.44385369334369895,
+      "epoch": 4.832425892317,
+      "grad_norm": 0.19921875,
+      "learning_rate": 9.119919273683477e-05,
+      "loss": 0.3875685501098633,
+      "mean_token_accuracy": 0.9209027971327305,
+      "num_tokens": 2575616.0,
+      "step": 2000
+    },
+    {
+      "epoch": 4.832425892317,
+      "eval_entropy": 0.4954172817811574,
+      "eval_loss": 0.5729158520698547,
+      "eval_mean_token_accuracy": 0.895403119799209,
+      "eval_num_tokens": 2575616.0,
+      "eval_runtime": 46.8739,
+      "eval_samples_per_second": 6.187,
+      "eval_steps_per_second": 1.557,
+      "step": 2000
+    },
+    {
+      "entropy": 0.43330108902535147,
+      "epoch": 5.314579552329098,
+      "grad_norm": 0.201171875,
+      "learning_rate": 8.749159331986178e-05,
+      "loss": 0.37568603515625,
+      "mean_token_accuracy": 0.9220367068029856,
+      "num_tokens": 2832652.0,
+      "step": 2200
+    },
+    {
+      "epoch": 5.314579552329098,
+      "eval_entropy": 0.4961876812046521,
+      "eval_loss": 0.5800752639770508,
+      "eval_mean_token_accuracy": 0.8957706052963048,
+      "eval_num_tokens": 2832652.0,
+      "eval_runtime": 47.319,
+      "eval_samples_per_second": 6.129,
+      "eval_steps_per_second": 1.543,
+      "step": 2200
+    },
+    {
+      "entropy": 0.430719858892262,
+      "epoch": 5.798548094373865,
+      "grad_norm": 0.26171875,
+      "learning_rate": 8.323937619458462e-05,
+      "loss": 0.3715415191650391,
+      "mean_token_accuracy": 0.9230316735059023,
+      "num_tokens": 3090576.0,
+      "step": 2400
+    },
+    {
+      "epoch": 5.798548094373865,
+      "eval_entropy": 0.4881298513445136,
+      "eval_loss": 0.5789266228675842,
+      "eval_mean_token_accuracy": 0.8953908681869507,
+      "eval_num_tokens": 3090576.0,
+      "eval_runtime": 47.2341,
+      "eval_samples_per_second": 6.14,
+      "eval_steps_per_second": 1.545,
+      "step": 2400
+    },
+    {
+      "entropy": 0.4203071318650934,
+      "epoch": 6.280701754385965,
+      "grad_norm": 0.2294921875,
+      "learning_rate": 7.850431074824228e-05,
+      "loss": 0.3627183151245117,
+      "mean_token_accuracy": 0.9248409140364289,
+      "num_tokens": 3346603.0,
+      "step": 2600
+    },
+    {
+      "epoch": 6.280701754385965,
+      "eval_entropy": 0.479939534647824,
+      "eval_loss": 0.5780320167541504,
+      "eval_mean_token_accuracy": 0.8957429552731448,
+      "eval_num_tokens": 3346603.0,
+      "eval_runtime": 46.6552,
+      "eval_samples_per_second": 6.216,
+      "eval_steps_per_second": 1.565,
+      "step": 2600
+    },
+    {
+      "entropy": 0.4207753050327301,
+      "epoch": 6.764670296430732,
+      "grad_norm": 0.2109375,
+      "learning_rate": 7.335518041369806e-05,
+      "loss": 0.36228099822998044,
+      "mean_token_accuracy": 0.924303879737854,
+      "num_tokens": 3605206.0,
+      "step": 2800
+    },
+    {
+      "epoch": 6.764670296430732,
+      "eval_entropy": 0.4856129292755911,
+      "eval_loss": 0.5810655355453491,
+      "eval_mean_token_accuracy": 0.8950460528674191,
+      "eval_num_tokens": 3605206.0,
+      "eval_runtime": 47.2855,
+      "eval_samples_per_second": 6.133,
+      "eval_steps_per_second": 1.544,
+      "step": 2800
+    },
+    {
+      "entropy": 0.4141107780421842,
+      "epoch": 7.246823956442832,
+      "grad_norm": 0.240234375,
+      "learning_rate": 6.786678349409785e-05,
+      "loss": 0.353370361328125,
+      "mean_token_accuracy": 0.9260965282076425,
+      "num_tokens": 3862259.0,
+      "step": 3000
+    },
+    {
+      "epoch": 7.246823956442832,
+      "eval_entropy": 0.47775410626032583,
+      "eval_loss": 0.5862641930580139,
+      "eval_mean_token_accuracy": 0.8956269344238386,
+      "eval_num_tokens": 3862259.0,
+      "eval_runtime": 47.1135,
+      "eval_samples_per_second": 6.155,
+      "eval_steps_per_second": 1.549,
+      "step": 3000
+    },
+    {
+      "entropy": 0.4134879305586219,
+      "epoch": 7.730792498487598,
+      "grad_norm": 0.212890625,
+      "learning_rate": 6.211884661314404e-05,
+      "loss": 0.3520899963378906,
+      "mean_token_accuracy": 0.9260335348546505,
+      "num_tokens": 4120020.0,
+      "step": 3200
+    },
+    {
+      "epoch": 7.730792498487598,
+      "eval_entropy": 0.47687137167747706,
+      "eval_loss": 0.5826675295829773,
+      "eval_mean_token_accuracy": 0.8959454404164667,
+      "eval_num_tokens": 4120020.0,
+      "eval_runtime": 46.9524,
+      "eval_samples_per_second": 6.176,
+      "eval_steps_per_second": 1.555,
+      "step": 3200
+    },
+    {
+      "entropy": 0.41237180912808763,
+      "epoch": 8.212946158499697,
+      "grad_norm": 0.23828125,
+      "learning_rate": 5.6194866574636726e-05,
+      "loss": 0.3477163314819336,
+      "mean_token_accuracy": 0.9267811967408194,
+      "num_tokens": 4377139.0,
+      "step": 3400
+    },
+    {
+      "epoch": 8.212946158499697,
+      "eval_entropy": 0.47165755008997984,
+      "eval_loss": 0.5873588919639587,
+      "eval_mean_token_accuracy": 0.8950836046101296,
+      "eval_num_tokens": 4377139.0,
+      "eval_runtime": 54.7647,
+      "eval_samples_per_second": 5.295,
+      "eval_steps_per_second": 1.333,
+      "step": 3400
+    },
+    {
+      "entropy": 0.40712683595716953,
+      "epoch": 8.696914700544465,
+      "grad_norm": 0.232421875,
+      "learning_rate": 5.018089745488805e-05,
+      "loss": 0.3462469863891602,
+      "mean_token_accuracy": 0.9266008900105953,
+      "num_tokens": 4635344.0,
+      "step": 3600
+    },
+    {
+      "epoch": 8.696914700544465,
+      "eval_entropy": 0.4733920579087244,
+      "eval_loss": 0.5913234353065491,
+      "eval_mean_token_accuracy": 0.895441471713863,
+      "eval_num_tokens": 4635344.0,
+      "eval_runtime": 98.6499,
+      "eval_samples_per_second": 2.94,
+      "eval_steps_per_second": 0.74,
+      "step": 3600
+    },
+    {
+      "entropy": 0.40435460647346083,
+      "epoch": 9.179068360556563,
+      "grad_norm": 0.2392578125,
+      "learning_rate": 4.416430054718493e-05,
+      "loss": 0.33988548278808595,
+      "mean_token_accuracy": 0.9284380353529151,
+      "num_tokens": 4891959.0,
+      "step": 3800
+    },
+    {
+      "epoch": 9.179068360556563,
+      "eval_entropy": 0.4687396479796057,
+      "eval_loss": 0.5922302603721619,
+      "eval_mean_token_accuracy": 0.8952091801656435,
+      "eval_num_tokens": 4891959.0,
+      "eval_runtime": 98.6504,
+      "eval_samples_per_second": 2.94,
+      "eval_steps_per_second": 0.74,
+      "step": 3800
+    },
+    {
+      "entropy": 0.40334540244191885,
+      "epoch": 9.66303690260133,
+      "grad_norm": 0.25,
+      "learning_rate": 3.823247531710108e-05,
+      "loss": 0.34033252716064455,
+      "mean_token_accuracy": 0.9283645600825549,
+      "num_tokens": 5150117.0,
+      "step": 4000
+    },
+    {
+      "epoch": 9.66303690260133,
+      "eval_entropy": 0.46922515271461174,
+      "eval_loss": 0.5946717262268066,
+      "eval_mean_token_accuracy": 0.8953593046697852,
+      "eval_num_tokens": 5150117.0,
+      "eval_runtime": 98.7836,
+      "eval_samples_per_second": 2.936,
+      "eval_steps_per_second": 0.739,
+      "step": 4000
+    },
+    {
+      "entropy": 0.39993977449469764,
+      "epoch": 10.14519056261343,
+      "grad_norm": 0.232421875,
+      "learning_rate": 3.247158980330338e-05,
+      "loss": 0.33406734466552734,
+      "mean_token_accuracy": 0.9284365516684137,
+      "num_tokens": 5406258.0,
+      "step": 4200
+    },
+    {
+      "epoch": 10.14519056261343,
+      "eval_entropy": 0.4659598641199608,
+      "eval_loss": 0.59522944688797,
+      "eval_mean_token_accuracy": 0.894354227471025,
+      "eval_num_tokens": 5406258.0,
+      "eval_runtime": 98.2511,
+      "eval_samples_per_second": 2.952,
+      "eval_steps_per_second": 0.743,
+      "step": 4200
+    },
+    {
+      "entropy": 0.4000014416128397,
+      "epoch": 10.629159104658196,
+      "grad_norm": 0.28125,
+      "learning_rate": 2.6965328906552368e-05,
+      "loss": 0.33594528198242185,
+      "mean_token_accuracy": 0.928432078063488,
+      "num_tokens": 5665214.0,
+      "step": 4400
+    },
+    {
+      "epoch": 10.629159104658196,
+      "eval_entropy": 0.4656533133493711,
+      "eval_loss": 0.5966197848320007,
+      "eval_mean_token_accuracy": 0.8958511099423447,
+      "eval_num_tokens": 5665214.0,
+      "eval_runtime": 97.708,
+      "eval_samples_per_second": 2.968,
+      "eval_steps_per_second": 0.747,
+      "step": 4400
+    },
+    {
+      "entropy": 0.39988665288335257,
+      "epoch": 11.111312764670297,
+      "grad_norm": 0.255859375,
+      "learning_rate": 2.1793678749745773e-05,
+      "loss": 0.33592803955078127,
+      "mean_token_accuracy": 0.9285333580623756,
+      "num_tokens": 5921976.0,
+      "step": 4600
+    },
+    {
+      "epoch": 11.111312764670297,
+      "eval_entropy": 0.4668905788088498,
+      "eval_loss": 0.595775306224823,
+      "eval_mean_token_accuracy": 0.895175910975835,
+      "eval_num_tokens": 5921976.0,
+      "eval_runtime": 98.0497,
+      "eval_samples_per_second": 2.958,
+      "eval_steps_per_second": 0.745,
+      "step": 4600
+    },
+    {
+      "entropy": 0.39747313819825647,
+      "epoch": 11.595281306715064,
+      "grad_norm": 0.265625,
+      "learning_rate": 1.703176476787164e-05,
+      "loss": 0.33493621826171877,
+      "mean_token_accuracy": 0.9288852622359991,
+      "num_tokens": 6180319.0,
+      "step": 4800
+    },
+    {
+      "epoch": 11.595281306715064,
+      "eval_entropy": 0.46607412214148536,
+      "eval_loss": 0.5940157175064087,
+      "eval_mean_token_accuracy": 0.8953437478574988,
+      "eval_num_tokens": 6180319.0,
+      "eval_runtime": 98.3503,
+      "eval_samples_per_second": 2.949,
+      "eval_steps_per_second": 0.742,
+      "step": 4800
+    },
+    {
+      "entropy": 0.3974911795655637,
+      "epoch": 12.077434966727163,
+      "grad_norm": 0.2333984375,
+      "learning_rate": 1.2748760406236098e-05,
+      "loss": 0.3305249786376953,
+      "mean_token_accuracy": 0.9298606089396937,
+      "num_tokens": 6436530.0,
+      "step": 5000
+    },
+    {
+      "epoch": 12.077434966727163,
+      "eval_entropy": 0.46489216652635024,
+      "eval_loss": 0.5976075530052185,
+      "eval_mean_token_accuracy": 0.8952453854965837,
+      "eval_num_tokens": 6436530.0,
+      "eval_runtime": 98.1183,
+      "eval_samples_per_second": 2.956,
+      "eval_steps_per_second": 0.744,
+      "step": 5000
+    },
+    {
+      "entropy": 0.39817900501191617,
+      "epoch": 12.56140350877193,
+      "grad_norm": 0.271484375,
+      "learning_rate": 9.006882279646356e-06,
+      "loss": 0.3339969635009766,
+      "mean_token_accuracy": 0.9290466893464326,
+      "num_tokens": 6694685.0,
+      "step": 5200
+    },
+    {
+      "epoch": 12.56140350877193,
+      "eval_entropy": 0.4657470506348022,
+      "eval_loss": 0.597287118434906,
+      "eval_mean_token_accuracy": 0.8953335562797442,
+      "eval_num_tokens": 6694685.0,
+      "eval_runtime": 47.8386,
+      "eval_samples_per_second": 6.062,
+      "eval_steps_per_second": 1.526,
+      "step": 5200
+    },
+    {
+      "entropy": 0.39675679298087374,
+      "epoch": 13.043557168784028,
+      "grad_norm": 0.25390625,
+      "learning_rate": 5.860486389264325e-06,
+      "loss": 0.3299782180786133,
+      "mean_token_accuracy": 0.9294995974417463,
+      "num_tokens": 6951487.0,
+      "step": 5400
+    },
+    {
+      "epoch": 13.043557168784028,
+      "eval_entropy": 0.4653073390052743,
+      "eval_loss": 0.5966233611106873,
+      "eval_mean_token_accuracy": 0.8951566096854536,
+      "eval_num_tokens": 6951487.0,
+      "eval_runtime": 98.2374,
+      "eval_samples_per_second": 2.952,
+      "eval_steps_per_second": 0.743,
+      "step": 5400
+    },
+    {
+      "entropy": 0.3993960711359978,
+      "epoch": 13.527525710828796,
+      "grad_norm": 0.26171875,
+      "learning_rate": 3.355278525842043e-06,
+      "loss": 0.33360870361328127,
+      "mean_token_accuracy": 0.9292359718680382,
+      "num_tokens": 7209994.0,
+      "step": 5600
+    },
+    {
+      "epoch": 13.527525710828796,
+      "eval_entropy": 0.46439201129625923,
+      "eval_loss": 0.596992552280426,
+      "eval_mean_token_accuracy": 0.8953333423562246,
+      "eval_num_tokens": 7209994.0,
+      "eval_runtime": 97.8986,
+      "eval_samples_per_second": 2.962,
+      "eval_steps_per_second": 0.746,
+      "step": 5600
+    },
+    {
+      "entropy": 0.3950923753053959,
+      "epoch": 14.009679370840896,
+      "grad_norm": 0.24609375,
+      "learning_rate": 1.5276503293336187e-06,
+      "loss": 0.3306077194213867,
+      "mean_token_accuracy": 0.9297496843218355,
+      "num_tokens": 7466401.0,
+      "step": 5800
+    },
+    {
+      "epoch": 14.009679370840896,
+      "eval_entropy": 0.4649810607302679,
+      "eval_loss": 0.5969449877738953,
+      "eval_mean_token_accuracy": 0.8953231295494184,
+      "eval_num_tokens": 7466401.0,
+      "eval_runtime": 97.4653,
+      "eval_samples_per_second": 2.975,
+      "eval_steps_per_second": 0.749,
+      "step": 5800
+    },
+    {
+      "entropy": 0.39669271767139436,
+      "epoch": 14.493647912885663,
+      "grad_norm": 0.2412109375,
+      "learning_rate": 4.0415064954446023e-07,
+      "loss": 0.33266185760498046,
+      "mean_token_accuracy": 0.929519795998931,
+      "num_tokens": 7724491.0,
+      "step": 6000
+    },
+    {
+      "epoch": 14.493647912885663,
+      "eval_entropy": 0.46521988633560807,
+      "eval_loss": 0.5968044400215149,
+      "eval_mean_token_accuracy": 0.8951437881548111,
+      "eval_num_tokens": 7724491.0,
+      "eval_runtime": 97.4586,
+      "eval_samples_per_second": 2.976,
+      "eval_steps_per_second": 0.749,
+      "step": 6000
+    },
+    {
+      "entropy": 0.3974524886161089,
+      "epoch": 14.977616454930429,
+      "grad_norm": 0.310546875,
+      "learning_rate": 1.099887041894343e-09,
+      "loss": 0.3300043487548828,
+      "mean_token_accuracy": 0.9291912470757961,
+      "num_tokens": 7982222.0,
+      "step": 6200
+    },
+    {
+      "epoch": 14.977616454930429,
+      "eval_entropy": 0.4640667352774372,
+      "eval_loss": 0.5965780019760132,
+      "eval_mean_token_accuracy": 0.8952341332827529,
+      "eval_num_tokens": 7982222.0,
+      "eval_runtime": 98.1877,
+      "eval_samples_per_second": 2.954,
+      "eval_steps_per_second": 0.743,
+      "step": 6200
+    },
+    {
+      "epoch": 15.0,
+      "eval_entropy": 0.4640667352774372,
+      "eval_loss": 0.5965780019760132,
+      "eval_mean_token_accuracy": 0.8952341332827529,
+      "eval_num_tokens": 7994175.0,
+      "eval_runtime": 97.9356,
+      "eval_samples_per_second": 2.961,
+      "eval_steps_per_second": 0.745,
+      "step": 6210
+    }
+  ],
+  "logging_steps": 200,
+  "max_steps": 6210,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 15,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.1728842516559872e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55c8b840e704687b5d8b477b3adbc0a714c1ea5538d61dd0840e331ed74563ce
+size 5240