---
library_name: peft
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B
tags:
- axolotl
- generated_from_trainer
model-index:
- name: 029f10e5-2608-4fa0-b8bb-cbdc2ba54829
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.4.1`
```yaml
adapter: lora
base_model: Qwen/Qwen2.5-1.5B
bf16: auto
dataset_prepared_path: null
datasets:
- data_files:
- b45999760ef35661_train_data.json
ds_type: json
format: custom
path: /root/G.O.D-test/core/data/b45999760ef35661_train_data.json
type:
field_input: system_prompt
field_instruction: problem
field_output: solution
format: '{instruction} {input}'
no_input_format: '{instruction}'
system_format: '{system}'
system_prompt: ''
debug: null
deepspeed: null
early_stopping_patience: null
eval_max_new_tokens: 128
eval_table_size: null
evals_per_epoch: 50
flash_attention: false
fp16: null
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 4
gradient_checkpointing: false
group_by_length: false
hub_model_id: souging/029f10e5-2608-4fa0-b8bb-cbdc2ba54829
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.0002
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 64
lora_dropout: 0.05
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 32
lora_target_linear: true
lr_scheduler: cosine
max_steps: 500
micro_batch_size: 3
mlflow_experiment_name: /tmp/b45999760ef35661_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 3
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
saves_per_epoch: 50
sequence_len: 1024
strict: false
tf32: false
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: 2fcab945-37d4-4262-b912-3bc4bdc46e98
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 2fcab945-37d4-4262-b912-3bc4bdc46e98
warmup_steps: 100
weight_decay: 0.01
xformers_attention: null
```
# 029f10e5-2608-4fa0-b8bb-cbdc2ba54829
This model is a fine-tuned version of [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5337
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 96
- total_eval_batch_size: 24
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 500
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 0.8119 | 0.0020 | 1 | 0.8480 |
| 0.8458 | 0.0081 | 4 | 0.8464 |
| 0.9139 | 0.0162 | 8 | 0.8185 |
| 0.7918 | 0.0243 | 12 | 0.7595 |
| 0.7346 | 0.0324 | 16 | 0.6953 |
| 0.7496 | 0.0405 | 20 | 0.6672 |
| 0.6417 | 0.0486 | 24 | 0.6474 |
| 0.6411 | 0.0567 | 28 | 0.6337 |
| 0.5976 | 0.0648 | 32 | 0.6236 |
| 0.6149 | 0.0729 | 36 | 0.6156 |
| 0.6517 | 0.0810 | 40 | 0.6091 |
| 0.578 | 0.0891 | 44 | 0.6025 |
| 0.5751 | 0.0972 | 48 | 0.5978 |
| 0.5774 | 0.1053 | 52 | 0.5933 |
| 0.5904 | 0.1134 | 56 | 0.5896 |
| 0.6291 | 0.1215 | 60 | 0.5862 |
| 0.6046 | 0.1296 | 64 | 0.5829 |
| 0.5971 | 0.1377 | 68 | 0.5801 |
| 0.5339 | 0.1457 | 72 | 0.5778 |
| 0.5838 | 0.1538 | 76 | 0.5762 |
| 0.5879 | 0.1619 | 80 | 0.5744 |
| 0.6243 | 0.1700 | 84 | 0.5727 |
| 0.5588 | 0.1781 | 88 | 0.5710 |
| 0.5582 | 0.1862 | 92 | 0.5703 |
| 0.4996 | 0.1943 | 96 | 0.5688 |
| 0.5783 | 0.2024 | 100 | 0.5679 |
| 0.5992 | 0.2105 | 104 | 0.5669 |
| 0.576 | 0.2186 | 108 | 0.5658 |
| 0.5396 | 0.2267 | 112 | 0.5652 |
| 0.5277 | 0.2348 | 116 | 0.5636 |
| 0.5383 | 0.2429 | 120 | 0.5633 |
| 0.5742 | 0.2510 | 124 | 0.5617 |
| 0.5488 | 0.2591 | 128 | 0.5610 |
| 0.5117 | 0.2672 | 132 | 0.5596 |
| 0.597 | 0.2753 | 136 | 0.5598 |
| 0.5429 | 0.2834 | 140 | 0.5586 |
| 0.5275 | 0.2915 | 144 | 0.5580 |
| 0.6284 | 0.2996 | 148 | 0.5569 |
| 0.5636 | 0.3077 | 152 | 0.5565 |
| 0.5918 | 0.3158 | 156 | 0.5560 |
| 0.5125 | 0.3239 | 160 | 0.5551 |
| 0.5762 | 0.3320 | 164 | 0.5548 |
| 0.5175 | 0.3401 | 168 | 0.5546 |
| 0.5223 | 0.3482 | 172 | 0.5540 |
| 0.6249 | 0.3563 | 176 | 0.5535 |
| 0.6143 | 0.3644 | 180 | 0.5531 |
| 0.5562 | 0.3725 | 184 | 0.5523 |
| 0.5256 | 0.3806 | 188 | 0.5519 |
| 0.5949 | 0.3887 | 192 | 0.5517 |
| 0.5851 | 0.3968 | 196 | 0.5515 |
| 0.5621 | 0.4049 | 200 | 0.5506 |
| 0.6014 | 0.4130 | 204 | 0.5499 |
| 0.6253 | 0.4211 | 208 | 0.5492 |
| 0.5418 | 0.4291 | 212 | 0.5492 |
| 0.5466 | 0.4372 | 216 | 0.5486 |
| 0.5654 | 0.4453 | 220 | 0.5484 |
| 0.519 | 0.4534 | 224 | 0.5480 |
| 0.5804 | 0.4615 | 228 | 0.5475 |
| 0.5876 | 0.4696 | 232 | 0.5470 |
| 0.5056 | 0.4777 | 236 | 0.5465 |
| 0.6075 | 0.4858 | 240 | 0.5466 |
| 0.5904 | 0.4939 | 244 | 0.5459 |
| 0.5132 | 0.5020 | 248 | 0.5459 |
| 0.5408 | 0.5101 | 252 | 0.5451 |
| 0.5664 | 0.5182 | 256 | 0.5449 |
| 0.5504 | 0.5263 | 260 | 0.5445 |
| 0.586 | 0.5344 | 264 | 0.5444 |
| 0.5636 | 0.5425 | 268 | 0.5437 |
| 0.5426 | 0.5506 | 272 | 0.5438 |
| 0.5506 | 0.5587 | 276 | 0.5432 |
| 0.5304 | 0.5668 | 280 | 0.5427 |
| 0.5274 | 0.5749 | 284 | 0.5425 |
| 0.5152 | 0.5830 | 288 | 0.5429 |
| 0.5268 | 0.5911 | 292 | 0.5418 |
| 0.5583 | 0.5992 | 296 | 0.5412 |
| 0.5721 | 0.6073 | 300 | 0.5413 |
| 0.5267 | 0.6154 | 304 | 0.5408 |
| 0.4977 | 0.6235 | 308 | 0.5404 |
| 0.5398 | 0.6316 | 312 | 0.5404 |
| 0.4885 | 0.6397 | 316 | 0.5399 |
| 0.4874 | 0.6478 | 320 | 0.5395 |
| 0.543 | 0.6559 | 324 | 0.5392 |
| 0.5303 | 0.6640 | 328 | 0.5388 |
| 0.4639 | 0.6721 | 332 | 0.5386 |
| 0.5597 | 0.6802 | 336 | 0.5383 |
| 0.5399 | 0.6883 | 340 | 0.5380 |
| 0.595 | 0.6964 | 344 | 0.5377 |
| 0.5646 | 0.7045 | 348 | 0.5375 |
| 0.5836 | 0.7126 | 352 | 0.5372 |
| 0.5705 | 0.7206 | 356 | 0.5372 |
| 0.5366 | 0.7287 | 360 | 0.5371 |
| 0.5268 | 0.7368 | 364 | 0.5370 |
| 0.5043 | 0.7449 | 368 | 0.5366 |
| 0.5235 | 0.7530 | 372 | 0.5365 |
| 0.5688 | 0.7611 | 376 | 0.5364 |
| 0.594 | 0.7692 | 380 | 0.5361 |
| 0.4947 | 0.7773 | 384 | 0.5360 |
| 0.5625 | 0.7854 | 388 | 0.5357 |
| 0.5373 | 0.7935 | 392 | 0.5357 |
| 0.5124 | 0.8016 | 396 | 0.5355 |
| 0.526 | 0.8097 | 400 | 0.5352 |
| 0.4984 | 0.8178 | 404 | 0.5351 |
| 0.5642 | 0.8259 | 408 | 0.5348 |
| 0.5353 | 0.8340 | 412 | 0.5347 |
| 0.4514 | 0.8421 | 416 | 0.5347 |
| 0.5735 | 0.8502 | 420 | 0.5345 |
| 0.5252 | 0.8583 | 424 | 0.5344 |
| 0.5355 | 0.8664 | 428 | 0.5343 |
| 0.4927 | 0.8745 | 432 | 0.5342 |
| 0.5029 | 0.8826 | 436 | 0.5341 |
| 0.547 | 0.8907 | 440 | 0.5340 |
| 0.5855 | 0.8988 | 444 | 0.5339 |
| 0.564 | 0.9069 | 448 | 0.5339 |
| 0.5686 | 0.9150 | 452 | 0.5339 |
| 0.5222 | 0.9231 | 456 | 0.5338 |
| 0.5158 | 0.9312 | 460 | 0.5337 |
| 0.523 | 0.9393 | 464 | 0.5337 |
| 0.564 | 0.9474 | 468 | 0.5337 |
| 0.4984 | 0.9555 | 472 | 0.5337 |
| 0.54 | 0.9636 | 476 | 0.5337 |
| 0.5355 | 0.9717 | 480 | 0.5337 |
| 0.5528 | 0.9798 | 484 | 0.5337 |
| 0.5192 | 0.9879 | 488 | 0.5336 |
| 0.4789 | 0.9960 | 492 | 0.5337 |
| 0.5134 | 1.0040 | 496 | 0.5336 |
| 0.5032 | 1.0121 | 500 | 0.5337 |
### Framework versions
- PEFT 0.13.2
- Transformers 4.46.0
- Pytorch 2.5.0+cu124
- Datasets 3.0.1
- Tokenizers 0.20.3