Hugging Face's logo

besimray
/
miner_id_1_383a850e-bb15-45a2-8f4b-fc96eb001a74_1729770655

PEFT
Safetensors
llama
axolotl
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files
xet
 Community
besimray commited on
Commit
18c1790
·
verified ·
1 Parent(s): ed93eb9

End of training

Browse files
Files changed (3) hide show
  1. README.md +182 -0
  2. adapter_model.bin +3 -0
  3. adapter_model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,182 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: llama3.2
4
+ base_model: unsloth/Llama-3.2-1B-Instruct
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: miner_id_1_383a850e-bb15-45a2-8f4b-fc96eb001a74_1729770655
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.4.1`
20
+ ```yaml
21
+ adapter: lora
22
+ base_model: unsloth/Llama-3.2-1B-Instruct
23
+ bf16: auto
24
+ chat_template: llama3
25
+ dataset_prepared_path: null
26
+ datasets:
27
+ - path: mhenrichsen/alpaca_2k_test
28
+ type: alpaca
29
+ debug: null
30
+ deepspeed: null
31
+ early_stopping_patience: 3
32
+ eval_max_new_tokens: 128
33
+ eval_steps: 5
34
+ eval_table_size: null
35
+ flash_attention: true
36
+ fp16: null
37
+ fsdp: null
38
+ fsdp_config: null
39
+ gradient_accumulation_steps: 1
40
+ gradient_checkpointing: true
41
+ group_by_length: false
42
+ hub_model_id: besimray/miner_id_1_383a850e-bb15-45a2-8f4b-fc96eb001a74_1729770655
43
+ hub_strategy: checkpoint
44
+ hub_token: null
45
+ learning_rate: 2.0e-05
46
+ load_in_4bit: false
47
+ load_in_8bit: true
48
+ local_rank: null
49
+ logging_steps: 1
50
+ lora_alpha: 16
51
+ lora_dropout: 0.05
52
+ lora_fan_in_fan_out: null
53
+ lora_model_dir: null
54
+ lora_r: 8
55
+ lora_target_linear: true
56
+ lr_scheduler: cosine
57
+ max_steps: 10000
58
+ micro_batch_size: 10
59
+ mlflow_experiment_name: mhenrichsen/alpaca_2k_test
60
+ model_type: LlamaForCausalLM
61
+ num_epochs: 5
62
+ optimizer: adamw_bnb_8bit
63
+ output_dir: miner_id_besimray
64
+ pad_to_sequence_len: true
65
+ resume_from_checkpoint: null
66
+ s2_attention: null
67
+ sample_packing: false
68
+ save_steps: 5
69
+ save_strategy: steps
70
+ sequence_len: 4096
71
+ strict: false
72
+ tf32: false
73
+ tokenizer_type: AutoTokenizer
74
+ train_on_inputs: false
75
+ val_set_size: 0.05
76
+ wandb_entity: besimray24-rayon
77
+ wandb_mode: online
78
+ wandb_project: Public_TuningSN
79
+ wandb_run: miner_id_24
80
+ wandb_runid: 383a850e-bb15-45a2-8f4b-fc96eb001a74
81
+ warmup_steps: 10
82
+ weight_decay: 0.0
83
+ xformers_attention: null
84
+
85
+ ```
86
+
87
+ </details><br>
88
+
89
+ # miner_id_1_383a850e-bb15-45a2-8f4b-fc96eb001a74_1729770655
90
+
91
+ This model is a fine-tuned version of [unsloth/Llama-3.2-1B-Instruct](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct) on the None dataset.
92
+ It achieves the following results on the evaluation set:
93
+ - Loss: 1.1623
94
+
95
+ ## Model description
96
+
97
+ More information needed
98
+
99
+ ## Intended uses & limitations
100
+
101
+ More information needed
102
+
103
+ ## Training and evaluation data
104
+
105
+ More information needed
106
+
107
+ ## Training procedure
108
+
109
+ ### Training hyperparameters
110
+
111
+ The following hyperparameters were used during training:
112
+ - learning_rate: 2e-05
113
+ - train_batch_size: 10
114
+ - eval_batch_size: 10
115
+ - seed: 42
116
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
117
+ - lr_scheduler_type: cosine
118
+ - lr_scheduler_warmup_steps: 10
119
+ - training_steps: 950
120
+
121
+ ### Training results
122
+
123
+ | Training Loss | Epoch | Step | Validation Loss |
124
+ |:-------------:|:------:|:----:|:---------------:|
125
+ | 1.3316 | 0.0053 | 1 | 1.2586 |
126
+ | 1.1351 | 0.0263 | 5 | 1.2596 |
127
+ | 1.2604 | 0.0526 | 10 | 1.2566 |
128
+ | 1.5396 | 0.0789 | 15 | 1.2454 |
129
+ | 1.4895 | 0.1053 | 20 | 1.2336 |
130
+ | 1.1625 | 0.1316 | 25 | 1.2236 |
131
+ | 1.3554 | 0.1579 | 30 | 1.2150 |
132
+ | 1.3275 | 0.1842 | 35 | 1.2100 |
133
+ | 1.1912 | 0.2105 | 40 | 1.2058 |
134
+ | 1.2335 | 0.2368 | 45 | 1.2030 |
135
+ | 1.0253 | 0.2632 | 50 | 1.1979 |
136
+ | 1.1242 | 0.2895 | 55 | 1.1970 |
137
+ | 0.9963 | 0.3158 | 60 | 1.1910 |
138
+ | 1.0977 | 0.3421 | 65 | 1.1919 |
139
+ | 1.1263 | 0.3684 | 70 | 1.1880 |
140
+ | 1.2144 | 0.3947 | 75 | 1.1860 |
141
+ | 1.3055 | 0.4211 | 80 | 1.1839 |
142
+ | 1.1513 | 0.4474 | 85 | 1.1818 |
143
+ | 1.0702 | 0.4737 | 90 | 1.1819 |
144
+ | 1.2561 | 0.5 | 95 | 1.1797 |
145
+ | 1.1373 | 0.5263 | 100 | 1.1775 |
146
+ | 1.2136 | 0.5526 | 105 | 1.1780 |
147
+ | 1.3591 | 0.5789 | 110 | 1.1771 |
148
+ | 1.5703 | 0.6053 | 115 | 1.1744 |
149
+ | 1.1601 | 0.6316 | 120 | 1.1754 |
150
+ | 1.1412 | 0.6579 | 125 | 1.1748 |
151
+ | 1.1449 | 0.6842 | 130 | 1.1731 |
152
+ | 1.1706 | 0.7105 | 135 | 1.1736 |
153
+ | 1.0503 | 0.7368 | 140 | 1.1730 |
154
+ | 1.1938 | 0.7632 | 145 | 1.1730 |
155
+ | 1.4802 | 0.7895 | 150 | 1.1710 |
156
+ | 1.1359 | 0.8158 | 155 | 1.1688 |
157
+ | 1.3575 | 0.8421 | 160 | 1.1709 |
158
+ | 1.0188 | 0.8684 | 165 | 1.1685 |
159
+ | 1.147 | 0.8947 | 170 | 1.1684 |
160
+ | 0.9949 | 0.9211 | 175 | 1.1668 |
161
+ | 1.3082 | 0.9474 | 180 | 1.1673 |
162
+ | 1.1995 | 0.9737 | 185 | 1.1654 |
163
+ | 1.2346 | 1.0 | 190 | 1.1654 |
164
+ | 1.0948 | 1.0263 | 195 | 1.1660 |
165
+ | 1.3838 | 1.0526 | 200 | 1.1643 |
166
+ | 0.9594 | 1.0789 | 205 | 1.1644 |
167
+ | 1.1423 | 1.1053 | 210 | 1.1635 |
168
+ | 1.1774 | 1.1316 | 215 | 1.1645 |
169
+ | 1.0085 | 1.1579 | 220 | 1.1642 |
170
+ | 1.0912 | 1.1842 | 225 | 1.1611 |
171
+ | 1.193 | 1.2105 | 230 | 1.1627 |
172
+ | 1.2437 | 1.2368 | 235 | 1.1640 |
173
+ | 1.1814 | 1.2632 | 240 | 1.1623 |
174
+
175
+
176
+ ### Framework versions
177
+
178
+ - PEFT 0.13.2
179
+ - Transformers 4.45.2
180
+ - Pytorch 2.3.1+cu121
181
+ - Datasets 3.0.1
182
+ - Tokenizers 0.20.1
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:918c7cb4aac5604877060d5a22a770436a6d2874b946e516d8689f4d8996e95d
3
+ size 22624970
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:eea176b2890f5c039b27382d7c55bd1ece54713333c0ee47f4372f34e426564c
3
  size 22573704
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:676d4ab8c76c93e64d02aab93625bc6aef38f8777aa2ac44e788c4c1c4138494
3
  size 22573704