--- license: mit library_name: peft tags: - trl - sft - generated_from_trainer metrics: - rouge base_model: TheBloke/zephyr-7B-beta-GPTQ model-index: - name: zephyr-Me results: [] --- # zephyr-Me This model is a fine-tuned version of [TheBloke/zephyr-7B-beta-GPTQ](https://huggingface.co/TheBloke/zephyr-7B-beta-GPTQ) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.1902 - Rouge1: 0.7317 - Rouge2: 0.5171 - Rougel: 0.6950 - Rougelsum: 0.7186 - Meteor: 0.7271 - F1 Score: 0.0012 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 16 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - num_epochs: 7 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Meteor | F1 Score | |:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:------:|:--------:| | 2.1246 | 0.15 | 4 | 2.0611 | 0.5115 | 0.1858 | 0.4164 | 0.4770 | 0.4928 | 0.0077 | | 1.6866 | 0.3 | 8 | 1.5126 | 0.6085 | 0.3404 | 0.5486 | 0.5946 | 0.5857 | 0.0010 | | 1.3126 | 0.44 | 12 | 1.3238 | 0.6578 | 0.4124 | 0.6046 | 0.6364 | 0.6452 | 0.0010 | | 1.1583 | 0.59 | 16 | 1.2688 | 0.6662 | 0.4166 | 0.6126 | 0.6434 | 0.6477 | 0.0008 | | 1.1011 | 0.74 | 20 | 1.1857 | 0.6776 | 0.4200 | 0.6254 | 0.6559 | 0.6568 | 0.0008 | | 1.0343 | 0.89 | 24 | 1.1439 | 0.6730 | 0.4174 | 0.6238 | 0.6512 | 0.6587 | 0.0008 | | 1.0673 | 1.04 | 28 | 1.1088 | 0.6922 | 0.4335 | 0.6360 | 0.6684 | 0.6753 | 0.0008 | | 0.8681 | 1.19 | 32 | 1.1131 | 0.6992 | 0.4523 | 0.6488 | 0.6769 | 0.6862 | 0.0010 | | 0.8605 | 1.33 | 36 | 1.0888 | 0.7064 | 0.4606 | 0.6585 | 0.6873 | 0.6958 | 0.0010 | | 0.8603 | 1.48 | 40 | 1.0543 | 0.7074 | 0.4638 | 0.6582 | 0.6869 | 0.6894 | 0.0010 | | 0.8354 | 1.63 | 44 | 1.0298 | 0.7050 | 0.4631 | 0.6586 | 0.6856 | 0.6840 | 0.0010 | | 0.7906 | 1.78 | 48 | 1.0037 | 0.7084 | 0.4712 | 0.6642 | 0.6902 | 0.6887 | 0.0010 | | 0.7108 | 1.93 | 52 | 0.9887 | 0.7113 | 0.4766 | 0.6658 | 0.6920 | 0.7017 | 0.0010 | | 0.7009 | 2.07 | 56 | 0.9943 | 0.7109 | 0.4787 | 0.6679 | 0.6925 | 0.7025 | 0.0010 | | 0.5753 | 2.22 | 60 | 0.9924 | 0.7235 | 0.4846 | 0.6741 | 0.7042 | 0.7064 | 0.0012 | | 0.5952 | 2.37 | 64 | 0.9602 | 0.7219 | 0.4928 | 0.6818 | 0.7066 | 0.7135 | 0.0010 | | 0.5637 | 2.52 | 68 | 0.9766 | 0.7168 | 0.4924 | 0.6807 | 0.7011 | 0.7109 | 0.0010 | | 0.5426 | 2.67 | 72 | 0.9713 | 0.7218 | 0.4944 | 0.6823 | 0.7065 | 0.7094 | 0.0010 | | 0.5503 | 2.81 | 76 | 0.9860 | 0.7183 | 0.4937 | 0.6805 | 0.7049 | 0.7118 | 0.0010 | | 0.4929 | 2.96 | 80 | 0.9701 | 0.7229 | 0.4987 | 0.6849 | 0.7076 | 0.7196 | 0.0010 | | 0.4505 | 3.11 | 84 | 0.9851 | 0.7209 | 0.5006 | 0.6871 | 0.7085 | 0.7173 | 0.0010 | | 0.4012 | 3.26 | 88 | 1.0122 | 0.7267 | 0.5080 | 0.6914 | 0.7115 | 0.7264 | 0.0010 | | 0.3709 | 3.41 | 92 | 0.9842 | 0.7326 | 0.5125 | 0.6941 | 0.7162 | 0.7294 | 0.0010 | | 0.3962 | 3.56 | 96 | 1.0035 | 0.7337 | 0.5154 | 0.6972 | 0.7201 | 0.7293 | 0.0010 | | 0.4 | 3.7 | 100 | 1.0013 | 0.7306 | 0.5143 | 0.6951 | 0.7179 | 0.7318 | 0.0010 | | 0.3689 | 3.85 | 104 | 0.9994 | 0.7315 | 0.5113 | 0.6936 | 0.7180 | 0.7294 | 0.0010 | | 0.3805 | 4.0 | 108 | 1.0023 | 0.7418 | 0.5219 | 0.7029 | 0.7279 | 0.7353 | 0.0010 | | 0.3162 | 4.15 | 112 | 1.0605 | 0.7406 | 0.5222 | 0.7023 | 0.7250 | 0.7380 | 0.0010 | | 0.2983 | 4.3 | 116 | 1.0506 | 0.7410 | 0.5240 | 0.7042 | 0.7272 | 0.7370 | 0.0010 | | 0.2766 | 4.44 | 120 | 1.0229 | 0.7384 | 0.5199 | 0.7048 | 0.7249 | 0.7361 | 0.0010 | | 0.2413 | 4.59 | 124 | 1.0696 | 0.7372 | 0.5198 | 0.7044 | 0.7251 | 0.7366 | 0.0010 | | 0.268 | 4.74 | 128 | 1.0809 | 0.7351 | 0.5153 | 0.6998 | 0.7222 | 0.7320 | 0.0010 | | 0.2555 | 4.89 | 132 | 1.0408 | 0.7357 | 0.5211 | 0.7011 | 0.7215 | 0.7332 | 0.0010 | | 0.2579 | 5.04 | 136 | 1.0279 | 0.7361 | 0.5178 | 0.6986 | 0.7215 | 0.7324 | 0.0010 | | 0.2071 | 5.19 | 140 | 1.0943 | 0.7380 | 0.5186 | 0.7001 | 0.7240 | 0.7357 | 0.0012 | | 0.2076 | 5.33 | 144 | 1.1801 | 0.7342 | 0.5154 | 0.6983 | 0.7220 | 0.7275 | 0.0012 | | 0.2211 | 5.48 | 148 | 1.1742 | 0.7334 | 0.5144 | 0.6960 | 0.7197 | 0.7273 | 0.0012 | | 0.1828 | 5.63 | 152 | 1.1487 | 0.7350 | 0.5180 | 0.6981 | 0.7213 | 0.7301 | 0.0012 | | 0.2016 | 5.78 | 156 | 1.1384 | 0.7373 | 0.5195 | 0.7000 | 0.7234 | 0.7314 | 0.0012 | | 0.2279 | 5.93 | 160 | 1.1391 | 0.7390 | 0.5228 | 0.7021 | 0.7252 | 0.7332 | 0.0012 | | 0.2045 | 6.07 | 164 | 1.1427 | 0.7378 | 0.5208 | 0.7014 | 0.7246 | 0.7323 | 0.0012 | | 0.1642 | 6.22 | 168 | 1.1598 | 0.7357 | 0.5188 | 0.6996 | 0.7227 | 0.7310 | 0.0012 | | 0.1858 | 6.37 | 172 | 1.1764 | 0.7341 | 0.5183 | 0.6980 | 0.7211 | 0.7288 | 0.0012 | | 0.1869 | 6.52 | 176 | 1.1846 | 0.7317 | 0.5162 | 0.6954 | 0.7184 | 0.7269 | 0.0012 | | 0.1754 | 6.67 | 180 | 1.1886 | 0.7324 | 0.5175 | 0.6955 | 0.7192 | 0.7273 | 0.0012 | | 0.1634 | 6.81 | 184 | 1.1898 | 0.7317 | 0.5171 | 0.6950 | 0.7186 | 0.7271 | 0.0012 | | 0.1853 | 6.96 | 188 | 1.1902 | 0.7317 | 0.5171 | 0.6950 | 0.7186 | 0.7271 | 0.0012 | ### Framework versions - PEFT 0.7.1 - Transformers 4.35.2 - Pytorch 2.1.0+cu121 - Datasets 2.16.0 - Tokenizers 0.15.0