---
license: apache-2.0
datasets:
- meta-math/MetaMathQA
base_model:
- meta-llama/Llama-3.1-8B
pipeline_tag: text-generation
---
### Meta-Llama-3.1-Math-QA-finetuning-Group-3
This code uses meta-math/MetaMathQA dataset to fine-tune Meta-Llama-3.1 Large Language Model.

LoRA was utilized in order to significantly decrease training time. 

Random (seed = 42) 50.000 lines were selected from the database to be used in training. 

Unsloth framework allows the fine-tuning process to be more memory and time efficient.

Training hyperparameters:

```
  num_train_epochs = 5
  max_steps = 50
  learning_rate = 5e-5
  logging_steps = 1
  optim = "adamw_8bit"
  weight_decay = 0.001
  lr_scheduler_type = "linear"
  seed = 3407
```

- 50th Epoch training loss: 0.551400