yhavinga commited on
Commit
ff33ed6
·
verified ·
1 Parent(s): 0faf64e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +20 -9
README.md CHANGED
@@ -89,26 +89,37 @@ optimizer: AdamW (β₁=0.9, β₂=0.999, ε=1e-8)
89
  neftune_noise_alpha: 5
90
  per_device_train_batch_size: 32
91
  gradient_accumulation_steps: 2
92
-
93
- # Steps & Efficiency
94
- total_training_steps: 2,960
95
  packing_enabled: true (~5x efficiency gain)
96
  eval_steps: 100
97
  save_steps: 200
98
  logging_steps: 10
99
-
100
- # Final Metrics
101
- final_train_loss: 1.127
102
- final_eval_loss: 1.041
103
  ```
104
 
105
  ### Hardware & Distributed Training
106
  - **GPUs**: 4× AMD MI300X (192GB total VRAM)
107
- - **Training Time**: 6 hours 47 minutes (2.0 epochs, 2,960 steps)
108
  - **FSDP**: Full sharding with activation checkpointing
109
- - **Peak Memory**: 71GB per GPU
110
  - **Effective Batch Size**: 256 (32 × 4 GPUs × 2 gradient accumulation)
111
  - **Packing**: Enabled for efficient sequence batching
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
112
 
113
  ### Dataset Engineering
114
 
 
89
  neftune_noise_alpha: 5
90
  per_device_train_batch_size: 32
91
  gradient_accumulation_steps: 2
 
 
 
92
  packing_enabled: true (~5x efficiency gain)
93
  eval_steps: 100
94
  save_steps: 200
95
  logging_steps: 10
 
 
 
 
96
  ```
97
 
98
  ### Hardware & Distributed Training
99
  - **GPUs**: 4× AMD MI300X (192GB total VRAM)
 
100
  - **FSDP**: Full sharding with activation checkpointing
 
101
  - **Effective Batch Size**: 256 (32 × 4 GPUs × 2 gradient accumulation)
102
  - **Packing**: Enabled for efficient sequence batching
103
+ - **Total Training Steps**: 2,960
104
+ - **Training Time**: 6 hours 47 minutes (2.0 epochs, 2,960 steps)
105
+ - **Peak Memory**: 71GB per GPU
106
+
107
+
108
+ ### Training Progress
109
+
110
+ The following charts show the training dynamics over 2,960 steps (2 epochs with sequence packing):
111
+
112
+ ![Training Loss](./train_loss_noot_eurollm.png)
113
+ *Training loss showing rapid initial convergence from 2.0 to ~1.3 within 500 steps, stabilizing at 1.127*
114
+
115
+ ![Evaluation Loss](./eval_loss_noot_eurollm.png)
116
+ *Evaluation loss measured every 100 steps, final value: 1.041*
117
+
118
+ **Key observations:**
119
+ - Rapid convergence in first 500 steps due to effective initialization from base model
120
+ - Smooth cosine learning rate schedule with 200 warmup steps
121
+ - Consistent improvement throughout both epochs
122
+ - Stable training with no signs of overfitting (eval loss < train loss)
123
 
124
  ### Dataset Engineering
125