The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: fineGrained).
The token `llm-2025-sahara` has been saved to /home/Competition2025/P06/shareP06/cache/stored_tokens
Your token has been saved to /home/Competition2025/P06/shareP06/cache/token
Login successful.
Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured.
wandb: Currently logged in as: kumapo (llm-2025-sahara) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
W0729 01:12:32.186000 449054 site-packages/torch/distributed/run.py:792] 
W0729 01:12:32.186000 449054 site-packages/torch/distributed/run.py:792] *****************************************
W0729 01:12:32.186000 449054 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0729 01:12:32.186000 449054 site-packages/torch/distributed/run.py:792] *****************************************
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/conda/lib/python3.11/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  76%|███████▋  | 13/17 [00:00<00:00, 124.04it/s]
Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 121.58it/s]

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 179.65it/s]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 180.02it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 181.49it/s]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards:  82%|████████▏ | 14/17 [00:00<00:00, 131.44it/s]
Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 136.04it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]

Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 211.62it/s]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 201.50it/s]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/Competition2025/P06/P06U023/deps/verl/verl/utils/dataset/sft_dataset.py:81: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 17/17 [00:00<00:00, 196.64it/s]
wandb: Currently logged in as: kumapo (llm-2025-sahara) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.21.0
wandb: Run data is saved locally in /home/Competition2025/P06/P06U023/llm-bridge-sahara/wandb/run-20250729_011348-eo9ykh5w
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run qwen3_32b_omr_peft-300801
wandb: ⭐️ View project at https://wandb.ai/llm-2025-sahara/qwen3_32b_omr_peft
wandb: 🚀 View run at https://wandb.ai/llm-2025-sahara/qwen3_32b_omr_peft/runs/eo9ykh5w

Epoch 1/1:   0%|          | 0/153 [00:00<?, ?it/s]
Epoch 1/1:   1%|          | 1/153 [06:18<15:58:26, 378.33s/it]
Epoch 1/1:   1%|▏         | 2/153 [12:29<15:40:47, 373.82s/it]
Epoch 1/1:   2%|▏         | 3/153 [18:51<15:44:46, 377.91s/it]
Epoch 1/1:   3%|▎         | 4/153 [25:10<15:39:10, 378.19s/it]
Epoch 1/1:   3%|▎         | 5/153 [31:25<15:29:59, 377.02s/it]
Epoch 1/1:   4%|▍         | 6/153 [37:55<15:34:56, 381.61s/it]
Epoch 1/1:   5%|▍         | 7/153 [44:07<15:20:57, 378.48s/it]
Epoch 1/1:   5%|▌         | 8/153 [50:33<15:19:57, 380.68s/it]
Epoch 1/1:   6%|▌         | 9/153 [57:08<15:24:42, 385.29s/it]
Epoch 1/1:   7%|▋         | 10/153 [1:03:19<15:07:51, 380.92s/it]
Epoch 1/1:   7%|▋         | 11/153 [1:09:37<14:59:21, 380.01s/it]
Epoch 1/1:   8%|▊         | 12/153 [1:15:54<14:50:47, 379.06s/it]
Epoch 1/1:   8%|▊         | 13/153 [1:22:25<14:53:01, 382.72s/it]
Epoch 1/1:   9%|▉         | 14/153 [1:28:36<14:37:54, 378.95s/it]
Epoch 1/1:  10%|▉         | 15/153 [1:34:55<14:31:49, 379.06s/it]
Epoch 1/1:  10%|█         | 16/153 [1:41:09<14:21:47, 377.43s/it]
Epoch 1/1:  11%|█         | 17/153 [1:47:43<14:26:49, 382.42s/it]
Epoch 1/1:  12%|█▏        | 18/153 [1:53:57<14:14:59, 379.99s/it]
Epoch 1/1:  12%|█▏        | 19/153 [2:00:19<14:09:48, 380.51s/it]
Epoch 1/1:  13%|█▎        | 20/153 [2:06:46<14:08:07, 382.62s/it]
Epoch 1/1:  14%|█▎        | 21/153 [2:13:11<14:03:22, 383.35s/it]
Epoch 1/1:  14%|█▍        | 22/153 [2:20:04<14:16:34, 392.32s/it]
Epoch 1/1:  15%|█▌        | 23/153 [2:26:24<14:01:34, 388.42s/it]
Epoch 1/1:  16%|█▌        | 24/153 [2:32:38<13:46:03, 384.22s/it]
Epoch 1/1:  16%|█▋        | 25/153 [2:38:36<13:22:47, 376.31s/it]
Epoch 1/1:  17%|█▋        | 26/153 [2:45:00<13:21:24, 378.62s/it]
Epoch 1/1:  18%|█▊        | 27/153 [2:51:22<13:17:21, 379.70s/it]
Epoch 1/1:  18%|█▊        | 28/153 [2:57:18<12:56:17, 372.62s/it]
Epoch 1/1:  19%|█▉        | 29/153 [3:03:52<13:03:20, 379.03s/it]
Epoch 1/1:  20%|█▉        | 30/153 [3:10:19<13:01:38, 381.29s/it]
Epoch 1/1:  20%|██        | 31/153 [3:16:35<12:52:09, 379.75s/it]
Epoch 1/1:  21%|██        | 32/153 [3:22:44<12:39:07, 376.43s/it]
Epoch 1/1:  22%|██▏       | 33/153 [3:29:02<12:33:48, 376.90s/it]
Epoch 1/1:  22%|██▏       | 34/153 [3:35:09<12:21:41, 373.96s/it]
Epoch 1/1:  23%|██▎       | 35/153 [3:41:31<12:20:33, 376.56s/it]
Epoch 1/1:  24%|██▎       | 36/153 [3:47:46<12:13:08, 375.97s/it]
Epoch 1/1:  24%|██▍       | 37/153 [3:53:59<12:05:01, 375.01s/it]
Epoch 1/1:  25%|██▍       | 38/153 [4:00:22<12:03:20, 377.40s/it]
Epoch 1/1:  25%|██▌       | 39/153 [4:06:33<11:53:26, 375.49s/it]
Epoch 1/1:  26%|██▌       | 40/153 [4:12:39<11:42:08, 372.82s/it]
Epoch 1/1:  27%|██▋       | 41/153 [4:19:03<11:41:55, 376.03s/it]
Epoch 1/1:  27%|██▋       | 42/153 [4:25:17<11:34:31, 375.42s/it]
Epoch 1/1:  28%|██▊       | 43/153 [4:31:36<11:30:04, 376.40s/it]
Epoch 1/1:  29%|██▉       | 44/153 [4:37:53<11:24:12, 376.63s/it]
Epoch 1/1:  29%|██▉       | 45/153 [4:44:24<11:25:49, 381.01s/it]
Epoch 1/1:  30%|███       | 46/153 [4:50:48<11:21:10, 381.97s/it]
Epoch 1/1:  31%|███       | 47/153 [4:57:02<11:10:17, 379.41s/it]
Epoch 1/1:  31%|███▏      | 48/153 [5:03:22<11:04:25, 379.67s/it]
Epoch 1/1:  32%|███▏      | 49/153 [5:09:42<10:58:23, 379.84s/it]
Epoch 1/1:  33%|███▎      | 50/153 [5:16:01<10:51:29, 379.51s/it]
Epoch 1/1:  33%|███▎      | 51/153 [5:22:16<10:42:53, 378.17s/it]
Epoch 1/1:  34%|███▍      | 52/153 [5:28:29<10:33:56, 376.60s/it]
Epoch 1/1:  35%|███▍      | 53/153 [5:34:32<10:21:02, 372.62s/it]
Epoch 1/1:  35%|███▌      | 54/153 [5:41:04<10:24:24, 378.43s/it]
Epoch 1/1:  36%|███▌      | 55/153 [5:47:28<10:20:55, 380.16s/it]
Epoch 1/1:  37%|███▋      | 56/153 [5:53:52<10:16:21, 381.25s/it]
Epoch 1/1:  37%|███▋      | 57/153 [6:00:11<10:08:44, 380.47s/it]
Epoch 1/1:  38%|███▊      | 58/153 [6:06:27<10:00:18, 379.15s/it]
Epoch 1/1:  39%|███▊      | 59/153 [6:12:51<9:56:16, 380.60s/it] 
Epoch 1/1:  39%|███▉      | 60/153 [6:19:11<9:49:47, 380.51s/it]
Epoch 1/1:  40%|███▉      | 61/153 [6:25:27<9:41:25, 379.19s/it]
Epoch 1/1:  41%|████      | 62/153 [6:31:49<9:36:05, 379.85s/it]
Epoch 1/1:  41%|████      | 63/153 [6:37:49<9:20:59, 373.99s/it]
Epoch 1/1:  42%|████▏     | 64/153 [6:44:09<9:17:31, 375.86s/it]
Epoch 1/1:  42%|████▏     | 65/153 [6:50:24<9:10:55, 375.63s/it]
Epoch 1/1:  43%|████▎     | 66/153 [6:56:47<9:07:39, 377.70s/it]
Epoch 1/1:  44%|████▍     | 67/153 [7:03:10<9:03:35, 379.25s/it]
Epoch 1/1:  44%|████▍     | 68/153 [7:09:20<8:53:31, 376.61s/it]
Epoch 1/1:  45%|████▌     | 69/153 [7:15:29<8:43:54, 374.22s/it]
Epoch 1/1:  46%|████▌     | 70/153 [7:21:55<8:42:31, 377.73s/it]
Epoch 1/1:  46%|████▋     | 71/153 [7:28:15<8:37:24, 378.60s/it]
Epoch 1/1:  47%|████▋     | 72/153 [7:34:30<8:29:37, 377.50s/it]
Epoch 1/1:  48%|████▊     | 73/153 [7:40:49<8:23:59, 377.99s/it]
Epoch 1/1:  48%|████▊     | 74/153 [7:47:04<8:16:23, 377.00s/it]
Epoch 1/1:  49%|████▉     | 75/153 [7:53:12<8:06:30, 374.23s/it]
Epoch 1/1:  50%|████▉     | 76/153 [7:59:33<8:03:03, 376.41s/it]
Epoch 1/1:  50%|█████     | 77/153 [8:05:59<8:00:08, 379.06s/it]
Epoch 1/1:  51%|█████     | 78/153 [8:12:15<7:52:42, 378.17s/it]
Epoch 1/1:  52%|█████▏    | 79/153 [8:18:51<7:53:10, 383.66s/it]
Epoch 1/1:  52%|█████▏    | 80/153 [8:24:56<7:39:42, 377.85s/it]
Epoch 1/1:  53%|█████▎    | 81/153 [8:31:00<7:28:43, 373.94s/it]
Epoch 1/1:  54%|█████▎    | 82/153 [8:37:16<7:23:16, 374.59s/it]
Epoch 1/1:  54%|█████▍    | 83/153 [8:43:23<7:14:20, 372.29s/it]
Epoch 1/1:  55%|█████▍    | 84/153 [8:49:23<7:03:38, 368.38s/it]
Epoch 1/1:  56%|█████▌    | 85/153 [8:55:38<6:59:51, 370.46s/it]
Epoch 1/1:  56%|█████▌    | 86/153 [9:02:00<6:57:42, 374.06s/it]
Epoch 1/1:  57%|█████▋    | 87/153 [9:08:20<6:53:08, 375.59s/it]
Epoch 1/1:  58%|█████▊    | 88/153 [9:14:26<6:43:59, 372.91s/it]
Epoch 1/1:  58%|█████▊    | 89/153 [9:20:48<6:40:43, 375.69s/it]
Epoch 1/1:  59%|█████▉    | 90/153 [9:27:02<6:33:44, 374.98s/it]
Epoch 1/1:  59%|█████▉    | 91/153 [9:33:15<6:26:56, 374.47s/it]
Epoch 1/1:  60%|██████    | 92/153 [9:39:34<6:21:59, 375.73s/it]
Epoch 1/1:  61%|██████    | 93/153 [9:45:42<6:13:34, 373.57s/it]
Epoch 1/1:  61%|██████▏   | 94/153 [9:51:54<6:06:52, 373.09s/it]
Epoch 1/1:  62%|██████▏   | 95/153 [9:58:10<6:01:29, 373.96s/it]
Epoch 1/1:  63%|██████▎   | 96/153 [10:04:27<5:55:59, 374.73s/it]
Epoch 1/1:  63%|██████▎   | 97/153 [10:10:30<5:46:25, 371.17s/it]
Epoch 1/1:  64%|██████▍   | 98/153 [10:16:53<5:43:43, 374.96s/it]
Epoch 1/1:  65%|██████▍   | 99/153 [10:23:02<5:35:43, 373.04s/it]
Epoch 1/1:  65%|██████▌   | 100/153 [10:29:19<5:30:39, 374.33s/it]
Epoch 1/1:  66%|██████▌   | 101/153 [10:35:39<5:25:54, 376.04s/it]
Epoch 1/1:  67%|██████▋   | 102/153 [10:42:12<5:23:46, 380.92s/it]
Epoch 1/1:  67%|██████▋   | 103/153 [10:48:45<5:20:39, 384.78s/it]
Epoch 1/1:  68%|██████▊   | 104/153 [10:54:58<5:11:11, 381.05s/it]
Epoch 1/1:  69%|██████▊   | 105/153 [11:01:25<5:06:18, 382.89s/it]
Epoch 1/1:  69%|██████▉   | 106/153 [11:07:41<4:58:17, 380.80s/it]
Epoch 1/1:  70%|██████▉   | 107/153 [11:14:12<4:54:15, 383.81s/it]
Epoch 1/1:  71%|███████   | 108/153 [11:20:26<4:45:45, 381.01s/it]
Epoch 1/1:  71%|███████   | 109/153 [11:26:45<4:38:55, 380.35s/it]
Epoch 1/1:  72%|███████▏  | 110/153 [11:33:01<4:31:36, 378.99s/it]
Epoch 1/1:  73%|███████▎  | 111/153 [11:39:32<4:27:53, 382.71s/it]
Epoch 1/1:  73%|███████▎  | 112/153 [11:45:53<4:21:06, 382.10s/it]
Epoch 1/1:  74%|███████▍  | 113/153 [11:52:23<4:16:21, 384.54s/it]
Epoch 1/1:  75%|███████▍  | 114/153 [11:58:49<4:10:13, 384.95s/it]
Epoch 1/1:  75%|███████▌  | 115/153 [12:05:10<4:02:58, 383.63s/it]
Epoch 1/1:  76%|███████▌  | 116/153 [12:11:15<3:53:09, 378.10s/it]
Epoch 1/1:  76%|███████▋  | 117/153 [12:17:25<3:45:26, 375.74s/it]
Epoch 1/1:  77%|███████▋  | 118/153 [12:23:36<3:38:24, 374.40s/it]
Epoch 1/1:  78%|███████▊  | 119/153 [12:30:11<3:35:35, 380.46s/it]
Epoch 1/1:  78%|███████▊  | 120/153 [12:36:47<3:31:47, 385.07s/it]
Epoch 1/1:  79%|███████▉  | 121/153 [12:43:00<3:23:26, 381.46s/it]
Epoch 1/1:  80%|███████▉  | 122/153 [12:49:26<3:17:46, 382.78s/it]
Epoch 1/1:  80%|████████  | 123/153 [12:55:34<3:09:11, 378.37s/it]
Epoch 1/1:  81%|████████  | 124/153 [13:01:46<3:01:57, 376.45s/it]
Epoch 1/1:  82%|████████▏ | 125/153 [13:08:07<2:56:24, 378.01s/it]
Epoch 1/1:  82%|████████▏ | 126/153 [13:14:21<2:49:31, 376.73s/it]
Epoch 1/1:  83%|████████▎ | 127/153 [13:20:38<2:43:15, 376.76s/it]
Epoch 1/1:  84%|████████▎ | 128/153 [13:26:58<2:37:24, 377.77s/it]
Epoch 1/1:  84%|████████▍ | 129/153 [13:33:29<2:32:43, 381.83s/it]
Epoch 1/1:  85%|████████▍ | 130/153 [13:39:47<2:25:50, 380.45s/it]
Epoch 1/1:  86%|████████▌ | 131/153 [13:46:01<2:18:51, 378.72s/it]
Epoch 1/1:  86%|████████▋ | 132/153 [13:52:15<2:12:03, 377.31s/it]
Epoch 1/1:  87%|████████▋ | 133/153 [13:58:43<2:06:50, 380.50s/it]
Epoch 1/1:  88%|████████▊ | 134/153 [14:05:09<2:00:59, 382.06s/it]
Epoch 1/1:  88%|████████▊ | 135/153 [14:11:25<1:54:07, 380.41s/it]
Epoch 1/1:  89%|████████▉ | 136/153 [14:17:55<1:48:31, 383.05s/it]
Epoch 1/1:  90%|████████▉ | 137/153 [14:24:28<1:43:00, 386.27s/it]
Epoch 1/1:  90%|█████████ | 138/153 [14:30:42<1:35:38, 382.58s/it]
Epoch 1/1:  91%|█████████ | 139/153 [14:36:55<1:28:32, 379.48s/it]
Epoch 1/1:  92%|█████████▏| 140/153 [14:43:01<1:21:23, 375.68s/it]
Epoch 1/1:  92%|█████████▏| 141/153 [14:49:15<1:15:01, 375.13s/it]
Epoch 1/1:  93%|█████████▎| 142/153 [14:55:35<1:09:01, 376.54s/it]
Epoch 1/1:  93%|█████████▎| 143/153 [15:01:48<1:02:33, 375.37s/it]
Epoch 1/1:  94%|█████████▍| 144/153 [15:07:59<56:07, 374.16s/it]  
Epoch 1/1:  95%|█████████▍| 145/153 [15:14:20<50:08, 376.12s/it]
Epoch 1/1:  95%|█████████▌| 146/153 [15:20:41<44:04, 377.72s/it]
Epoch 1/1:  96%|█████████▌| 147/153 [15:27:01<37:49, 378.32s/it]
Epoch 1/1:  97%|█████████▋| 148/153 [15:33:16<31:26, 377.34s/it]
Epoch 1/1:  97%|█████████▋| 149/153 [15:39:35<25:11, 377.88s/it]
Epoch 1/1:  98%|█████████▊| 150/153 [15:46:00<19:00, 380.10s/it]
Epoch 1/1:  99%|█████████▊| 151/153 [15:52:17<12:37, 378.97s/it]
Epoch 1/1:  99%|█████████▉| 152/153 [15:58:48<06:22, 382.59s/it]
Epoch 1/1:  99%|█████████▉| 152/153 [16:07:36<06:21, 381.95s/it]
wandb: uploading config.yaml
wandb:                                                                                
wandb: 
wandb: Run history:
wandb:     train/loss ▇█▇▇▃▃▂▂▂▁▂▂▂▂▂▂▂▁▁▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▂▁
wandb: train/lr(1e-3) ▂▄▅▆██████▇▇▇▇▇▇▇▇▆▆▅▄▄▄▄▄▃▃▃▂▂▂▂▁▁▁▁▁▁▁
wandb:       val/loss ▁
wandb: 
wandb: Run summary:
wandb:     train/loss 0.37469
wandb: train/lr(1e-3) 0
wandb:       val/loss 0.97934
wandb: 
wandb: 🚀 View run qwen3_32b_omr_peft-300801 at: https://wandb.ai/llm-2025-sahara/qwen3_32b_omr_peft/runs/eo9ykh5w
wandb: ⭐️ View project at: https://wandb.ai/llm-2025-sahara/qwen3_32b_omr_peft
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20250729_011348-eo9ykh5w/logs