no change /home/appli/miniconda3/24.7.1-py311/condabin/conda no change /home/appli/miniconda3/24.7.1-py311/bin/conda no change /home/appli/miniconda3/24.7.1-py311/bin/conda-env no change /home/appli/miniconda3/24.7.1-py311/bin/activate no change /home/appli/miniconda3/24.7.1-py311/bin/deactivate no change /home/appli/miniconda3/24.7.1-py311/etc/profile.d/conda.sh no change /home/appli/miniconda3/24.7.1-py311/etc/fish/conf.d/conda.fish no change /home/appli/miniconda3/24.7.1-py311/shell/condabin/Conda.psm1 no change /home/appli/miniconda3/24.7.1-py311/shell/condabin/conda-hook.ps1 no change /home/appli/miniconda3/24.7.1-py311/lib/python3.11/site-packages/xontrib/conda.xsh no change /home/appli/miniconda3/24.7.1-py311/etc/profile.d/conda.csh no change /home/Competition2025/P06/P06U023/.bashrc No action taken. HF cache dir : /home/Competition2025/P06/P06U023/.hf_cache trainer.default_local_dir : /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints [2025-07-29 01:12:44,493] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,493] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,494] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,494] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,494] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,494] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,494] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:44,495] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-29 01:12:46,202] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:46,203] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:47,518] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:48,280] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:48,520] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:48,766] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:49,796] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False [2025-07-29 01:12:49,829] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Normalize batch size by dp 2 Using sequence parallel size: 4 Using remove padding: True Using SP rank 0 and size 2 for data distribution Each SP rank gets different data, but the same data WITHIN the same rank Using FSDP rank 0 and size 2 for data distribution [2025-07-29 01:13:01,597][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before model allocation, memory allocated (GB): 0.00, memory reserved (GB): 0.00, device memory used/total (GB): 0.51/79.44 Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None Monkey patch state_dict in AutoModelForCausalLMWithValueHead. Monkey patch _flash_attention_forward in transformers.integrations.flash_attention Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is None [2025-07-29 01:13:05,182][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After model allocation, memory allocated (GB): 0.00, memory reserved (GB): 0.00, device memory used/total (GB): 0.51/79.44 functools.partial(, policies=[functools.partial(, lambda_fn=.lambda_policy_fn at 0x15260c4e5b20>), functools.partial(, transformer_layer_cls={})]) NCCL version 2.21.5+cuda12.4 Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txtCheckpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txtCheckpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt [2025-07-29 01:13:46,717][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After FSDP wrapping, memory allocated (GB): 69.37, memory reserved (GB): 69.38, device memory used/total (GB): 72.70/79.44 [2025-07-29 01:13:46,726][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After initialize optimizer, memory allocated (GB): 69.37, memory reserved (GB): 69.38, device memory used/total (GB): 72.70/79.44 Number of steps/epoch 153, number of epochs 1, total number of steps 153 Checkpoint tracker file does not exist: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt {'data': {'train_batch_size': 256, 'micro_batch_size': None, 'micro_batch_size_per_gpu': 2, 'train_files': '/home/Competition2025/P06/P06U023/data/open_math_reasoning/train.parquet', 'val_files': '~/data/gsm8k/test.parquet', 'prompt_key': 'extra_info', 'response_key': 'extra_info', 'prompt_dict_keys': ['question'], 'response_dict_keys': ['answer'], 'multiturn': {'enable': False, 'messages_key': 'messages', 'tools_key': 'tools', 'enable_thinking_key': 'enable_thinking'}, 'max_length': 12288, 'truncation': 'right', 'balance_dp_token': False, 'chat_template': None, 'custom_cls': {'path': None, 'name': None}, 'use_shm': False}, 'model': {'partial_pretrain': 'Qwen/Qwen3-32B', 'use_shm': False, 'fsdp_config': {'model_dtype': 'bf16', 'wrap_policy': {'min_num_params': 0}, 'cpu_offload': False, 'offload_params': False}, 'external_lib': None, 'enable_gradient_checkpointing': True, 'trust_remote_code': False, 'lora_rank': 32, 'lora_alpha': 32, 'target_modules': 'all-linear', 'use_liger': False, 'strategy': 'fsdp2'}, 'optim': {'lr': 0.0001, 'betas': [0.9, 0.95], 'weight_decay': 0.01, 'warmup_steps_ratio': 0.1, 'clip_grad': 1.0, 'lr_scheduler': 'cosine'}, 'ulysses_sequence_parallel_size': 4, 'use_remove_padding': True, 'trainer': {'default_local_dir': '/home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints', 'default_hdfs_dir': None, 'project_name': 'qwen3_32b_omr_peft', 'experiment_name': 'qwen3_32b_omr_peft-300801', 'total_epochs': 1, 'total_training_steps': None, 'logger': ['console', 'wandb'], 'seed': 42, 'save_freq': 100, 'test_freq': -1, 'nnodes': 1, 'n_gpus_per_node': 8, 'max_ckpt_to_keep': 1, 'resume_mode': 'auto', 'resume_from_path': None, 'checkpoint': {'save_contents': ['model', 'optimizer', 'extra'], 'load_contents': '${trainer.checkpoint.save_contents}'}, 'device': 'cuda'}} [2025-07-29 01:13:49,955][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][INFO] - [Rank 0] Total training steps: 153, [2025-07-29 01:13:54,267][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 7.92, memory reserved (GB): 69.38, device memory used/total (GB): 72.70/79.44 [2025-07-29 01:13:54,270][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 7.92, memory reserved (GB): 69.38, device memory used/total (GB): 72.70/79.44 NCCL version 2.21.5+cuda12.4 [2025-07-29 01:20:06,756][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.08, memory reserved (GB): 29.86, device memory used/total (GB): 36.51/79.44 [2025-07-29 01:20:08,264][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:20:08,265][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 step:1 - train/loss:0.48927566409111023 - train/lr(1e-3):0.006666666666666666 [2025-07-29 01:20:08,305][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:20:08,307][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:26:18,907][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.18, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:26:18,951][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.18, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:26:18,952][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.18, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 step:2 - train/loss:0.502138614654541 - train/lr(1e-3):0.013333333333333332 [2025-07-29 01:26:18,971][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:26:18,973][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 30.03, device memory used/total (GB): 36.68/79.44 [2025-07-29 01:32:41,506][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:32:41,729][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:32:41,729][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:3 - train/loss:0.4939456582069397 - train/lr(1e-3):0.02 [2025-07-29 01:32:41,755][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:32:41,756][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:39:00,305][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:39:00,350][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:39:00,351][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:4 - train/loss:0.49726566672325134 - train/lr(1e-3):0.026666666666666665 [2025-07-29 01:39:00,370][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:39:00,373][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:45:15,260][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:45:15,306][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:45:15,306][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:5 - train/loss:0.49197182059288025 - train/lr(1e-3):0.03333333333333333 [2025-07-29 01:45:15,326][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:45:15,328][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:51:45,776][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:51:45,817][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:51:45,817][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:6 - train/loss:0.49353623390197754 - train/lr(1e-3):0.04 [2025-07-29 01:51:45,841][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:51:45,843][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:57:57,814][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:57:57,846][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:57:57,846][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:7 - train/loss:0.49380552768707275 - train/lr(1e-3):0.04666666666666667 [2025-07-29 01:57:57,867][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 01:57:57,869][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:04:23,184][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:04:23,224][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:04:23,224][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:8 - train/loss:0.4896022081375122 - train/lr(1e-3):0.05333333333333333 [2025-07-29 02:04:23,246][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:04:23,248][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:10:58,635][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:10:58,672][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:10:58,672][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:9 - train/loss:0.48374390602111816 - train/lr(1e-3):0.060000000000000005 [2025-07-29 02:10:58,693][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:10:58,695][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:17:09,764][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:17:09,805][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:17:09,806][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:10 - train/loss:0.47333505749702454 - train/lr(1e-3):0.06666666666666667 [2025-07-29 02:17:09,825][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:17:09,827][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:23:27,707][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:23:27,747][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:23:27,748][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:11 - train/loss:0.4494769275188446 - train/lr(1e-3):0.07333333333333333 [2025-07-29 02:23:27,768][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:23:27,770][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:29:44,599][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:29:44,640][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:29:44,640][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:12 - train/loss:0.44925227761268616 - train/lr(1e-3):0.08 [2025-07-29 02:29:44,662][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:29:44,664][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:36:15,750][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:36:15,796][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:36:15,797][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:13 - train/loss:0.4417742192745209 - train/lr(1e-3):0.08666666666666667 [2025-07-29 02:36:15,818][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:36:15,820][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:42:25,987][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:42:26,028][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:42:26,029][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:14 - train/loss:0.41446593403816223 - train/lr(1e-3):0.09333333333333334 [2025-07-29 02:42:26,049][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:42:26,051][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:48:44,867][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:48:45,138][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:48:45,139][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:15 - train/loss:0.40411776304244995 - train/lr(1e-3):0.1 [2025-07-29 02:48:45,345][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:48:45,348][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:54:58,916][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:54:58,964][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:54:58,964][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:16 - train/loss:0.4080301821231842 - train/lr(1e-3):0.09998704424206746 [2025-07-29 02:54:58,984][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 02:54:58,986][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:01:32,973][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:01:33,008][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:01:33,009][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:17 - train/loss:0.40999501943588257 - train/lr(1e-3):0.09994818368233639 [2025-07-29 03:01:33,032][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:01:33,035][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:07:47,311][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:07:47,349][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:07:47,351][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:18 - train/loss:0.40056535601615906 - train/lr(1e-3):0.09988343845952696 [2025-07-29 03:07:47,374][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:07:47,377][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:14:09,027][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:14:09,062][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:14:09,063][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:19 - train/loss:0.40851113200187683 - train/lr(1e-3):0.09979284212657658 [2025-07-29 03:14:09,083][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:14:09,086][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:20:36,544][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:20:36,585][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:20:36,585][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:20 - train/loss:0.39902764558792114 - train/lr(1e-3):0.09967644163325157 [2025-07-29 03:20:36,608][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:20:36,610][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:27:01,617][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:27:01,657][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:27:01,657][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:21 - train/loss:0.39263248443603516 - train/lr(1e-3):0.09953429730181654 [2025-07-29 03:27:01,679][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:27:01,681][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:33:54,859][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:33:54,903][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:33:54,903][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:22 - train/loss:0.40035223960876465 - train/lr(1e-3):0.0993664827957735 [2025-07-29 03:33:54,926][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:33:54,928][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:40:14,165][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:40:14,213][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:40:14,214][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:23 - train/loss:0.3827296197414398 - train/lr(1e-3):0.09917308508168711 [2025-07-29 03:40:14,236][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:40:14,238][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:46:28,588][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:46:28,628][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:46:28,629][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:24 - train/loss:0.3852523863315582 - train/lr(1e-3):0.09895420438411616 [2025-07-29 03:46:28,649][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:46:28,651][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:52:26,456][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:52:26,495][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:52:26,496][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:25 - train/loss:0.3770380914211273 - train/lr(1e-3):0.09870995413367398 [2025-07-29 03:52:26,517][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:52:26,519][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:58:50,443][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:58:50,493][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:58:50,494][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:26 - train/loss:0.38516733050346375 - train/lr(1e-3):0.09844046090824532 [2025-07-29 03:58:50,514][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 03:58:50,516][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:05:12,674][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:05:12,709][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:05:12,709][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:27 - train/loss:0.3796369135379791 - train/lr(1e-3):0.09814586436738998 [2025-07-29 04:05:12,730][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:05:12,732][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:11:08,774][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:11:08,812][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:11:08,812][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:28 - train/loss:0.38097819685935974 - train/lr(1e-3):0.09782631717996675 [2025-07-29 04:11:08,832][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:11:08,834][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:17:42,777][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:17:42,816][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:17:42,817][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:29 - train/loss:0.39208462834358215 - train/lr(1e-3):0.09748198494501596 [2025-07-29 04:17:42,837][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:17:42,839][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:24:09,324][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:24:09,362][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:24:09,362][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:30 - train/loss:0.3739989697933197 - train/lr(1e-3):0.09711304610594104 [2025-07-29 04:24:09,382][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:24:09,384][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:30:25,476][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:30:25,519][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:30:25,520][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:31 - train/loss:0.38401558995246887 - train/lr(1e-3):0.09671969185803356 [2025-07-29 04:30:25,539][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:30:25,541][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:36:34,153][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:36:34,192][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:36:34,192][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:32 - train/loss:0.37584152817726135 - train/lr(1e-3):0.09630212604939026 [2025-07-29 04:36:34,215][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:36:34,217][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:42:52,154][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:42:52,202][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:42:52,202][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:33 - train/loss:0.38263288140296936 - train/lr(1e-3):0.09586056507527266 [2025-07-29 04:42:52,221][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:42:52,223][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:48:59,255][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:48:59,293][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:48:59,293][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:34 - train/loss:0.38096144795417786 - train/lr(1e-3):0.09539523776596445 [2025-07-29 04:48:59,316][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:48:59,317][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:55:21,873][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:55:21,912][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:55:21,912][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:35 - train/loss:0.3740430772304535 - train/lr(1e-3):0.0949063852681848 [2025-07-29 04:55:21,932][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 04:55:21,934][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:01:36,485][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:01:36,526][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:01:36,526][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:36 - train/loss:0.38744255900382996 - train/lr(1e-3):0.09439426092011875 [2025-07-29 05:01:36,548][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:01:36,550][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:07:49,265][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:07:49,301][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:07:49,301][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:37 - train/loss:0.38397741317749023 - train/lr(1e-3):0.09385913012012972 [2025-07-29 05:07:49,321][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:07:49,322][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:14:12,203][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:14:12,268][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:14:12,269][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:38 - train/loss:0.37641531229019165 - train/lr(1e-3):0.09330127018922194 [2025-07-29 05:14:12,288][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:14:12,290][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:20:23,275][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:20:23,318][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:20:23,319][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:39 - train/loss:0.3775506019592285 - train/lr(1e-3):0.09272097022732444 [2025-07-29 05:20:23,337][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:20:23,339][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:26:29,860][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:26:29,904][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:26:29,904][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:40 - train/loss:0.3829081952571869 - train/lr(1e-3):0.09211853096347059 [2025-07-29 05:26:29,924][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:26:29,926][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:32:53,383][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:32:53,420][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:32:53,420][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:41 - train/loss:0.37503933906555176 - train/lr(1e-3):0.09149426459995126 [2025-07-29 05:32:53,439][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:32:53,441][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:39:07,386][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:39:07,426][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:39:07,426][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:42 - train/loss:0.37386250495910645 - train/lr(1e-3):0.0908484946505221 [2025-07-29 05:39:07,447][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:39:07,449][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:45:25,882][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:45:26,104][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:45:26,104][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:43 - train/loss:0.3770744502544403 - train/lr(1e-3):0.09018155577274892 [2025-07-29 05:45:26,128][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:45:26,130][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:51:43,013][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:51:43,271][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:51:43,272][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:44 - train/loss:0.37810567021369934 - train/lr(1e-3):0.08949379359457793 [2025-07-29 05:51:43,296][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:51:43,298][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:58:14,461][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:58:14,499][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:58:14,500][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:45 - train/loss:0.38017380237579346 - train/lr(1e-3):0.088785564535221 [2025-07-29 05:58:14,521][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 05:58:14,523][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:04:38,667][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:04:38,708][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:04:38,709][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:46 - train/loss:0.37463775277137756 - train/lr(1e-3):0.08805723562044825 [2025-07-29 06:04:38,730][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:04:38,732][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:10:52,116][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:10:52,164][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:10:52,165][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:47 - train/loss:0.37674713134765625 - train/lr(1e-3):0.08730918429238428 [2025-07-29 06:10:52,185][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:10:52,187][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:17:12,395][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:17:12,434][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:17:12,435][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:48 - train/loss:0.3733048737049103 - train/lr(1e-3):0.08654179821390622 [2025-07-29 06:17:12,458][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:17:12,460][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:23:32,629][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:23:32,671][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:23:32,672][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:49 - train/loss:0.37479475140571594 - train/lr(1e-3):0.08575547506774497 [2025-07-29 06:23:32,692][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:23:32,694][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:29:51,355][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:29:51,397][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:29:51,397][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:50 - train/loss:0.3702385127544403 - train/lr(1e-3):0.0849506223503941 [2025-07-29 06:29:51,417][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:29:51,419][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:36:06,407][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:36:06,442][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:36:06,443][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:51 - train/loss:0.3803344666957855 - train/lr(1e-3):0.08412765716093272 [2025-07-29 06:36:06,466][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:36:06,469][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:42:19,352][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:42:19,386][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:42:19,387][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:52 - train/loss:0.3804817795753479 - train/lr(1e-3):0.08328700598487203 [2025-07-29 06:42:19,410][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:42:19,412][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:48:22,694][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:48:22,732][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:48:22,733][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:53 - train/loss:0.37815171480178833 - train/lr(1e-3):0.0824291044731378 [2025-07-29 06:48:22,760][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:48:22,764][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:54:54,651][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:54:54,698][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:54:54,699][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:54 - train/loss:0.37224283814430237 - train/lr(1e-3):0.08155439721630264 [2025-07-29 06:54:54,723][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 06:54:54,726][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:01:18,872][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:01:18,905][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:01:18,906][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:55 - train/loss:0.3773203194141388 - train/lr(1e-3):0.08066333751418582 [2025-07-29 07:01:18,931][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:01:18,934][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:07:42,657][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:07:42,691][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:07:42,692][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:56 - train/loss:0.3921739161014557 - train/lr(1e-3):0.0797563871409395 [2025-07-29 07:07:42,715][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:07:42,718][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:14:01,279][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:14:01,331][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:14:01,331][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:57 - train/loss:0.36699244379997253 - train/lr(1e-3):0.07883401610574337 [2025-07-29 07:14:01,355][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:14:01,357][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:20:17,351][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:20:17,398][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:20:17,398][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:58 - train/loss:0.36495405435562134 - train/lr(1e-3):0.07789670240923167 [2025-07-29 07:20:17,421][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:20:17,424][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:26:41,132][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:26:41,381][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:26:41,382][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:59 - train/loss:0.3690050542354584 - train/lr(1e-3):0.07694493179577878 [2025-07-29 07:26:41,408][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:26:41,411][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:33:01,665][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:33:01,703][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:33:01,704][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:60 - train/loss:0.37953850626945496 - train/lr(1e-3):0.07597919750177168 [2025-07-29 07:33:01,727][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:33:01,729][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:39:17,777][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:39:17,817][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:39:17,818][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:61 - train/loss:0.37410956621170044 - train/lr(1e-3):0.07500000000000001 [2025-07-29 07:39:17,842][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:39:17,844][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:45:39,148][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:45:39,183][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:45:39,183][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:62 - train/loss:0.3837984800338745 - train/lr(1e-3):0.07400784674029579 [2025-07-29 07:45:39,209][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:45:39,212][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:51:39,479][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:51:39,516][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:51:39,516][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:63 - train/loss:0.3628824055194855 - train/lr(1e-3):0.07300325188655761 [2025-07-29 07:51:39,538][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:51:39,541][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:57:59,697][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:57:59,745][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:57:59,745][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:64 - train/loss:0.3721928000450134 - train/lr(1e-3):0.07198673605029528 [2025-07-29 07:57:59,767][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 07:57:59,769][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:04:14,811][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:04:14,844][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:04:14,844][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:65 - train/loss:0.3690522611141205 - train/lr(1e-3):0.07095882602083321 [2025-07-29 08:04:14,869][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:04:14,871][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:10:37,311][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:10:37,355][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:10:37,355][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:66 - train/loss:0.3782886564731598 - train/lr(1e-3):0.06992005449231208 [2025-07-29 08:10:37,379][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:10:37,381][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:17:00,179][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:17:00,212][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:17:00,213][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:67 - train/loss:0.37617892026901245 - train/lr(1e-3):0.06887095978763072 [2025-07-29 08:17:00,235][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:17:00,237][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:23:10,617][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:23:10,668][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:23:10,669][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:68 - train/loss:0.36986514925956726 - train/lr(1e-3):0.06781208557947085 [2025-07-29 08:23:10,708][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:23:10,710][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:29:19,268][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:29:19,311][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:29:19,311][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:69 - train/loss:0.36530801653862 - train/lr(1e-3):0.06674398060854932 [2025-07-29 08:29:19,332][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:29:19,334][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:35:45,199][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:35:45,235][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:35:45,235][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:70 - train/loss:0.3648270070552826 - train/lr(1e-3):0.06566719839924412 [2025-07-29 08:35:45,257][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:35:45,260][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:42:05,595][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:42:05,854][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:42:05,855][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:71 - train/loss:0.38293910026550293 - train/lr(1e-3):0.06458229697274125 [2025-07-29 08:42:05,880][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:42:05,882][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:48:20,779][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:48:20,811][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:48:20,811][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:72 - train/loss:0.370816171169281 - train/lr(1e-3):0.06348983855785122 [2025-07-29 08:48:20,832][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:48:20,834][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:54:39,896][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:54:39,939][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:54:39,939][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:73 - train/loss:0.36970484256744385 - train/lr(1e-3):0.062390389299645 [2025-07-29 08:54:39,964][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 08:54:39,968][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:00:54,597][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:00:54,641][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:00:54,641][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:74 - train/loss:0.37763655185699463 - train/lr(1e-3):0.06128451896606053 [2025-07-29 09:00:54,663][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:00:54,666][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:07:02,364][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:07:02,411][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:07:02,412][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:75 - train/loss:0.3673696517944336 - train/lr(1e-3):0.0601728006526317 [2025-07-29 09:07:02,434][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:07:02,436][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:13:23,878][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:13:23,911][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:13:23,911][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:76 - train/loss:0.3762687146663666 - train/lr(1e-3):0.05905581048549279 [2025-07-29 09:13:23,955][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:13:23,958][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:19:49,098][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:19:49,139][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:19:49,139][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:77 - train/loss:0.3651543855667114 - train/lr(1e-3):0.05793412732281257 [2025-07-29 09:19:49,160][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:19:49,163][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:26:05,020][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:26:05,242][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:26:05,246][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:78 - train/loss:0.3730051517486572 - train/lr(1e-3):0.05680833245481234 [2025-07-29 09:26:05,271][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:26:05,273][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:32:41,662][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:32:41,703][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:32:41,704][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:79 - train/loss:0.38064640760421753 - train/lr(1e-3):0.055679009302523744 [2025-07-29 09:32:41,729][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:32:41,731][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:38:45,964][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:38:45,997][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:38:45,998][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:80 - train/loss:0.36939799785614014 - train/lr(1e-3):0.05454674311544235 [2025-07-29 09:38:46,018][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:38:46,022][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:44:50,796][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:44:50,831][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:44:50,831][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:81 - train/loss:0.3635561466217041 - train/lr(1e-3):0.05341212066823355 [2025-07-29 09:44:50,851][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:44:50,854][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:51:06,902][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:51:06,935][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:51:06,935][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:82 - train/loss:0.3664884567260742 - train/lr(1e-3):0.05227572995664819 [2025-07-29 09:51:06,956][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:51:06,958][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:57:13,817][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:57:13,851][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:57:13,852][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:83 - train/loss:0.3703444302082062 - train/lr(1e-3):0.05113815989280528 [2025-07-29 09:57:13,874][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 09:57:13,876][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:03:13,056][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:03:13,101][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:03:13,102][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:84 - train/loss:0.3666549324989319 - train/lr(1e-3):0.05 [2025-07-29 10:03:13,122][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:03:13,125][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:09:28,362][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:09:28,409][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:09:28,410][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:85 - train/loss:0.37358328700065613 - train/lr(1e-3):0.048861840107194715 [2025-07-29 10:09:28,431][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:09:28,433][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:15:50,847][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:15:50,892][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:15:50,893][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:86 - train/loss:0.36911144852638245 - train/lr(1e-3):0.047724270043351835 [2025-07-29 10:15:50,915][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:15:50,917][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:22:09,996][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:22:10,030][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:22:10,031][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:87 - train/loss:0.37562835216522217 - train/lr(1e-3):0.04658787933176646 [2025-07-29 10:22:10,051][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:22:10,053][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:28:16,649][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:28:16,694][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:28:16,694][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:88 - train/loss:0.3674798309803009 - train/lr(1e-3):0.045453256884557654 [2025-07-29 10:28:16,714][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:28:16,716][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:34:38,773][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:34:38,860][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:34:38,860][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:89 - train/loss:0.38109299540519714 - train/lr(1e-3):0.04432099069747625 [2025-07-29 10:34:38,884][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:34:38,886][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:40:52,159][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:40:52,207][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:40:52,207][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:90 - train/loss:0.3649253845214844 - train/lr(1e-3):0.043191667545187676 [2025-07-29 10:40:52,227][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:40:52,230][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:47:05,424][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:47:05,460][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:47:05,461][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:91 - train/loss:0.36195677518844604 - train/lr(1e-3):0.042065872677187435 [2025-07-29 10:47:05,482][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:47:05,484][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:53:24,107][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:53:24,149][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:53:24,149][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:92 - train/loss:0.3641708493232727 - train/lr(1e-3):0.04094418951450721 [2025-07-29 10:53:24,169][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:53:24,172][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:59:32,643][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:59:32,688][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:59:32,689][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:93 - train/loss:0.3717505633831024 - train/lr(1e-3):0.03982719934736832 [2025-07-29 10:59:32,710][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 10:59:32,713][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:05:44,602][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:05:44,640][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:05:44,640][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:94 - train/loss:0.3687405586242676 - train/lr(1e-3):0.03871548103393947 [2025-07-29 11:05:44,663][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:05:44,666][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:12:00,583][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:12:00,625][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:12:00,625][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:95 - train/loss:0.37281692028045654 - train/lr(1e-3):0.03760961070035501 [2025-07-29 11:12:00,646][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:12:00,649][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:18:17,118][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:18:17,166][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:18:17,167][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:96 - train/loss:0.37044841051101685 - train/lr(1e-3):0.03651016144214878 [2025-07-29 11:18:17,193][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:18:17,196][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:24:19,999][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:24:20,038][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:24:20,038][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:97 - train/loss:0.36620038747787476 - train/lr(1e-3):0.03541770302725875 [2025-07-29 11:24:20,059][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:24:20,061][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:30:43,809][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:30:43,847][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:30:43,847][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:98 - train/loss:0.36444032192230225 - train/lr(1e-3):0.03433280160075589 [2025-07-29 11:30:43,868][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:30:43,871][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:36:52,331][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:36:52,381][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:36:52,382][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:99 - train/loss:0.38232284784317017 - train/lr(1e-3):0.033256019391450696 [2025-07-29 11:36:52,403][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:36:52,405][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:42:57,742][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:42:57,783][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:42:57,783][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:100 - train/loss:0.37200331687927246 - train/lr(1e-3):0.03218791442052917 Saving checkpoint to: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100 [2025-07-29 11:43:07,586][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_3.pt [2025-07-29 11:43:07,737][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_7.pt [2025-07-29 11:43:07,884][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_3.pt [2025-07-29 11:43:07,902][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_3.pt [2025-07-29 11:43:07,918][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_6.pt [2025-07-29 11:43:07,988][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_2.pt [2025-07-29 11:43:08,034][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_7.pt [2025-07-29 11:43:08,037][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_7.pt [2025-07-29 11:43:08,039][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_5.pt [2025-07-29 11:43:08,074][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_1.pt [2025-07-29 11:43:08,225][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_6.pt [2025-07-29 11:43:08,228][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_6.pt [2025-07-29 11:43:08,234][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_4.pt [2025-07-29 11:43:08,280][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_2.pt [2025-07-29 11:43:08,282][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_2.pt [2025-07-29 11:43:08,377][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_1.pt [2025-07-29 11:43:08,379][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_1.pt [2025-07-29 11:43:08,527][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_4.pt [2025-07-29 11:43:08,529][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_4.pt [2025-07-29 11:43:08,615][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_5.pt [2025-07-29 11:43:08,617][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_5.pt [2025-07-29 11:43:08,971][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/model_world_size_8_rank_0.pt [2025-07-29 11:43:09,258][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/optim_world_size_8_rank_0.pt [2025-07-29 11:43:09,260][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/extra_state_world_size_8_rank_0.pt [2025-07-29 11:43:09,714][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved model config and tokenizer class to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/huggingface Saved dataloader state to: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100/data.pt Updated checkpoint tracker: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt [2025-07-29 11:43:09,768][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:43:09,772][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:49:29,742][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:49:29,783][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:49:29,783][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:101 - train/loss:0.37323981523513794 - train/lr(1e-3):0.03112904021236929 [2025-07-29 11:49:29,809][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:49:29,812][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:56:02,033][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:56:02,069][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:56:02,070][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:102 - train/loss:0.37325724959373474 - train/lr(1e-3):0.03007994550768793 [2025-07-29 11:56:02,090][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 11:56:02,093][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:02:35,833][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:02:35,870][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:02:35,871][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:103 - train/loss:0.3684852123260498 - train/lr(1e-3):0.029041173979166814 [2025-07-29 12:02:35,893][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:02:35,895][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:08:48,145][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:08:48,208][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:08:48,211][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:104 - train/loss:0.37162700295448303 - train/lr(1e-3):0.028013263949704704 [2025-07-29 12:08:48,232][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:08:48,234][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:15:15,375][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:15:15,409][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:15:15,410][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:105 - train/loss:0.3769713342189789 - train/lr(1e-3):0.026996748113442395 [2025-07-29 12:15:15,430][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:15:15,432][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:21:31,270][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:21:31,314][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:21:31,314][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:106 - train/loss:0.36926761269569397 - train/lr(1e-3):0.025992153259704227 [2025-07-29 12:21:31,336][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:21:31,338][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:28:02,121][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:28:02,158][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:28:02,159][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:107 - train/loss:0.375434011220932 - train/lr(1e-3):0.025000000000000012 [2025-07-29 12:28:02,181][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:28:02,183][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:34:16,570][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:34:16,617][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:34:16,617][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:108 - train/loss:0.36851033568382263 - train/lr(1e-3):0.024020802498228334 [2025-07-29 12:34:16,639][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:34:16,641][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:40:35,382][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:40:35,431][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:40:35,431][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:109 - train/loss:0.3699564039707184 - train/lr(1e-3):0.023055068204221225 [2025-07-29 12:40:35,455][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:40:35,458][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:46:51,201][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:46:51,234][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:46:51,234][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:110 - train/loss:0.3816758692264557 - train/lr(1e-3):0.022103297590768334 [2025-07-29 12:46:51,257][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:46:51,260][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:53:22,609][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:53:22,645][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:53:22,645][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:111 - train/loss:0.37761905789375305 - train/lr(1e-3):0.021165983894256647 [2025-07-29 12:53:22,670][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:53:22,673][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:59:43,286][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:59:43,325][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:59:43,326][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:112 - train/loss:0.37119513750076294 - train/lr(1e-3):0.020243612859060525 [2025-07-29 12:59:43,350][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 12:59:43,352][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:06:13,528][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:06:13,562][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:06:13,562][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:113 - train/loss:0.374712735414505 - train/lr(1e-3):0.019336662485814178 [2025-07-29 13:06:13,586][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:06:13,589][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:12:39,424][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:12:39,459][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:12:39,460][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:114 - train/loss:0.3739401698112488 - train/lr(1e-3):0.018445602783697373 [2025-07-29 13:12:39,484][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:12:39,486][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:18:59,982][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:19:00,026][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:19:00,027][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:115 - train/loss:0.3671330511569977 - train/lr(1e-3):0.017570895526862203 [2025-07-29 13:19:00,050][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:19:00,053][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:25:05,181][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:25:05,225][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:25:05,226][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:116 - train/loss:0.36488962173461914 - train/lr(1e-3):0.016712994015127977 [2025-07-29 13:25:05,250][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:25:05,253][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:31:15,415][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:31:15,450][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:31:15,451][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:117 - train/loss:0.3776390552520752 - train/lr(1e-3):0.015872342839067306 [2025-07-29 13:31:15,477][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:31:15,479][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:37:26,702][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:37:26,735][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:37:26,735][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:118 - train/loss:0.3713444769382477 - train/lr(1e-3):0.015049377649605905 [2025-07-29 13:37:26,755][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:37:26,757][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:44:01,299][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:44:01,333][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:44:01,333][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:119 - train/loss:0.37413060665130615 - train/lr(1e-3):0.014244524932255026 [2025-07-29 13:44:01,364][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:44:01,367][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:50:37,110][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:50:37,155][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:50:37,157][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:120 - train/loss:0.37140199542045593 - train/lr(1e-3):0.013458201786093794 [2025-07-29 13:50:37,179][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:50:37,213][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:56:50,170][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:56:50,204][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:56:50,205][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:121 - train/loss:0.3671881854534149 - train/lr(1e-3):0.012690815707615728 [2025-07-29 13:56:50,225][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 13:56:50,227][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:03:16,020][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:03:16,053][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:03:16,054][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:122 - train/loss:0.3691574037075043 - train/lr(1e-3):0.011942764379551768 [2025-07-29 14:03:16,075][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:03:16,077][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:09:24,094][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:09:24,128][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:09:24,129][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:123 - train/loss:0.3725578784942627 - train/lr(1e-3):0.011214435464779005 [2025-07-29 14:09:24,149][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:09:24,152][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:15:36,073][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:15:36,113][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:15:36,114][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:124 - train/loss:0.377387672662735 - train/lr(1e-3):0.01050620640542208 [2025-07-29 14:15:36,137][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:15:36,139][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:21:57,719][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:21:57,753][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:21:57,753][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:125 - train/loss:0.3717491328716278 - train/lr(1e-3):0.009818444227251089 [2025-07-29 14:21:57,775][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:21:57,777][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:28:11,454][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:28:11,504][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:28:11,505][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:126 - train/loss:0.36843299865722656 - train/lr(1e-3):0.009151505349477902 [2025-07-29 14:28:11,524][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:28:11,526][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:34:28,306][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:34:28,340][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:34:28,341][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:127 - train/loss:0.36831924319267273 - train/lr(1e-3):0.008505735400048748 [2025-07-29 14:34:28,362][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:34:28,363][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:40:48,407][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:40:48,444][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:40:48,445][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:128 - train/loss:0.3693549633026123 - train/lr(1e-3):0.007881469036529427 [2025-07-29 14:40:48,464][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:40:48,466][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:47:19,708][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:47:19,741][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:47:19,742][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:129 - train/loss:0.37121301889419556 - train/lr(1e-3):0.007279029772675571 [2025-07-29 14:47:19,761][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:47:19,764][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:53:36,919][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:53:36,969][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:53:36,969][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:130 - train/loss:0.3689507246017456 - train/lr(1e-3):0.006698729810778065 [2025-07-29 14:53:36,989][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:53:36,992][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:59:51,600][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:59:51,646][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:59:51,647][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:131 - train/loss:0.3707675337791443 - train/lr(1e-3):0.006140869879870287 [2025-07-29 14:59:51,668][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 14:59:51,671][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:06:05,646][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:06:05,679][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:06:05,679][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:132 - train/loss:0.36552202701568604 - train/lr(1e-3):0.005605739079881239 [2025-07-29 15:06:05,700][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:06:05,702][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:12:33,593][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:12:33,628][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:12:33,629][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.27, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:133 - train/loss:0.36779263615608215 - train/lr(1e-3):0.005093614731815199 [2025-07-29 15:12:33,652][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:12:33,654][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:18:59,283][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:18:59,328][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:18:59,328][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:134 - train/loss:0.3798401355743408 - train/lr(1e-3):0.004604762234035548 [2025-07-29 15:18:59,351][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:18:59,353][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:25:15,840][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:25:15,880][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:25:15,880][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:135 - train/loss:0.37669065594673157 - train/lr(1e-3):0.004139434924727359 [2025-07-29 15:25:15,902][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:25:15,904][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:31:45,047][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:31:45,087][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:31:45,088][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:136 - train/loss:0.37761449813842773 - train/lr(1e-3):0.003697873950609737 [2025-07-29 15:31:45,114][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:31:45,116][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:38:18,836][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:38:18,885][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:38:18,886][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:137 - train/loss:0.36930346488952637 - train/lr(1e-3):0.0032803081419664486 [2025-07-29 15:38:18,914][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:38:18,917][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:44:32,810][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:44:32,846][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:44:32,846][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.25, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:138 - train/loss:0.3631953001022339 - train/lr(1e-3):0.00288695389405898 [2025-07-29 15:44:32,873][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:44:32,876][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:50:45,049][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:50:45,094][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:50:45,094][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:139 - train/loss:0.3768501579761505 - train/lr(1e-3):0.002518015054984041 [2025-07-29 15:50:45,121][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:50:45,123][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:56:51,870][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:56:51,905][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:56:51,906][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.22, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:140 - train/loss:0.36562734842300415 - train/lr(1e-3):0.0021736828200332623 [2025-07-29 15:56:51,932][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 15:56:51,935][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:03:05,709][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:03:05,746][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:03:05,747][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:141 - train/loss:0.37323257327079773 - train/lr(1e-3):0.0018541356326100434 [2025-07-29 16:03:05,773][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:03:05,775][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:09:25,557][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:09:25,594][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:09:25,595][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:142 - train/loss:0.3664059638977051 - train/lr(1e-3):0.0015595390917546859 [2025-07-29 16:09:25,620][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:09:25,623][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:15:38,187][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:15:38,224][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:15:38,224][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:143 - train/loss:0.3748567998409271 - train/lr(1e-3):0.0012900458663260507 [2025-07-29 16:15:38,250][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:15:38,253][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:21:49,519][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:21:49,553][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:21:49,553][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.19, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:144 - train/loss:0.36987045407295227 - train/lr(1e-3):0.0010457956158838543 [2025-07-29 16:21:49,579][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:21:49,582][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:28:10,227][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:28:10,264][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:28:10,265][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:145 - train/loss:0.36719805002212524 - train/lr(1e-3):0.0008269149183128988 [2025-07-29 16:28:10,291][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:28:10,293][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:34:31,665][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:34:31,701][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:34:31,702][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:146 - train/loss:0.37482917308807373 - train/lr(1e-3):0.0006335172042265192 [2025-07-29 16:34:31,728][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:34:31,730][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:40:51,392][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:40:51,441][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:40:51,442][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.24, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:147 - train/loss:0.37063464522361755 - train/lr(1e-3):0.00046570269818346224 [2025-07-29 16:40:51,468][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:40:51,471][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:47:06,441][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:47:06,477][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:47:06,478][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.28, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:148 - train/loss:0.3727622926235199 - train/lr(1e-3):0.0003235583667484443 [2025-07-29 16:47:06,506][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:47:06,509][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:53:25,572][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:53:25,608][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:53:25,608][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.26, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:149 - train/loss:0.3654772639274597 - train/lr(1e-3):0.00020715787342343584 [2025-07-29 16:53:25,634][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:53:25,637][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:59:50,856][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:59:50,906][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:59:50,907][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.23, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:150 - train/loss:0.3760713040828705 - train/lr(1e-3):0.00011656154047303691 [2025-07-29 16:59:50,932][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 16:59:50,935][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:06:07,179][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:06:07,225][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:06:07,225][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:151 - train/loss:0.36653822660446167 - train/lr(1e-3):5.181631766362216e-05 [2025-07-29 17:06:07,251][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:06:07,254][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:12:38,245][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:12:38,282][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:12:38,282][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.21, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:152 - train/loss:0.3736474812030792 - train/lr(1e-3):1.2955757932542334e-05 [2025-07-29 17:12:38,307][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer zero_grad, memory allocated (GB): 8.17, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:12:38,311][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer zero_grad, memory allocated (GB): 8.11, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:19:15,034][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - Before optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:19:15,076][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After optimizer step, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 [2025-07-29 17:19:15,077][/home/Competition2025/P06/P06U023/deps/verl/verl/trainer/fsdp_sft_trainer.py][DEBUG] - After offload weights, memory allocated (GB): 8.20, memory reserved (GB): 33.51, device memory used/total (GB): 40.16/79.44 step:153 - train/loss:0.3746890723705292 - train/lr(1e-3):0.0 step:153 - val/loss:0.9793385863304138 Saving checkpoint to: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153 Checkpoint manager remove previous save local path: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_100 [2025-07-29 17:21:25,477][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_2.pt [2025-07-29 17:21:25,587][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_5.pt [2025-07-29 17:21:25,661][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_7.pt [2025-07-29 17:21:25,681][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_3.pt [2025-07-29 17:21:25,771][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_4.pt [2025-07-29 17:21:25,787][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_2.pt [2025-07-29 17:21:25,792][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 2] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_2.pt [2025-07-29 17:21:25,823][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_6.pt [2025-07-29 17:21:25,891][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_5.pt [2025-07-29 17:21:25,893][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 5] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_5.pt [2025-07-29 17:21:25,982][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_3.pt [2025-07-29 17:21:25,985][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 3] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_3.pt [2025-07-29 17:21:25,990][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_1.pt [2025-07-29 17:21:26,074][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_4.pt [2025-07-29 17:21:26,076][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 4] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_4.pt [2025-07-29 17:21:26,126][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_6.pt [2025-07-29 17:21:26,128][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 6] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_6.pt [2025-07-29 17:21:26,184][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved model to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/model_world_size_8_rank_0.pt [2025-07-29 17:21:26,234][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_7.pt [2025-07-29 17:21:26,237][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 7] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_7.pt [2025-07-29 17:21:26,282][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_1.pt [2025-07-29 17:21:26,284][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 1] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_1.pt [2025-07-29 17:21:26,477][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved optim to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/optim_world_size_8_rank_0.pt [2025-07-29 17:21:26,479][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved extra_state to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/extra_state_world_size_8_rank_0.pt [2025-07-29 17:21:26,927][/home/Competition2025/P06/P06U023/deps/verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py][INFO] - [Rank 0] Saved model config and tokenizer class to /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/huggingface Saved dataloader state to: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/global_step_153/data.pt Updated checkpoint tracker: /home/Competition2025/P06/P06U023/training/sft/open_math_reasoning/qwen3_32b_omr_peft-300801/checkpoints/latest_checkpointed_iteration.txt Final validation metrics: {'val/loss': 0.9793385863304138}