safe_v2_2nodes/logs/22110707_4.err

Setting Data Parallel size to 16
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-05-05 06:44:38.655991 - 0:00:00 - Signal handler installed.
WARNING 26-05-05 06:44:38.656353 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-05-05 06:44:38.656427 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-05-05 06:44:38.656482 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-05-05 06:44:38.656528 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-05-05 06:44:38.656566 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-05-05 06:44:38.656603 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-05-05 06:44:38.656640 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-05-05 06:44:38.656674 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmpu2zh2ug3
/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/torch/autograd/graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at ../aten/src/ATen/native/cudnn/MHA.cpp:674.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank4]:[W505 06:50:07.922653546 CPUAllocator.cpp:249] Memory block of unknown size was allocated before the profiling started, profiler results will not include the deallocation event
WARNING 26-05-05 08:14:49.351035 - 1:30:11 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 09:38:44.825862 - 2:54:06 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 11:03:01.703429 - 4:18:23 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 12:27:12.636731 - 5:42:34 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 13:51:31.728181 - 7:06:53 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 15:15:46.908300 - 8:31:08 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 16:40:03.277043 - 9:55:25 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 18:04:21.047095 - 11:19:42 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 19:28:39.000709 - 12:44:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 20:53:00.852759 - 14:08:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 22:17:23.810742 - 15:32:45 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 23:41:48.937108 - 16:57:10 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 01:06:00.852151 - 18:21:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 02:30:26.969385 - 19:45:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 03:54:51.993603 - 21:10:13 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 05:19:11.139526 - 22:34:32 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 06:43:18.419510 - 23:58:40 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 08:07:49.915832 - 1 day, 1:23:11 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 09:32:11.976173 - 1 day, 2:47:33 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 10:56:26.204192 - 1 day, 4:11:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 12:20:48.071326 - 1 day, 5:36:09 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 13:45:08.867768 - 1 day, 7:00:30 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 15:09:26.956500 - 1 day, 8:24:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 16:33:42.727668 - 1 day, 9:49:04 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 17:57:59.068223 - 1 day, 11:13:20 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 19:22:06.429810 - 1 day, 12:37:28 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 20:46:14.726670 - 1 day, 14:01:36 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 22:10:25.133631 - 1 day, 15:25:46 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 23:34:36.198843 - 1 day, 16:49:58 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 00:58:51.295338 - 1 day, 18:14:13 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 02:23:00.281084 - 1 day, 19:38:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 03:47:10.840719 - 1 day, 21:02:32 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 05:11:19.523128 - 1 day, 22:26:41 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 06:35:38.189352 - 1 day, 23:51:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 07:59:56.706454 - 2 days, 1:15:18 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 09:24:15.659013 - 2 days, 2:39:37 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 10:48:28.291834 - 2 days, 4:03:50 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 12:12:38.789474 - 2 days, 5:28:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 13:36:46.657932 - 2 days, 6:52:08 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.