safe_v2_2nodes/logs/22110707_6.err

Setting Data Parallel size to 16
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-05-05 06:44:38.657682 - 0:00:00 - Signal handler installed.
WARNING 26-05-05 06:44:38.657954 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-05-05 06:44:38.658025 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-05-05 06:44:38.658078 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-05-05 06:44:38.658123 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-05-05 06:44:38.658160 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-05-05 06:44:38.658194 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-05-05 06:44:38.658228 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-05-05 06:44:38.658259 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmpcn_fy3my
/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/torch/autograd/graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at ../aten/src/ATen/native/cudnn/MHA.cpp:674.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank6]:[W505 06:50:07.922413021 CPUAllocator.cpp:249] Memory block of unknown size was allocated before the profiling started, profiler results will not include the deallocation event
WARNING 26-05-05 08:14:49.351046 - 1:30:11 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 09:38:44.825855 - 2:54:06 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 11:03:01.703403 - 4:18:23 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 12:27:12.636723 - 5:42:34 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 13:51:31.728120 - 7:06:53 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 15:15:46.908216 - 8:31:08 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 16:40:03.276947 - 9:55:25 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 18:04:21.047104 - 11:19:42 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 19:28:39.000697 - 12:44:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 20:53:00.852742 - 14:08:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 22:17:23.810513 - 15:32:45 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-05 23:41:48.936973 - 16:57:10 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 01:06:00.852287 - 18:21:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 02:30:26.969378 - 19:45:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 03:54:51.993335 - 21:10:13 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 05:19:11.139498 - 22:34:32 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 06:43:18.419203 - 23:58:40 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 08:07:49.915844 - 1 day, 1:23:11 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 09:32:11.975965 - 1 day, 2:47:33 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 10:56:26.204288 - 1 day, 4:11:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 12:20:48.071296 - 1 day, 5:36:09 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 13:45:08.867432 - 1 day, 7:00:30 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 15:09:26.956402 - 1 day, 8:24:48 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 16:33:42.727641 - 1 day, 9:49:04 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 17:57:59.067504 - 1 day, 11:13:20 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 19:22:06.429763 - 1 day, 12:37:28 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 20:46:14.726646 - 1 day, 14:01:36 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 22:10:25.133628 - 1 day, 15:25:46 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-06 23:34:36.198795 - 1 day, 16:49:58 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 00:58:51.295321 - 1 day, 18:14:13 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 02:23:00.280994 - 1 day, 19:38:22 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 03:47:10.840692 - 1 day, 21:02:32 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 05:11:19.523094 - 1 day, 22:26:41 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 06:35:38.189377 - 1 day, 23:51:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 07:59:56.706672 - 2 days, 1:15:18 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 09:24:15.659160 - 2 days, 2:39:37 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 10:48:28.291770 - 2 days, 4:03:50 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 12:12:38.789443 - 2 days, 5:28:00 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.
WARNING 26-05-07 13:36:46.657853 - 2 days, 6:52:08 - Error occurred while importing eval args: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'.Skipping eval launch.