safe_v2_2nodes/logs/22108862_2.err

Setting Data Parallel size to 16
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-05-04 19:35:46.743782 - 0:00:00 - Signal handler installed.
WARNING 26-05-04 19:35:46.744093 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-05-04 19:35:46.744169 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-05-04 19:35:46.744225 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-05-04 19:35:46.744271 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-05-04 19:35:46.744308 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-05-04 19:35:46.744345 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-05-04 19:35:46.744379 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-05-04 19:35:46.744411 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmputsgpfur
/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/torch/autograd/graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at ../aten/src/ATen/native/cudnn/MHA.cpp:674.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank2]:[W504 19:46:50.697561443 CPUAllocator.cpp:249] Memory block of unknown size was allocated before the profiling started, profiler results will not include the deallocation event
[rank2]: Traceback (most recent call last):
[rank2]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank2]:   File "<frozen runpy>", line 88, in _run_code
[rank2]:   File "/fsx/craffel/lingua_logs/safe_v2_2nodes/code/apps/main/train.py", line 695, in <module>
[rank2]:     main()
[rank2]:   File "/fsx/craffel/lingua_logs/safe_v2_2nodes/code/apps/main/train.py", line 691, in main
[rank2]:     train(cfg)
[rank2]:   File "/fsx/craffel/lingua_logs/safe_v2_2nodes/code/apps/main/train.py", line 581, in train
[rank2]:     from apps.main.eval import (
[rank2]:   File "/fsx/craffel/lingua_logs/safe_v2_2nodes/code/apps/main/eval.py", line 14, in <module>
[rank2]:     from lm_eval import simple_evaluate
[rank2]:   File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
[rank2]:   File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/lm_eval/__init__.py", line 23, in __getattr__
[rank2]:     from .evaluator import simple_evaluate
[rank2]:   File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/lm_eval/evaluator.py", line 17, in <module>
[rank2]:     from lm_eval.evaluator_utils import (
[rank2]:   File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/lm_eval/evaluator_utils.py", line 16, in <module>
[rank2]:     from lm_eval.result_schema import EvalResults, _SampleCount, _TaskMetrics
[rank2]:   File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/lm_eval/result_schema.py", line 110, in <module>
[rank2]:     class _TaskMetrics(TypedDict, Generic[T], extra_items=T):
[rank2]: TypeError: _TypedDictMeta.__new__() got an unexpected keyword argument 'extra_items'