block-ar training:   0%|                                                                                                          | 0/500000 [00:00<?, ?it/s][rank0]:W0417 11:04:00.322000 3293200 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
block-ar training:   1%|▍                                                                  | 2855/500000 [17:57<50:15:55,  2.75it/s, leaf=2.4575, lr=3.0e-04]Traceback (most recent call last):
step=     0 | total=11.0256 | leaf=11.0256 | ancestor=0.0000 | lr=0.00e+00 | t=39.1s
step=   100 | total=9.2571 | leaf=9.2571 | ancestor=0.0000 | lr=1.50e-05 | t=36.1s
step=   200 | total=6.8835 | leaf=6.8835 | ancestor=0.0000 | lr=3.00e-05 | t=36.3s
step=   300 | total=5.2697 | leaf=5.2697 | ancestor=0.0000 | lr=4.50e-05 | t=36.3s
step=   400 | total=4.7753 | leaf=4.7753 | ancestor=0.0000 | lr=6.00e-05 | t=36.3s
step=   500 | total=4.3787 | leaf=4.3787 | ancestor=0.0000 | lr=7.50e-05 | t=36.4s
step=   600 | total=4.0279 | leaf=4.0279 | ancestor=0.0000 | lr=9.00e-05 | t=36.4s
step=   700 | total=3.7670 | leaf=3.7670 | ancestor=0.0000 | lr=1.05e-04 | t=36.4s
step=   800 | total=3.5210 | leaf=3.5210 | ancestor=0.0000 | lr=1.20e-04 | t=36.4s
step=   900 | total=3.4035 | leaf=3.4035 | ancestor=0.0000 | lr=1.35e-04 | t=36.4s
step=  1000 | total=3.2758 | leaf=3.2758 | ancestor=0.0000 | lr=1.50e-04 | t=36.4s
step=  1100 | total=3.1991 | leaf=3.1991 | ancestor=0.0000 | lr=1.65e-04 | t=36.4s
step=  1200 | total=3.1314 | leaf=3.1314 | ancestor=0.0000 | lr=1.80e-04 | t=36.4s
step=  1300 | total=2.9663 | leaf=2.9663 | ancestor=0.0000 | lr=1.95e-04 | t=36.4s
step=  1400 | total=2.9252 | leaf=2.9252 | ancestor=0.0000 | lr=2.10e-04 | t=36.4s
step=  1500 | total=2.9277 | leaf=2.9277 | ancestor=0.0000 | lr=2.25e-04 | t=36.4s
step=  1600 | total=2.8112 | leaf=2.8112 | ancestor=0.0000 | lr=2.40e-04 | t=36.5s
step=  1700 | total=2.7337 | leaf=2.7337 | ancestor=0.0000 | lr=2.55e-04 | t=36.4s
step=  1800 | total=2.7037 | leaf=2.7037 | ancestor=0.0000 | lr=2.70e-04 | t=36.4s
step=  1900 | total=2.6052 | leaf=2.6052 | ancestor=0.0000 | lr=2.85e-04 | t=36.4s
step=  2000 | total=2.6613 | leaf=2.6613 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2100 | total=2.7009 | leaf=2.7009 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2200 | total=2.5992 | leaf=2.5992 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2300 | total=2.5785 | leaf=2.5785 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2400 | total=2.5328 | leaf=2.5328 | ancestor=0.0000 | lr=3.00e-04 | t=36.5s
step=  2500 | total=2.5404 | leaf=2.5404 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2600 | total=2.4896 | leaf=2.4896 | ancestor=0.0000 | lr=3.00e-04 | t=36.5s
step=  2700 | total=2.5038 | leaf=2.5038 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
step=  2800 | total=2.4831 | leaf=2.4831 | ancestor=0.0000 | lr=3.00e-04 | t=36.4s
  File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 550, in <module>
    dist.destroy_process_group()
    ^^^^^^
  File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 472, in main
    "attention_mask": full_batch["attention_mask"].to(device, non_blocking=True),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 264, in block_ar_step
    )  # [B, L, V]

  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 550, in <module>
[rank0]:     dist.destroy_process_group()
[rank0]: ^^^^^^
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 472, in main
[rank0]:     "attention_mask": full_batch["attention_mask"].to(device, non_blocking=True),
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 264, in block_ar_step
[rank0]:     )  # [B, L, V]
[rank0]:
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: KeyboardInterrupt