block-ar training:   0%|                                                                                                            | 0/5000 [00:00<?, ?it/s][rank0]:W0417 10:56:49.197000 3238869 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
block-ar training:   1%|▋                                                                         | 47/5000 [00:55<29:50,  2.77it/s, leaf=7.2638, lr=2.8e-04]Traceback (most recent call last):
step=     0 | total=11.0256 | leaf=11.0256 | ancestor=0.0000 | lr=0.00e+00 | t=39.3s
  File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 545, in <module>
    main()
  File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 471, in main
    loss.backward()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 545, in <module>
[rank0]:     main()
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train_block_ar.py", line 471, in main
[rank0]:     loss.backward()
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/_tensor.py", line 626, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/__init__.py", line 347, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: KeyboardInterrupt