full-denoise training:   0%|                                                                                                        | 0/500000 [00:00<?, ?it/s][rank0]:W0420 07:14:09.099000 1775411 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
full-denoise training:   0%|                                                                  | 525/500000 [02:57<35:01:15,  3.96it/s, leaf=4.0747, lr=7.9e-05]Traceback (most recent call last):
step=     0 | total=11.0243 | leaf=11.0243 | ancestor=0.0000 | lr=0.00e+00 | t=45.6s
step=   100 | total=9.1935 | leaf=9.1935 | ancestor=0.0000 | lr=1.50e-05 | t=25.1s
step=   200 | total=6.6110 | leaf=6.6110 | ancestor=0.0000 | lr=3.00e-05 | t=25.2s
step=   300 | total=5.0653 | leaf=5.0653 | ancestor=0.0000 | lr=4.50e-05 | t=25.2s
step=   400 | total=4.4472 | leaf=4.4472 | ancestor=0.0000 | lr=6.00e-05 | t=25.2s
step=   500 | total=4.1173 | leaf=4.1173 | ancestor=0.0000 | lr=7.50e-05 | t=25.3s
  File "/home/sunhc/diffusion/sad/scripts/train.py", line 579, in <module>
    main()
  File "/home/sunhc/diffusion/sad/scripts/train.py", line 495, in main
    loss.backward()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train.py", line 579, in <module>
[rank0]:     main()
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train.py", line 495, in main
[rank0]:     loss.backward()
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/_tensor.py", line 626, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/__init__.py", line 347, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: KeyboardInterrupt