full-denoise training:   0%|                                                                                                | 0/500000 [00:00<?, ?it/s][rank0]:W0420 11:02:31.942000 2226207 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
full-denoise training:   1%|▎                                                        | 3101/500000 [13:31<34:50:59,  3.96it/s, leaf=3.6741, lr=3.0e-04]
step=     0 | total=11.0214 | leaf=11.0214 | ancestor=0.0000 | lr=0.00e+00 | t=21.8s
step=   100 | total=9.2237 | leaf=9.2237 | ancestor=0.0000 | lr=1.50e-05 | t=25.0s
step=   200 | total=7.4041 | leaf=7.4041 | ancestor=0.0000 | lr=3.00e-05 | t=25.0s
step=   300 | total=7.1109 | leaf=7.1109 | ancestor=0.0000 | lr=4.50e-05 | t=25.1s
step=   400 | total=6.5568 | leaf=6.5568 | ancestor=0.0000 | lr=6.00e-05 | t=25.2s
step=   500 | total=6.2397 | leaf=6.2397 | ancestor=0.0000 | lr=7.50e-05 | t=25.2s
step=   600 | total=5.9209 | leaf=5.9209 | ancestor=0.0000 | lr=9.00e-05 | t=25.2s
step=   700 | total=5.4500 | leaf=5.4500 | ancestor=0.0000 | lr=1.05e-04 | t=25.2s
step=   800 | total=5.2818 | leaf=5.2818 | ancestor=0.0000 | lr=1.20e-04 | t=25.2s
step=   900 | total=4.8201 | leaf=4.8201 | ancestor=0.0000 | lr=1.35e-04 | t=25.8s
step=  1000 | total=4.8005 | leaf=4.8005 | ancestor=0.0000 | lr=1.50e-04 | t=25.2s
step=  1100 | total=4.6132 | leaf=4.6132 | ancestor=0.0000 | lr=1.65e-04 | t=25.2s
step=  1200 | total=4.4769 | leaf=4.4769 | ancestor=0.0000 | lr=1.80e-04 | t=25.2s
step=  1300 | total=4.4299 | leaf=4.4299 | ancestor=0.0000 | lr=1.95e-04 | t=25.2s
step=  1400 | total=4.3169 | leaf=4.3169 | ancestor=0.0000 | lr=2.10e-04 | t=25.2s
step=  1500 | total=4.3255 | leaf=4.3255 | ancestor=0.0000 | lr=2.25e-04 | t=25.2s
step=  1600 | total=4.0984 | leaf=4.0984 | ancestor=0.0000 | lr=2.40e-04 | t=25.2s
step=  1700 | total=4.0924 | leaf=4.0924 | ancestor=0.0000 | lr=2.55e-04 | t=25.2s
step=  1800 | total=3.9297 | leaf=3.9297 | ancestor=0.0000 | lr=2.70e-04 | t=25.2s
step=  1900 | total=3.8761 | leaf=3.8761 | ancestor=0.0000 | lr=2.85e-04 | t=25.2s
step=  2000 | total=3.9243 | leaf=3.9243 | ancestor=0.0000 | lr=3.00e-04 | t=25.5s
step=  2100 | total=3.9088 | leaf=3.9088 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  2200 | total=3.8014 | leaf=3.8014 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  2300 | total=3.7209 | leaf=3.7209 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  2400 | total=3.7611 | leaf=3.7611 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  2500 | total=3.7441 | leaf=3.7441 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  2600 | total=3.6873 | leaf=3.6873 | ancestor=0.0000 | lr=3.00e-04 | t=31.9s
step=  2700 | total=3.6602 | leaf=3.6602 | ancestor=0.0000 | lr=3.00e-04 | t=26.0s
step=  2800 | total=3.7666 | leaf=3.7666 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  2900 | total=3.7145 | leaf=3.7145 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3000 | total=3.7486 | leaf=3.7486 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3100 | total=3.6741 | leaf=3.6741 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3200 | total=3.7744 | leaf=3.7744 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3300 | total=3.7501 | leaf=3.7501 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3400 | total=3.7052 | leaf=3.7052 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3500 | total=3.5933 | leaf=3.5933 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3600 | total=3.7499 | leaf=3.7499 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3700 | total=3.6265 | leaf=3.6265 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3800 | total=3.5721 | leaf=3.5721 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  3900 | total=3.6155 | leaf=3.6155 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  4000 | total=3.6021 | leaf=3.6021 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  4100 | total=3.6329 | leaf=3.6329 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  4200 | total=3.4786 | leaf=3.4786 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  4300 | total=3.5117 | leaf=3.5117 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  4400 | total=3.4163 | leaf=3.4163 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  4500 | total=3.5415 | leaf=3.5415 | ancestor=0.0000 | lr=3.00e-04 | t=29.2s
step=  4600 | total=3.5517 | leaf=3.5517 | ancestor=0.0000 | lr=3.00e-04 | t=35.0s
step=  4700 | total=3.5124 | leaf=3.5124 | ancestor=0.0000 | lr=3.00e-04 | t=35.1s
step=  4800 | total=3.4378 | leaf=3.4378 | ancestor=0.0000 | lr=3.00e-04 | t=35.2s
step=  4900 | total=3.5203 | leaf=3.5203 | ancestor=0.0000 | lr=3.00e-04 | t=35.2s
step=  5000 | total=3.5932 | leaf=3.5932 | ancestor=0.0000 | lr=3.00e-04 | t=35.1s
  torch._dynamo.utils.warn_once(msg)
  VAL | loss_leaf=3.4748 | loss_ancestor=0.0000 | loss_total=3.4748
full-denoise training:   2%|▉                                                        | 8201/500000 [38:06<34:32:59,  3.95it/s, leaf=3.3876, lr=3.0e-04]
step=  5100 | total=3.5437 | leaf=3.5437 | ancestor=0.0000 | lr=3.00e-04 | t=55.0s
step=  5200 | total=3.3895 | leaf=3.3895 | ancestor=0.0000 | lr=3.00e-04 | t=32.3s
step=  5300 | total=3.4342 | leaf=3.4342 | ancestor=0.0000 | lr=3.00e-04 | t=25.4s
step=  5400 | total=3.5141 | leaf=3.5141 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  5500 | total=3.5244 | leaf=3.5244 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  5600 | total=3.4166 | leaf=3.4166 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  5700 | total=3.5063 | leaf=3.5063 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  5800 | total=3.4510 | leaf=3.4510 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  5900 | total=3.3643 | leaf=3.3643 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6000 | total=3.4608 | leaf=3.4608 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6100 | total=3.4578 | leaf=3.4578 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6200 | total=3.4017 | leaf=3.4017 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6300 | total=3.3813 | leaf=3.3813 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6400 | total=3.4445 | leaf=3.4445 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6500 | total=3.3977 | leaf=3.3977 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  6600 | total=3.4680 | leaf=3.4680 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  6700 | total=3.3196 | leaf=3.3196 | ancestor=0.0000 | lr=3.00e-04 | t=30.5s
step=  6800 | total=3.3828 | leaf=3.3828 | ancestor=0.0000 | lr=3.00e-04 | t=38.0s
step=  6900 | total=3.4396 | leaf=3.4396 | ancestor=0.0000 | lr=3.00e-04 | t=38.1s
step=  7000 | total=3.3581 | leaf=3.3581 | ancestor=0.0000 | lr=3.00e-04 | t=37.8s
step=  7100 | total=3.3953 | leaf=3.3953 | ancestor=0.0000 | lr=3.00e-04 | t=37.8s
step=  7200 | total=3.3414 | leaf=3.3414 | ancestor=0.0000 | lr=3.00e-04 | t=37.9s
step=  7300 | total=3.3144 | leaf=3.3144 | ancestor=0.0000 | lr=3.00e-04 | t=38.1s
step=  7400 | total=3.3657 | leaf=3.3657 | ancestor=0.0000 | lr=3.00e-04 | t=39.8s
step=  7500 | total=3.2703 | leaf=3.2703 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  7600 | total=3.3674 | leaf=3.3674 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  7700 | total=3.4416 | leaf=3.4416 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  7800 | total=3.2971 | leaf=3.2971 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  7900 | total=3.3517 | leaf=3.3517 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8000 | total=3.3473 | leaf=3.3473 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8100 | total=3.4161 | leaf=3.4161 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8200 | total=3.3876 | leaf=3.3876 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8300 | total=3.4620 | leaf=3.4620 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8400 | total=3.4335 | leaf=3.4335 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8500 | total=3.3688 | leaf=3.3688 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8600 | total=3.4655 | leaf=3.4655 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8700 | total=3.2878 | leaf=3.2878 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8800 | total=3.5483 | leaf=3.5483 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  8900 | total=3.4507 | leaf=3.4507 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9000 | total=3.2938 | leaf=3.2938 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9100 | total=3.2965 | leaf=3.2965 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9200 | total=3.2935 | leaf=3.2935 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9300 | total=3.2921 | leaf=3.2921 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9400 | total=3.2421 | leaf=3.2421 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9500 | total=3.2382 | leaf=3.2382 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  9600 | total=3.3259 | leaf=3.3259 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9700 | total=3.3433 | leaf=3.3433 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step=  9800 | total=3.2652 | leaf=3.2652 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step=  9900 | total=3.4807 | leaf=3.4807 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 10000 | total=3.3404 | leaf=3.3404 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
  VAL | loss_leaf=3.3210 | loss_ancestor=0.0000 | loss_total=3.3210
  Saved checkpoint: outputs/sad_new/ckpt_10000.pt
step= 10100 | total=3.3546 | leaf=3.3546 | ancestor=0.0000 | lr=3.00e-04 | t=40.1s
step= 10200 | total=3.2498 | leaf=3.2498 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10300 | total=3.3691 | leaf=3.3691 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10400 | total=3.3702 | leaf=3.3702 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10500 | total=3.2674 | leaf=3.2674 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10600 | total=3.2377 | leaf=3.2377 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10700 | total=3.2975 | leaf=3.2975 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10800 | total=3.3108 | leaf=3.3108 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 10900 | total=3.3743 | leaf=3.3743 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 11000 | total=3.3789 | leaf=3.3789 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11100 | total=3.3482 | leaf=3.3482 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11200 | total=3.3281 | leaf=3.3281 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11300 | total=3.3218 | leaf=3.3218 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 11400 | total=3.2368 | leaf=3.2368 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11500 | total=3.4175 | leaf=3.4175 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11600 | total=3.2666 | leaf=3.2666 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11700 | total=3.2828 | leaf=3.2828 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11800 | total=3.2481 | leaf=3.2481 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 11900 | total=3.2677 | leaf=3.2677 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 12000 | total=3.2846 | leaf=3.2846 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 12100 | total=3.2809 | leaf=3.2809 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 12200 | total=3.2023 | leaf=3.2023 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 12300 | total=3.2578 | leaf=3.2578 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 12400 | total=3.1575 | leaf=3.1575 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 12500 | total=3.1901 | leaf=3.1901 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 12600 | total=3.3181 | leaf=3.3181 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 12700 | total=3.2068 | leaf=3.2068 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 12800 | total=3.2020 | leaf=3.2020 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 12900 | total=3.1308 | leaf=3.1308 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 13000 | total=3.2645 | leaf=3.2645 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 13100 | total=3.0602 | leaf=3.0602 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13200 | total=3.1954 | leaf=3.1954 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13300 | total=3.2361 | leaf=3.2361 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13400 | total=3.1889 | leaf=3.1889 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13500 | total=3.1953 | leaf=3.1953 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13600 | total=3.2798 | leaf=3.2798 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13700 | total=3.1551 | leaf=3.1551 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13800 | total=3.2281 | leaf=3.2281 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 13900 | total=3.1193 | leaf=3.1193 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 14000 | total=3.2154 | leaf=3.2154 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 14100 | total=3.1710 | leaf=3.1710 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 14200 | total=3.1717 | leaf=3.1717 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14300 | total=3.2034 | leaf=3.2034 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14400 | total=3.0951 | leaf=3.0951 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14500 | total=3.1057 | leaf=3.1057 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14600 | total=3.1210 | leaf=3.1210 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14700 | total=3.1972 | leaf=3.1972 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14800 | total=3.0965 | leaf=3.0965 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 14900 | total=3.0895 | leaf=3.0895 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 15000 | total=3.2244 | leaf=3.2244 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
  VAL | loss_leaf=3.1538 | loss_ancestor=0.0000 | loss_total=3.1538
step= 15100 | total=3.0544 | leaf=3.0544 | ancestor=0.0000 | lr=3.00e-04 | t=29.8s
step= 15200 | total=3.2268 | leaf=3.2268 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 15300 | total=3.2518 | leaf=3.2518 | ancestor=0.0000 | lr=3.00e-04 | t=25.2s
step= 15400 | total=3.1401 | leaf=3.1401 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 15500 | total=3.2060 | leaf=3.2060 | ancestor=0.0000 | lr=3.00e-04 | t=50.4s
step= 15600 | total=3.2063 | leaf=3.2063 | ancestor=0.0000 | lr=3.00e-04 | t=25.3s
step= 15700 | total=3.2735 | leaf=3.2735 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 15800 | total=3.0863 | leaf=3.0863 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 15900 | total=3.1721 | leaf=3.1721 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16000 | total=3.1487 | leaf=3.1487 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16100 | total=3.0039 | leaf=3.0039 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16200 | total=3.2593 | leaf=3.2593 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16300 | total=3.2720 | leaf=3.2720 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16400 | total=3.1846 | leaf=3.1846 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16500 | total=3.1506 | leaf=3.1506 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16600 | total=3.0642 | leaf=3.0642 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16700 | total=3.0712 | leaf=3.0712 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16800 | total=3.1939 | leaf=3.1939 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 16900 | total=3.1484 | leaf=3.1484 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 17000 | total=3.1316 | leaf=3.1316 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 17100 | total=2.9943 | leaf=2.9943 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 17200 | total=3.1431 | leaf=3.1431 | ancestor=0.0000 | lr=2.99e-04 | t=25.5s
step= 17300 | total=3.0771 | leaf=3.0771 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 17400 | total=3.1609 | leaf=3.1609 | ancestor=0.0000 | lr=2.99e-04 | t=25.5s
step= 17500 | total=3.0894 | leaf=3.0894 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 17600 | total=2.9487 | leaf=2.9487 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 17700 | total=3.1859 | leaf=3.1859 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 17800 | total=3.1126 | leaf=3.1126 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 17900 | total=3.1230 | leaf=3.1230 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 18000 | total=3.0351 | leaf=3.0351 | ancestor=0.0000 | lr=2.99e-04 | t=25.5s
step= 18100 | total=3.1393 | leaf=3.1393 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 18200 | total=3.2122 | leaf=3.2122 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 18300 | total=3.1003 | leaf=3.1003 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 18400 | total=3.0321 | leaf=3.0321 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 18500 | total=3.0267 | leaf=3.0267 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 18600 | total=3.1515 | leaf=3.1515 | ancestor=0.0000 | lr=2.99e-04 | t=25.5s
step= 18700 | total=3.1392 | leaf=3.1392 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 18800 | total=3.1016 | leaf=3.1016 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 18900 | total=3.0369 | leaf=3.0369 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19000 | total=3.0396 | leaf=3.0396 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19100 | total=3.0491 | leaf=3.0491 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19200 | total=2.9415 | leaf=2.9415 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 19300 | total=3.0430 | leaf=3.0430 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19400 | total=3.0329 | leaf=3.0329 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19500 | total=3.0653 | leaf=3.0653 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19600 | total=3.0324 | leaf=3.0324 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 19700 | total=3.2003 | leaf=3.2003 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19800 | total=3.1313 | leaf=3.1313 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 19900 | total=2.9890 | leaf=2.9890 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 20000 | total=3.1194 | leaf=3.1194 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
  VAL | loss_leaf=3.0731 | loss_ancestor=0.0000 | loss_total=3.0731
  Saved checkpoint: outputs/sad_new/ckpt_20000.pt
step= 20100 | total=3.0406 | leaf=3.0406 | ancestor=0.0000 | lr=2.99e-04 | t=40.5s
step= 20200 | total=3.0688 | leaf=3.0688 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20300 | total=3.0162 | leaf=3.0162 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20400 | total=3.0160 | leaf=3.0160 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20500 | total=3.0127 | leaf=3.0127 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20600 | total=3.0022 | leaf=3.0022 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20700 | total=3.0416 | leaf=3.0416 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20800 | total=3.0833 | leaf=3.0833 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 20900 | total=3.1247 | leaf=3.1247 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 21000 | total=3.0256 | leaf=3.0256 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 21100 | total=3.0663 | leaf=3.0663 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 21200 | total=3.0004 | leaf=3.0004 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 21300 | total=3.1635 | leaf=3.1635 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 21400 | total=3.0964 | leaf=3.0964 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 21500 | total=3.0112 | leaf=3.0112 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 21600 | total=3.1084 | leaf=3.1084 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 21700 | total=3.1499 | leaf=3.1499 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 21800 | total=3.1297 | leaf=3.1297 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 21900 | total=3.1145 | leaf=3.1145 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22000 | total=2.9626 | leaf=2.9626 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22100 | total=3.0667 | leaf=3.0667 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22200 | total=2.9159 | leaf=2.9159 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22300 | total=3.1230 | leaf=3.1230 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22400 | total=2.9670 | leaf=2.9670 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22500 | total=3.0457 | leaf=3.0457 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22600 | total=3.1071 | leaf=3.1071 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22700 | total=3.0663 | leaf=3.0663 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22800 | total=2.9850 | leaf=2.9850 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 22900 | total=2.9161 | leaf=2.9161 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23000 | total=3.1537 | leaf=3.1537 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23100 | total=3.1620 | leaf=3.1620 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23200 | total=3.0340 | leaf=3.0340 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23300 | total=3.0901 | leaf=3.0901 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23400 | total=3.0502 | leaf=3.0502 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23500 | total=3.0568 | leaf=3.0568 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23600 | total=3.0492 | leaf=3.0492 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23700 | total=2.9927 | leaf=2.9927 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23800 | total=2.9358 | leaf=2.9358 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 23900 | total=3.0147 | leaf=3.0147 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24000 | total=3.1459 | leaf=3.1459 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24100 | total=2.9956 | leaf=2.9956 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24200 | total=3.0410 | leaf=3.0410 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24300 | total=2.9605 | leaf=2.9605 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24400 | total=2.9235 | leaf=2.9235 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24500 | total=2.8614 | leaf=2.8614 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24600 | total=2.9865 | leaf=2.9865 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24700 | total=3.1837 | leaf=3.1837 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24800 | total=2.9821 | leaf=2.9821 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 24900 | total=2.9082 | leaf=2.9082 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 25000 | total=2.9617 | leaf=2.9617 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
  VAL | loss_leaf=2.9972 | loss_ancestor=0.0000 | loss_total=2.9972
step= 25100 | total=2.9522 | leaf=2.9522 | ancestor=0.0000 | lr=2.99e-04 | t=29.8s
step= 25200 | total=2.9819 | leaf=2.9819 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 25300 | total=3.1124 | leaf=3.1124 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 25400 | total=2.9668 | leaf=2.9668 | ancestor=0.0000 | lr=2.99e-04 | t=25.2s
step= 25500 | total=2.9270 | leaf=2.9270 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 25600 | total=2.9093 | leaf=2.9093 | ancestor=0.0000 | lr=2.99e-04 | t=25.3s
step= 25700 | total=2.9623 | leaf=2.9623 | ancestor=0.0000 | lr=2.98e-04 | t=25.3s
step= 25800 | total=3.0778 | leaf=3.0778 | ancestor=0.0000 | lr=2.98e-04 | t=25.3s
step= 25900 | total=2.9701 | leaf=2.9701 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26000 | total=2.9792 | leaf=2.9792 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26100 | total=2.8969 | leaf=2.8969 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26200 | total=2.9406 | leaf=2.9406 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26300 | total=3.0110 | leaf=3.0110 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26400 | total=2.8757 | leaf=2.8757 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26500 | total=3.0232 | leaf=3.0232 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26600 | total=2.9233 | leaf=2.9233 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26700 | total=2.9912 | leaf=2.9912 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26800 | total=2.9754 | leaf=2.9754 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 26900 | total=2.9274 | leaf=2.9274 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27000 | total=3.0147 | leaf=3.0147 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27100 | total=2.8821 | leaf=2.8821 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27200 | total=3.0007 | leaf=3.0007 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27300 | total=2.8985 | leaf=2.8985 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27400 | total=2.8985 | leaf=2.8985 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27500 | total=2.9936 | leaf=2.9936 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27600 | total=2.9468 | leaf=2.9468 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27700 | total=2.9357 | leaf=2.9357 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27800 | total=2.9866 | leaf=2.9866 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 27900 | total=2.8513 | leaf=2.8513 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28000 | total=2.9831 | leaf=2.9831 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28100 | total=2.9239 | leaf=2.9239 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28200 | total=3.0686 | leaf=3.0686 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28300 | total=3.0998 | leaf=3.0998 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28400 | total=3.0430 | leaf=3.0430 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28500 | total=2.9748 | leaf=2.9748 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28600 | total=2.8785 | leaf=2.8785 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28700 | total=3.0248 | leaf=3.0248 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28800 | total=3.0809 | leaf=3.0809 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 28900 | total=2.9881 | leaf=2.9881 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29000 | total=2.9008 | leaf=2.9008 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29100 | total=2.8662 | leaf=2.8662 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29200 | total=2.8609 | leaf=2.8609 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29300 | total=3.0067 | leaf=3.0067 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29400 | total=3.0289 | leaf=3.0289 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29500 | total=2.9565 | leaf=2.9565 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29600 | total=3.0115 | leaf=3.0115 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29700 | total=3.0453 | leaf=3.0453 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29800 | total=2.9781 | leaf=2.9781 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 29900 | total=2.9574 | leaf=2.9574 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30000 | total=2.9010 | leaf=2.9010 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
  VAL | loss_leaf=2.9506 | loss_ancestor=0.0000 | loss_total=2.9506
  Saved checkpoint: outputs/sad_new/ckpt_30000.pt
step= 30100 | total=2.9762 | leaf=2.9762 | ancestor=0.0000 | lr=2.98e-04 | t=40.0s
step= 30200 | total=2.9069 | leaf=2.9069 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30300 | total=2.8901 | leaf=2.8901 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30400 | total=2.9027 | leaf=2.9027 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30500 | total=2.8711 | leaf=2.8711 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30600 | total=3.0156 | leaf=3.0156 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30700 | total=2.9854 | leaf=2.9854 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30800 | total=2.9162 | leaf=2.9162 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 30900 | total=2.9782 | leaf=2.9782 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31000 | total=2.8504 | leaf=2.8504 | ancestor=0.0000 | lr=2.98e-04 | t=25.4s
step= 31100 | total=2.9082 | leaf=2.9082 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31200 | total=3.0451 | leaf=3.0451 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31300 | total=2.9818 | leaf=2.9818 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31400 | total=2.9764 | leaf=2.9764 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31500 | total=2.9588 | leaf=2.9588 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31600 | total=2.8851 | leaf=2.8851 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31700 | total=3.0096 | leaf=3.0096 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31800 | total=2.9168 | leaf=2.9168 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 31900 | total=2.9499 | leaf=2.9499 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32000 | total=2.9082 | leaf=2.9082 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32100 | total=2.9507 | leaf=2.9507 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32200 | total=2.8827 | leaf=2.8827 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32300 | total=2.9369 | leaf=2.9369 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32400 | total=2.9657 | leaf=2.9657 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32500 | total=2.9792 | leaf=2.9792 | ancestor=0.0000 | lr=2.98e-04 | t=25.2s
step= 32600 | total=2.9578 | leaf=2.9578 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 32700 | total=2.9765 | leaf=2.9765 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 32800 | total=2.8749 | leaf=2.8749 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 32900 | total=2.9521 | leaf=2.9521 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33000 | total=2.8547 | leaf=2.8547 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33100 | total=2.8590 | leaf=2.8590 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33200 | total=2.9889 | leaf=2.9889 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33300 | total=2.8185 | leaf=2.8185 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33400 | total=3.0279 | leaf=3.0279 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33500 | total=2.7595 | leaf=2.7595 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33600 | total=2.8655 | leaf=2.8655 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33700 | total=2.9571 | leaf=2.9571 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33800 | total=2.9997 | leaf=2.9997 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 33900 | total=2.7370 | leaf=2.7370 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34000 | total=2.9047 | leaf=2.9047 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34100 | total=2.9721 | leaf=2.9721 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34200 | total=2.9147 | leaf=2.9147 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34300 | total=2.9669 | leaf=2.9669 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34400 | total=2.8655 | leaf=2.8655 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34500 | total=2.9742 | leaf=2.9742 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34600 | total=2.9347 | leaf=2.9347 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34700 | total=2.9496 | leaf=2.9496 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34800 | total=2.8603 | leaf=2.8603 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 34900 | total=2.8780 | leaf=2.8780 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35000 | total=2.9980 | leaf=2.9980 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
  VAL | loss_leaf=2.9088 | loss_ancestor=0.0000 | loss_total=2.9088
step= 35100 | total=2.9497 | leaf=2.9497 | ancestor=0.0000 | lr=2.97e-04 | t=29.8s
step= 35200 | total=2.8837 | leaf=2.8837 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35300 | total=2.9248 | leaf=2.9248 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35400 | total=2.8670 | leaf=2.8670 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35500 | total=2.8870 | leaf=2.8870 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35600 | total=2.9052 | leaf=2.9052 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35700 | total=2.8047 | leaf=2.8047 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35800 | total=2.9084 | leaf=2.9084 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 35900 | total=2.9215 | leaf=2.9215 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36000 | total=2.9370 | leaf=2.9370 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36100 | total=2.8487 | leaf=2.8487 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36200 | total=2.8000 | leaf=2.8000 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36300 | total=2.9258 | leaf=2.9258 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36400 | total=2.9676 | leaf=2.9676 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36500 | total=2.9208 | leaf=2.9208 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36600 | total=2.8301 | leaf=2.8301 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36700 | total=2.9390 | leaf=2.9390 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36800 | total=2.9328 | leaf=2.9328 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 36900 | total=2.9138 | leaf=2.9138 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37000 | total=2.8586 | leaf=2.8586 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37100 | total=2.8851 | leaf=2.8851 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37200 | total=2.8336 | leaf=2.8336 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37300 | total=2.9325 | leaf=2.9325 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37400 | total=2.8580 | leaf=2.8580 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37500 | total=2.9069 | leaf=2.9069 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37600 | total=2.8284 | leaf=2.8284 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37700 | total=2.9428 | leaf=2.9428 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37800 | total=2.8887 | leaf=2.8887 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 37900 | total=2.9002 | leaf=2.9002 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 38000 | total=2.8885 | leaf=2.8885 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 38100 | total=2.8718 | leaf=2.8718 | ancestor=0.0000 | lr=2.97e-04 | t=25.2s
step= 38200 | total=2.8985 | leaf=2.8985 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38300 | total=2.9056 | leaf=2.9056 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38400 | total=2.8768 | leaf=2.8768 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38500 | total=2.8216 | leaf=2.8216 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38600 | total=2.8862 | leaf=2.8862 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38700 | total=7.2650 | leaf=7.2650 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38800 | total=9.5553 | leaf=9.5553 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 38900 | total=13.2414 | leaf=13.2414 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 39000 | total=8.1947 | leaf=8.1947 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 39100 | total=7.4666 | leaf=7.4666 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39200 | total=7.6086 | leaf=7.6086 | ancestor=0.0000 | lr=2.96e-04 | t=25.2s
step= 39300 | total=7.6530 | leaf=7.6530 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39400 | total=7.7629 | leaf=7.7629 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39500 | total=7.7041 | leaf=7.7041 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39600 | total=7.7121 | leaf=7.7121 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39700 | total=7.7596 | leaf=7.7596 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39800 | total=7.8762 | leaf=7.8762 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 39900 | total=7.8072 | leaf=7.8072 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40000 | total=7.7928 | leaf=7.7928 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
  VAL | loss_leaf=7.8566 | loss_ancestor=0.0000 | loss_total=7.8566
  Saved checkpoint: outputs/sad_new/ckpt_40000.pt
step= 40100 | total=7.8504 | leaf=7.8504 | ancestor=0.0000 | lr=2.96e-04 | t=36.6s
step= 40200 | total=7.9569 | leaf=7.9569 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40300 | total=7.9929 | leaf=7.9929 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40400 | total=7.9284 | leaf=7.9284 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40500 | total=8.0211 | leaf=8.0211 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40600 | total=7.9990 | leaf=7.9990 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40700 | total=8.1780 | leaf=8.1780 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40800 | total=8.1894 | leaf=8.1894 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 40900 | total=8.2163 | leaf=8.2163 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 41000 | total=8.0698 | leaf=8.0698 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 41100 | total=8.1528 | leaf=8.1528 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 41200 | total=8.1718 | leaf=8.1718 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 41300 | total=9.2637 | leaf=9.2637 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 41400 | total=7.8985 | leaf=7.8985 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 41500 | total=7.6503 | leaf=7.6503 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 41600 | total=7.5030 | leaf=7.5030 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 41700 | total=7.4248 | leaf=7.4248 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 41800 | total=7.6491 | leaf=7.6491 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 41900 | total=7.5745 | leaf=7.5745 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42000 | total=7.4889 | leaf=7.4889 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 42100 | total=7.5917 | leaf=7.5917 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42200 | total=7.4550 | leaf=7.4550 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42300 | total=7.4906 | leaf=7.4906 | ancestor=0.0000 | lr=2.96e-04 | t=25.1s
step= 42400 | total=7.5844 | leaf=7.5844 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42500 | total=7.4961 | leaf=7.4961 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42600 | total=7.4459 | leaf=7.4459 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42700 | total=7.8283 | leaf=7.8283 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42800 | total=7.5669 | leaf=7.5669 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 42900 | total=7.5305 | leaf=7.5305 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 43000 | total=7.4679 | leaf=7.4679 | ancestor=0.0000 | lr=2.96e-04 | t=25.0s
step= 43100 | total=7.4455 | leaf=7.4455 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 43200 | total=7.5184 | leaf=7.5184 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43300 | total=7.4544 | leaf=7.4544 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43400 | total=7.4910 | leaf=7.4910 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43500 | total=7.4160 | leaf=7.4160 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 43600 | total=7.3589 | leaf=7.3589 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43700 | total=7.4001 | leaf=7.4001 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43800 | total=7.3581 | leaf=7.3581 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 43900 | total=7.4714 | leaf=7.4714 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44000 | total=7.3329 | leaf=7.3329 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 44100 | total=7.4336 | leaf=7.4336 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44200 | total=7.3477 | leaf=7.3477 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 44300 | total=7.3139 | leaf=7.3139 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44400 | total=7.3275 | leaf=7.3275 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44500 | total=7.4131 | leaf=7.4131 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 44600 | total=7.3144 | leaf=7.3144 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44700 | total=7.3398 | leaf=7.3398 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44800 | total=7.3014 | leaf=7.3014 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 44900 | total=7.2794 | leaf=7.2794 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45000 | total=7.3067 | leaf=7.3067 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
  VAL | loss_leaf=7.3098 | loss_ancestor=0.0000 | loss_total=7.3098
step= 45100 | total=7.3626 | leaf=7.3626 | ancestor=0.0000 | lr=2.95e-04 | t=29.6s
step= 45200 | total=7.3193 | leaf=7.3193 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45300 | total=7.3873 | leaf=7.3873 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45400 | total=7.2356 | leaf=7.2356 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45500 | total=7.2904 | leaf=7.2904 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45600 | total=7.2188 | leaf=7.2188 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 45700 | total=7.2864 | leaf=7.2864 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 45800 | total=7.2962 | leaf=7.2962 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 45900 | total=7.2639 | leaf=7.2639 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46000 | total=7.2115 | leaf=7.2115 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46100 | total=7.2972 | leaf=7.2972 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46200 | total=7.1134 | leaf=7.1134 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 46300 | total=7.1750 | leaf=7.1750 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 46400 | total=7.1669 | leaf=7.1669 | ancestor=0.0000 | lr=2.95e-04 | t=25.2s
step= 46500 | total=7.1363 | leaf=7.1363 | ancestor=0.0000 | lr=2.95e-04 | t=25.1s
step= 46600 | total=7.2791 | leaf=7.2791 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46700 | total=7.4642 | leaf=7.4642 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46800 | total=7.1892 | leaf=7.1892 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 46900 | total=7.1366 | leaf=7.1366 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47000 | total=7.0224 | leaf=7.0224 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47100 | total=7.1273 | leaf=7.1273 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47200 | total=7.0058 | leaf=7.0058 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47300 | total=7.0375 | leaf=7.0375 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47400 | total=7.1482 | leaf=7.1482 | ancestor=0.0000 | lr=2.95e-04 | t=25.0s
step= 47500 | total=7.1219 | leaf=7.1219 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 47600 | total=7.0699 | leaf=7.0699 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 47700 | total=6.9736 | leaf=6.9736 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 47800 | total=7.0667 | leaf=7.0667 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 47900 | total=7.0476 | leaf=7.0476 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48000 | total=7.2662 | leaf=7.2662 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48100 | total=7.0390 | leaf=7.0390 | ancestor=0.0000 | lr=2.94e-04 | t=25.1s
step= 48200 | total=7.0794 | leaf=7.0794 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48300 | total=7.1032 | leaf=7.1032 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48400 | total=7.0874 | leaf=7.0874 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48500 | total=7.1858 | leaf=7.1858 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48600 | total=7.1095 | leaf=7.1095 | ancestor=0.0000 | lr=2.94e-04 | t=25.1s
step= 48700 | total=7.0474 | leaf=7.0474 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48800 | total=6.9920 | leaf=6.9920 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 48900 | total=7.2171 | leaf=7.2171 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49000 | total=7.3447 | leaf=7.3447 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49100 | total=7.2320 | leaf=7.2320 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49200 | total=7.0523 | leaf=7.0523 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49300 | total=7.0974 | leaf=7.0974 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49400 | total=7.1254 | leaf=7.1254 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49500 | total=7.1064 | leaf=7.1064 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49600 | total=7.1602 | leaf=7.1602 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49700 | total=7.0757 | leaf=7.0757 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49800 | total=7.1087 | leaf=7.1087 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 49900 | total=7.2125 | leaf=7.2125 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50000 | total=7.1337 | leaf=7.1337 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
  VAL | loss_leaf=7.1627 | loss_ancestor=0.0000 | loss_total=7.1627
  Saved checkpoint: outputs/sad_new/ckpt_50000.pt
step= 50100 | total=7.1345 | leaf=7.1345 | ancestor=0.0000 | lr=2.94e-04 | t=36.3s
step= 50200 | total=7.3263 | leaf=7.3263 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50300 | total=7.1270 | leaf=7.1270 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50400 | total=8.5118 | leaf=8.5118 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50500 | total=8.3490 | leaf=8.3490 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50600 | total=7.1888 | leaf=7.1888 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50700 | total=7.2208 | leaf=7.2208 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50800 | total=7.7603 | leaf=7.7603 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 50900 | total=7.1234 | leaf=7.1234 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 51000 | total=7.1227 | leaf=7.1227 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 51100 | total=7.0863 | leaf=7.0863 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 51200 | total=7.1147 | leaf=7.1147 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 51300 | total=7.0953 | leaf=7.0953 | ancestor=0.0000 | lr=2.94e-04 | t=25.0s
step= 51400 | total=7.0936 | leaf=7.0936 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 51500 | total=7.1418 | leaf=7.1418 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 51600 | total=7.2281 | leaf=7.2281 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 51700 | total=7.1376 | leaf=7.1376 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 51800 | total=9.1308 | leaf=9.1308 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 51900 | total=7.5337 | leaf=7.5337 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52000 | total=7.2166 | leaf=7.2166 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52100 | total=7.1758 | leaf=7.1758 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52200 | total=7.2197 | leaf=7.2197 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52300 | total=7.2495 | leaf=7.2495 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52400 | total=7.2231 | leaf=7.2231 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52500 | total=7.0173 | leaf=7.0173 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52600 | total=7.2212 | leaf=7.2212 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52700 | total=7.3045 | leaf=7.3045 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52800 | total=7.2077 | leaf=7.2077 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 52900 | total=7.1458 | leaf=7.1458 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53000 | total=7.1528 | leaf=7.1528 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53100 | total=7.1638 | leaf=7.1638 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 53200 | total=7.5502 | leaf=7.5502 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 53300 | total=7.3189 | leaf=7.3189 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53400 | total=7.2129 | leaf=7.2129 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53500 | total=7.1513 | leaf=7.1513 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53600 | total=7.1481 | leaf=7.1481 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 53700 | total=7.0789 | leaf=7.0789 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 53800 | total=7.0833 | leaf=7.0833 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 53900 | total=7.1917 | leaf=7.1917 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54000 | total=7.1875 | leaf=7.1875 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 54100 | total=7.0966 | leaf=7.0966 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 54200 | total=7.4634 | leaf=7.4634 | ancestor=0.0000 | lr=2.93e-04 | t=25.0s
step= 54300 | total=7.2124 | leaf=7.2124 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54400 | total=7.5024 | leaf=7.5024 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54500 | total=7.0928 | leaf=7.0928 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54600 | total=7.1648 | leaf=7.1648 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54700 | total=7.1977 | leaf=7.1977 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54800 | total=7.0683 | leaf=7.0683 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 54900 | total=7.0877 | leaf=7.0877 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
step= 55000 | total=7.7774 | leaf=7.7774 | ancestor=0.0000 | lr=2.93e-04 | t=24.9s
  VAL | loss_leaf=7.6965 | loss_ancestor=0.0000 | loss_total=7.6965
step= 55100 | total=7.1996 | leaf=7.1996 | ancestor=0.0000 | lr=2.92e-04 | t=29.4s
step= 55200 | total=7.5079 | leaf=7.5079 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 55300 | total=7.2143 | leaf=7.2143 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 55400 | total=7.1911 | leaf=7.1911 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 55500 | total=7.2000 | leaf=7.2000 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 55600 | total=7.1331 | leaf=7.1331 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 55700 | total=7.1712 | leaf=7.1712 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 55800 | total=7.1382 | leaf=7.1382 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 55900 | total=7.1184 | leaf=7.1184 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 56000 | total=11.6476 | leaf=11.6476 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56100 | total=7.1423 | leaf=7.1423 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56200 | total=7.0985 | leaf=7.0985 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56300 | total=7.0400 | leaf=7.0400 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 56400 | total=7.1298 | leaf=7.1298 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 56500 | total=7.1239 | leaf=7.1239 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56600 | total=7.0188 | leaf=7.0188 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56700 | total=7.1104 | leaf=7.1104 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 56800 | total=7.2801 | leaf=7.2801 | ancestor=0.0000 | lr=2.92e-04 | t=25.0s
step= 56900 | total=17.3822 | leaf=17.3822 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57000 | total=7.9903 | leaf=7.9903 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57100 | total=7.0950 | leaf=7.0950 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57200 | total=7.1058 | leaf=7.1058 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57300 | total=7.3063 | leaf=7.3063 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57400 | total=7.1254 | leaf=7.1254 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57500 | total=7.1470 | leaf=7.1470 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57600 | total=7.0886 | leaf=7.0886 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57700 | total=7.0030 | leaf=7.0030 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57800 | total=7.1073 | leaf=7.1073 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 57900 | total=7.2480 | leaf=7.2480 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58000 | total=6.9807 | leaf=6.9807 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58100 | total=7.0626 | leaf=7.0626 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58200 | total=7.1462 | leaf=7.1462 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58300 | total=7.2298 | leaf=7.2298 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58400 | total=7.9558 | leaf=7.9558 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58500 | total=7.5988 | leaf=7.5988 | ancestor=0.0000 | lr=2.92e-04 | t=24.9s
step= 58600 | total=7.7663 | leaf=7.7663 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 58700 | total=7.3081 | leaf=7.3081 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 58800 | total=10.6933 | leaf=10.6933 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 58900 | total=7.2981 | leaf=7.2981 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59000 | total=7.3336 | leaf=7.3336 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59100 | total=7.2056 | leaf=7.2056 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59200 | total=7.3418 | leaf=7.3418 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59300 | total=7.1945 | leaf=7.1945 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59400 | total=7.3197 | leaf=7.3197 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59500 | total=7.3731 | leaf=7.3731 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59600 | total=7.2197 | leaf=7.2197 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59700 | total=7.2936 | leaf=7.2936 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59800 | total=7.3356 | leaf=7.3356 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 59900 | total=7.1529 | leaf=7.1529 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60000 | total=7.2744 | leaf=7.2744 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
  VAL | loss_leaf=7.2535 | loss_ancestor=0.0000 | loss_total=7.2535
  Saved checkpoint: outputs/sad_new/ckpt_60000.pt
step= 60100 | total=7.2375 | leaf=7.2375 | ancestor=0.0000 | lr=2.91e-04 | t=36.4s
step= 60200 | total=7.1818 | leaf=7.1818 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60300 | total=7.1945 | leaf=7.1945 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60400 | total=7.1733 | leaf=7.1733 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60500 | total=7.2723 | leaf=7.2723 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60600 | total=7.2862 | leaf=7.2862 | ancestor=0.0000 | lr=2.91e-04 | t=25.0s
step= 60700 | total=7.6188 | leaf=7.6188 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 60800 | total=7.3209 | leaf=7.3209 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 60900 | total=7.5594 | leaf=7.5594 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61000 | total=7.2506 | leaf=7.2506 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61100 | total=7.2049 | leaf=7.2049 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61200 | total=7.2323 | leaf=7.2323 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61300 | total=7.1627 | leaf=7.1627 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61400 | total=7.1991 | leaf=7.1991 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61500 | total=7.1213 | leaf=7.1213 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61600 | total=7.1545 | leaf=7.1545 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61700 | total=7.0835 | leaf=7.0835 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61800 | total=7.0241 | leaf=7.0241 | ancestor=0.0000 | lr=2.91e-04 | t=24.9s
step= 61900 | total=7.0618 | leaf=7.0618 | ancestor=0.0000 | lr=2.90e-04 | t=25.1s
step= 62000 | total=6.9673 | leaf=6.9673 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62100 | total=7.0789 | leaf=7.0789 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62200 | total=6.9444 | leaf=6.9444 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62300 | total=7.0863 | leaf=7.0863 | ancestor=0.0000 | lr=2.90e-04 | t=25.0s
step= 62400 | total=7.4010 | leaf=7.4010 | ancestor=0.0000 | lr=2.90e-04 | t=25.0s
step= 62500 | total=6.9021 | leaf=6.9021 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62600 | total=7.0156 | leaf=7.0156 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62700 | total=7.9125 | leaf=7.9125 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62800 | total=6.9418 | leaf=6.9418 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 62900 | total=7.2859 | leaf=7.2859 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63000 | total=6.9997 | leaf=6.9997 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63100 | total=6.9670 | leaf=6.9670 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63200 | total=6.9730 | leaf=6.9730 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63300 | total=7.2600 | leaf=7.2600 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63400 | total=7.0294 | leaf=7.0294 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63500 | total=6.9947 | leaf=6.9947 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63600 | total=6.9897 | leaf=6.9897 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63700 | total=6.9720 | leaf=6.9720 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63800 | total=9.5317 | leaf=9.5317 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 63900 | total=7.3411 | leaf=7.3411 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64000 | total=7.1066 | leaf=7.1066 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64100 | total=7.0499 | leaf=7.0499 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64200 | total=7.0174 | leaf=7.0174 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64300 | total=7.0267 | leaf=7.0267 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64400 | total=6.9896 | leaf=6.9896 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64500 | total=7.1282 | leaf=7.1282 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64600 | total=7.1594 | leaf=7.1594 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64700 | total=7.0679 | leaf=7.0679 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64800 | total=7.7117 | leaf=7.7117 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 64900 | total=7.0947 | leaf=7.0947 | ancestor=0.0000 | lr=2.90e-04 | t=24.9s
step= 65000 | total=6.9497 | leaf=6.9497 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
  VAL | loss_leaf=7.0453 | loss_ancestor=0.0000 | loss_total=7.0453
step= 65100 | total=7.2183 | leaf=7.2183 | ancestor=0.0000 | lr=2.89e-04 | t=29.5s
step= 65200 | total=7.2714 | leaf=7.2714 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65300 | total=7.1818 | leaf=7.1818 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65400 | total=7.9129 | leaf=7.9129 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65500 | total=7.3840 | leaf=7.3840 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65600 | total=7.1181 | leaf=7.1181 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65700 | total=7.2459 | leaf=7.2459 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65800 | total=8.3846 | leaf=8.3846 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 65900 | total=7.2403 | leaf=7.2403 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66000 | total=7.3371 | leaf=7.3371 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66100 | total=7.3545 | leaf=7.3545 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66200 | total=7.3163 | leaf=7.3163 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66300 | total=7.3992 | leaf=7.3992 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66400 | total=7.3011 | leaf=7.3011 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66500 | total=7.3321 | leaf=7.3321 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66600 | total=7.3643 | leaf=7.3643 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66700 | total=7.2779 | leaf=7.2779 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66800 | total=7.3297 | leaf=7.3297 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 66900 | total=7.3004 | leaf=7.3004 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67000 | total=7.2346 | leaf=7.2346 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67100 | total=7.3594 | leaf=7.3594 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67200 | total=7.2243 | leaf=7.2243 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67300 | total=7.3441 | leaf=7.3441 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67400 | total=7.2385 | leaf=7.2385 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67500 | total=7.3411 | leaf=7.3411 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67600 | total=7.3059 | leaf=7.3059 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67700 | total=7.2857 | leaf=7.2857 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67800 | total=7.3491 | leaf=7.3491 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 67900 | total=7.3284 | leaf=7.3284 | ancestor=0.0000 | lr=2.89e-04 | t=24.9s
step= 68000 | total=7.4420 | leaf=7.4420 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68100 | total=7.3333 | leaf=7.3333 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68200 | total=7.3468 | leaf=7.3468 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68300 | total=7.3154 | leaf=7.3154 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68400 | total=7.3691 | leaf=7.3691 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68500 | total=7.5169 | leaf=7.5169 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68600 | total=7.5270 | leaf=7.5270 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68700 | total=7.3973 | leaf=7.3973 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68800 | total=7.4943 | leaf=7.4943 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 68900 | total=7.4076 | leaf=7.4076 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69000 | total=7.4766 | leaf=7.4766 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69100 | total=7.4471 | leaf=7.4471 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69200 | total=7.4442 | leaf=7.4442 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69300 | total=7.4813 | leaf=7.4813 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69400 | total=7.5140 | leaf=7.5140 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69500 | total=7.5187 | leaf=7.5187 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69600 | total=7.5262 | leaf=7.5262 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69700 | total=7.4994 | leaf=7.4994 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69800 | total=7.5506 | leaf=7.5506 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 69900 | total=7.4878 | leaf=7.4878 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70000 | total=7.6476 | leaf=7.6476 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
  VAL | loss_leaf=7.5540 | loss_ancestor=0.0000 | loss_total=7.5540
  Saved checkpoint: outputs/sad_new/ckpt_70000.pt
step= 70100 | total=7.5471 | leaf=7.5471 | ancestor=0.0000 | lr=2.88e-04 | t=36.0s
step= 70200 | total=7.5363 | leaf=7.5363 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70300 | total=7.5880 | leaf=7.5880 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70400 | total=7.6178 | leaf=7.6178 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70500 | total=7.5535 | leaf=7.5535 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70600 | total=7.6508 | leaf=7.6508 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70700 | total=7.5751 | leaf=7.5751 | ancestor=0.0000 | lr=2.88e-04 | t=24.9s
step= 70800 | total=7.6316 | leaf=7.6316 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 70900 | total=7.4851 | leaf=7.4851 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71000 | total=7.6295 | leaf=7.6295 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71100 | total=7.6137 | leaf=7.6137 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71200 | total=7.7132 | leaf=7.7132 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71300 | total=7.7172 | leaf=7.7172 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71400 | total=7.6915 | leaf=7.6915 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71500 | total=7.7787 | leaf=7.7787 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71600 | total=7.8100 | leaf=7.8100 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71700 | total=7.6935 | leaf=7.6935 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71800 | total=7.7583 | leaf=7.7583 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 71900 | total=7.8120 | leaf=7.8120 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72000 | total=7.7553 | leaf=7.7553 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72100 | total=7.7589 | leaf=7.7589 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72200 | total=7.8042 | leaf=7.8042 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72300 | total=7.8526 | leaf=7.8526 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72400 | total=7.8481 | leaf=7.8481 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72500 | total=7.9341 | leaf=7.9341 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72600 | total=7.8854 | leaf=7.8854 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72700 | total=7.8468 | leaf=7.8468 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72800 | total=7.9169 | leaf=7.9169 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 72900 | total=7.8251 | leaf=7.8251 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73000 | total=7.9233 | leaf=7.9233 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73100 | total=7.8965 | leaf=7.8965 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73200 | total=7.8761 | leaf=7.8761 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73300 | total=7.9293 | leaf=7.9293 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73400 | total=7.9133 | leaf=7.9133 | ancestor=0.0000 | lr=2.87e-04 | t=24.9s
step= 73500 | total=8.0005 | leaf=8.0005 | ancestor=0.0000 | lr=2.86e-04 | t=24.9s
  File "/home/sunhc/diffusion/sad/scripts/train.py", line 579, in <module>
    main()
  File "/home/sunhc/diffusion/sad/scripts/train.py", line 491, in main
    loss, metrics = full_denoise_step(
                    ^^^^^^^^^^^^^^^^^^
  File "/home/sunhc/diffusion/sad/scripts/train.py", line 270, in full_denoise_step
    loss, metrics = loss_fn(
                    ^^^^^^^^
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sunhc/diffusion/sad/src/losses/sad_loss.py", line 168, in forward
    assert not torch.isnan(total), "NaN loss detected!"
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: NaN loss detected!
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train.py", line 579, in <module>
[rank0]:     main()
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train.py", line 491, in main
[rank0]:     loss, metrics = full_denoise_step(
[rank0]:                     ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sunhc/diffusion/sad/scripts/train.py", line 270, in full_denoise_step
[rank0]:     loss, metrics = loss_fn(
[rank0]:                     ^^^^^^^^
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sunhc/diffusion/sad/src/losses/sad_loss.py", line 168, in forward
[rank0]:     assert not torch.isnan(total), "NaN loss detected!"
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AssertionError: NaN loss detected!