block-ar training:   0%|                                                                                                         | 0/5000 [00:00<?, ?it/s][rank0]:W0417 08:49:20.686000 2814700 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
block-ar training:  62%|████████████████████████▊               | 3101/5000 [19:44<11:41,  2.71it/s, leaf=2.2782, lr=1.2e-04]                                
step=     0 | total=11.0256 | leaf=11.0256 | ancestor=0.0000 | lr=0.00e+00 | t=39.4s
step=   100 | total=4.7179 | leaf=4.7179 | ancestor=0.0000 | lr=3.00e-04 | t=36.6s
step=   200 | total=3.5896 | leaf=3.5896 | ancestor=0.0000 | lr=2.99e-04 | t=36.8s
step=   300 | total=3.3361 | leaf=3.3361 | ancestor=0.0000 | lr=2.98e-04 | t=36.8s
step=   400 | total=3.1872 | leaf=3.1872 | ancestor=0.0000 | lr=2.97e-04 | t=36.9s
step=   500 | total=3.0455 | leaf=3.0455 | ancestor=0.0000 | lr=2.95e-04 | t=36.9s
step=   600 | total=2.8997 | leaf=2.8997 | ancestor=0.0000 | lr=2.92e-04 | t=37.0s
step=   700 | total=2.8434 | leaf=2.8434 | ancestor=0.0000 | lr=2.89e-04 | t=37.0s
step=   800 | total=2.7669 | leaf=2.7669 | ancestor=0.0000 | lr=2.85e-04 | t=37.0s
step=   900 | total=2.7214 | leaf=2.7214 | ancestor=0.0000 | lr=2.81e-04 | t=37.0s
step=  1000 | total=2.6468 | leaf=2.6468 | ancestor=0.0000 | lr=2.76e-04 | t=37.0s
step=  1100 | total=2.6522 | leaf=2.6522 | ancestor=0.0000 | lr=2.71e-04 | t=37.0s
step=  1200 | total=2.6147 | leaf=2.6147 | ancestor=0.0000 | lr=2.66e-04 | t=37.0s
step=  1300 | total=2.5260 | leaf=2.5260 | ancestor=0.0000 | lr=2.60e-04 | t=36.9s
step=  1400 | total=2.5401 | leaf=2.5401 | ancestor=0.0000 | lr=2.53e-04 | t=37.0s
step=  1500 | total=2.5395 | leaf=2.5395 | ancestor=0.0000 | lr=2.47e-04 | t=37.0s
step=  1600 | total=2.4713 | leaf=2.4713 | ancestor=0.0000 | lr=2.40e-04 | t=37.0s
step=  1700 | total=2.4466 | leaf=2.4466 | ancestor=0.0000 | lr=2.32e-04 | t=37.0s
step=  1800 | total=2.4480 | leaf=2.4480 | ancestor=0.0000 | lr=2.25e-04 | t=37.0s
step=  1900 | total=2.3727 | leaf=2.3727 | ancestor=0.0000 | lr=2.17e-04 | t=37.0s
step=  2000 | total=2.4443 | leaf=2.4443 | ancestor=0.0000 | lr=2.09e-04 | t=37.0s
step=  2100 | total=2.4892 | leaf=2.4892 | ancestor=0.0000 | lr=2.01e-04 | t=37.0s
step=  2200 | total=2.4116 | leaf=2.4116 | ancestor=0.0000 | lr=1.93e-04 | t=37.0s
step=  2300 | total=2.3893 | leaf=2.3893 | ancestor=0.0000 | lr=1.84e-04 | t=37.0s
step=  2400 | total=2.3561 | leaf=2.3561 | ancestor=0.0000 | lr=1.76e-04 | t=37.0s
step=  2500 | total=2.3743 | leaf=2.3743 | ancestor=0.0000 | lr=1.67e-04 | t=37.0s
step=  2600 | total=2.3472 | leaf=2.3472 | ancestor=0.0000 | lr=1.59e-04 | t=37.0s
step=  2700 | total=2.3530 | leaf=2.3530 | ancestor=0.0000 | lr=1.50e-04 | t=37.0s
step=  2800 | total=2.3471 | leaf=2.3471 | ancestor=0.0000 | lr=1.42e-04 | t=37.0s
step=  2900 | total=2.2885 | leaf=2.2885 | ancestor=0.0000 | lr=1.33e-04 | t=37.0s
step=  3000 | total=2.3249 | leaf=2.3249 | ancestor=0.0000 | lr=1.25e-04 | t=37.0s
step=  3100 | total=2.2782 | leaf=2.2782 | ancestor=0.0000 | lr=1.17e-04 | t=37.0s
step=  3200 | total=2.3131 | leaf=2.3131 | ancestor=0.0000 | lr=1.09e-04 | t=37.0s
step=  3300 | total=2.3654 | leaf=2.3654 | ancestor=0.0000 | lr=1.01e-04 | t=37.0s
step=  3400 | total=2.4052 | leaf=2.4052 | ancestor=0.0000 | lr=9.38e-05 | t=37.0s
step=  3500 | total=2.2770 | leaf=2.2770 | ancestor=0.0000 | lr=8.67e-05 | t=37.0s
step=  3600 | total=2.2161 | leaf=2.2161 | ancestor=0.0000 | lr=7.99e-05 | t=37.0s
step=  3700 | total=2.3421 | leaf=2.3421 | ancestor=0.0000 | lr=7.34e-05 | t=37.0s
step=  3800 | total=2.2882 | leaf=2.2882 | ancestor=0.0000 | lr=6.73e-05 | t=37.0s
step=  3900 | total=2.3176 | leaf=2.3176 | ancestor=0.0000 | lr=6.16e-05 | t=37.0s
step=  4000 | total=2.2266 | leaf=2.2266 | ancestor=0.0000 | lr=5.63e-05 | t=37.0s
step=  4100 | total=2.2229 | leaf=2.2229 | ancestor=0.0000 | lr=5.14e-05 | t=37.0s
step=  4200 | total=2.3394 | leaf=2.3394 | ancestor=0.0000 | lr=4.70e-05 | t=37.0s
step=  4300 | total=2.2308 | leaf=2.2308 | ancestor=0.0000 | lr=4.31e-05 | t=37.0s
step=  4400 | total=2.2706 | leaf=2.2706 | ancestor=0.0000 | lr=3.97e-05 | t=37.0s
step=  4500 | total=2.2314 | leaf=2.2314 | ancestor=0.0000 | lr=3.67e-05 | t=37.0s
step=  4600 | total=2.2693 | leaf=2.2693 | ancestor=0.0000 | lr=3.43e-05 | t=37.0s
step=  4700 | total=2.2513 | leaf=2.2513 | ancestor=0.0000 | lr=3.24e-05 | t=37.0s
step=  4800 | total=2.1752 | leaf=2.1752 | ancestor=0.0000 | lr=3.11e-05 | t=37.0s
step=  4900 | total=2.1706 | leaf=2.1706 | ancestor=0.0000 | lr=3.03e-05 | t=37.0s
  Saved checkpoint: outputs/sad_block_ar/ckpt_5000.pt
Training complete.