2026-04-21 01:40:47,458 INFO    MainThread:3762127 [wandb_setup.py:_flush():81] Current SDK version is 0.25.1
2026-04-21 01:40:47,458 INFO    MainThread:3762127 [wandb_setup.py:_flush():81] Configure stats pid to 3762127
2026-04-21 01:40:47,458 INFO    MainThread:3762127 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-04-21 01:40:47,458 INFO    MainThread:3762127 [wandb_init.py:setup_run_log_directory():717] Logging user logs to /home/sunhc/diffusion/sad/wandb/run-20260421_014047-mefloecc/logs/debug.log
2026-04-21 01:40:47,459 INFO    MainThread:3762127 [wandb_init.py:setup_run_log_directory():718] Logging internal logs to /home/sunhc/diffusion/sad/wandb/run-20260421_014047-mefloecc/logs/debug-internal.log
2026-04-21 01:40:47,459 INFO    MainThread:3762127 [wandb_init.py:init():844] calling init triggers
2026-04-21 01:40:47,459 INFO    MainThread:3762127 [wandb_init.py:init():849] wandb.init called with sweep_config: {}
config: {'model': {'vocab_size': 50258, 'hidden_size': 768, 'n_blocks': 12, 'n_heads': 12, 'cond_dim': 128, 'max_seq_len': 512, 'block_size': 8, 'dropout': 0.0, 'num_levels': 2, 'level_sizes': [50258, 128]}, 'ancestor': {'lut_path': 'data/ancestor_lut_50257-128_top3_t1.0.pt', 'proto_path': 'data/hierarchy_prototypes_50257-128.pt'}, 'loss': {'lambda_ancestor': 0.0, 'mask_only': True}, 'training': {'seed': 0, 'batch_size': 64, 'num_steps': 500000, 'lr': 0.0003, 'lr_min': 3e-05, 'warmup_steps': 2000, 'weight_decay': 0.01, 'grad_clip': 1.0, 'dtype': 'bf16', 'compile': 'default', 'log_interval': 100, 'eval_interval': 5000, 'save_interval': 10000}, 'data': {'dataset': 'openwebtext', 'seq_len': 512, 'cache_dir': 'data/owt_cache', 'num_workers': 4, 'max_train_samples': None, 'max_val_samples': 100000, 'mode': 'subsample'}, 'logging': {'use_wandb': True, 'project': 'sad_new_owt', 'save_dir': 'outputs/sad_new'}, '_wandb': {}}
2026-04-21 01:40:47,459 INFO    MainThread:3762127 [wandb_init.py:init():892] starting backend
2026-04-21 01:40:47,676 INFO    MainThread:3762127 [wandb_init.py:init():895] sending inform_init request
2026-04-21 01:40:47,681 INFO    MainThread:3762127 [wandb_init.py:init():903] backend started and connected
2026-04-21 01:40:47,682 INFO    MainThread:3762127 [wandb_init.py:init():973] updated telemetry
2026-04-21 01:40:47,683 INFO    MainThread:3762127 [wandb_init.py:init():997] communicating run to backend with 90.0 second timeout
2026-04-21 01:40:49,347 INFO    MainThread:3762127 [wandb_init.py:init():1042] starting run threads in backend
2026-04-21 01:40:49,468 INFO    MainThread:3762127 [wandb_run.py:_console_start():2524] atexit reg
2026-04-21 01:40:49,468 INFO    MainThread:3762127 [wandb_run.py:_redirect():2373] redirect: wrap_raw
2026-04-21 01:40:49,468 INFO    MainThread:3762127 [wandb_run.py:_redirect():2442] Wrapping output streams.
2026-04-21 01:40:49,468 INFO    MainThread:3762127 [wandb_run.py:_redirect():2465] Redirects installed.
2026-04-21 01:40:49,471 INFO    MainThread:3762127 [wandb_init.py:init():1082] run started, returning control to user process
2026-04-21 02:10:06,760 INFO    wandb-AsyncioManager-main:3762127 [service_client.py:_forward_responses():134] Reached EOF.
2026-04-21 02:10:06,760 INFO    wandb-AsyncioManager-main:3762127 [mailbox.py:close():155] Closing mailbox, abandoning 1 handles.
2026-04-21 02:10:24,522 ERROR   wandb-AsyncioManager-main:3762127 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
    await fn()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish
    await self._send_server_request(request)
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request
    await self._drain_writer()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer
    await self._writer.drain()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost