2026-04-20 07:05:20,500 INFO MainThread:1749141 [wandb_setup.py:_flush():81] Current SDK version is 0.25.1 2026-04-20 07:05:20,500 INFO MainThread:1749141 [wandb_setup.py:_flush():81] Configure stats pid to 1749141 2026-04-20 07:05:20,500 INFO MainThread:1749141 [wandb_setup.py:_flush():81] Loading settings from environment variables 2026-04-20 07:05:20,500 INFO MainThread:1749141 [wandb_init.py:setup_run_log_directory():717] Logging user logs to /home/sunhc/diffusion/sad/wandb/run-20260420_070520-78zw38po/logs/debug.log 2026-04-20 07:05:20,500 INFO MainThread:1749141 [wandb_init.py:setup_run_log_directory():718] Logging internal logs to /home/sunhc/diffusion/sad/wandb/run-20260420_070520-78zw38po/logs/debug-internal.log 2026-04-20 07:05:20,501 INFO MainThread:1749141 [wandb_init.py:init():844] calling init triggers 2026-04-20 07:05:20,501 INFO MainThread:1749141 [wandb_init.py:init():849] wandb.init called with sweep_config: {} config: {'model': {'vocab_size': 50258, 'hidden_size': 768, 'n_blocks': 12, 'n_heads': 12, 'cond_dim': 128, 'max_seq_len': 512, 'block_size': 8, 'dropout': 0.0, 'num_levels': 2, 'level_sizes': [50258, 128]}, 'ancestor': {'lut_path': 'data/ancestor_lut_50257-128_top3_t1.0.pt', 'proto_path': 'data/hierarchy_prototypes_50257-128.pt'}, 'loss': {'lambda_ancestor': 0.0, 'mask_only': False}, 'training': {'seed': 0, 'batch_size': 64, 'num_steps': 500000, 'lr': 0.0003, 'lr_min': 3e-05, 'warmup_steps': 2000, 'weight_decay': 0.01, 'grad_clip': 1.0, 'dtype': 'bf16', 'compile': 'default', 'log_interval': 100, 'eval_interval': 5000, 'save_interval': 10000}, 'data': {'dataset': 'openwebtext', 'seq_len': 512, 'cache_dir': 'data/owt_cache', 'num_workers': 4, 'max_train_samples': None, 'max_val_samples': 100000, 'mode': 'subsample'}, 'logging': {'use_wandb': True, 'project': 'sad_new_owt', 'save_dir': 'outputs/sad_new'}, '_wandb': {}} 2026-04-20 07:05:20,501 INFO MainThread:1749141 [wandb_init.py:init():892] starting backend 2026-04-20 07:05:20,722 INFO MainThread:1749141 [wandb_init.py:init():895] sending inform_init request 2026-04-20 07:05:20,727 INFO MainThread:1749141 [wandb_init.py:init():903] backend started and connected 2026-04-20 07:05:20,729 INFO MainThread:1749141 [wandb_init.py:init():973] updated telemetry 2026-04-20 07:05:20,729 INFO MainThread:1749141 [wandb_init.py:init():997] communicating run to backend with 90.0 second timeout 2026-04-20 07:05:22,686 INFO MainThread:1749141 [wandb_init.py:init():1042] starting run threads in backend 2026-04-20 07:05:22,806 INFO MainThread:1749141 [wandb_run.py:_console_start():2524] atexit reg 2026-04-20 07:05:22,806 INFO MainThread:1749141 [wandb_run.py:_redirect():2373] redirect: wrap_raw 2026-04-20 07:05:22,806 INFO MainThread:1749141 [wandb_run.py:_redirect():2442] Wrapping output streams. 2026-04-20 07:05:22,806 INFO MainThread:1749141 [wandb_run.py:_redirect():2465] Redirects installed. 2026-04-20 07:05:22,810 INFO MainThread:1749141 [wandb_init.py:init():1082] run started, returning control to user process 2026-04-20 07:05:23,256 INFO wandb-AsyncioManager-main:1749141 [service_client.py:_forward_responses():134] Reached EOF. 2026-04-20 07:05:23,256 INFO wandb-AsyncioManager-main:1749141 [mailbox.py:close():155] Closing mailbox, abandoning 1 handles. 2026-04-20 07:05:32,544 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,545 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,546 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,551 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,552 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,552 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,553 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,553 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,554 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,554 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,555 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,555 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,555 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,557 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,560 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,562 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,565 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,566 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,566 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,566 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,567 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,567 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,567 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,568 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,568 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,568 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,568 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,569 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,569 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,569 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,569 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,570 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,570 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,570 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,570 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,570 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,571 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,571 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,571 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,571 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,572 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,572 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,572 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,572 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,573 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,573 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,573 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,573 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,574 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,574 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,575 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,576 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,576 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,576 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,577 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,577 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,577 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,577 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,578 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,578 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,578 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,579 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,579 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,579 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,579 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,580 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,580 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,580 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,580 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,581 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,581 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,581 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,581 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,582 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,582 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,582 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,582 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,583 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,583 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,583 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,583 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,583 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,584 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,584 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,584 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,584 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,585 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,585 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,585 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,585 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,586 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,586 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,586 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,586 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,587 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,587 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,587 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,587 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,587 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,588 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,588 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,588 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,588 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,589 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,589 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,589 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,589 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,590 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,590 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,590 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,591 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,591 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,591 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,592 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,592 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,592 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,593 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,593 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,593 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,593 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,594 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,594 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,594 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,595 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,595 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,595 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,596 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,596 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,596 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,596 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,597 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,597 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,597 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,598 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,598 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,598 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,598 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,599 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,599 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,599 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,600 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,600 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,600 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,600 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,601 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,601 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,601 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,602 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,602 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,602 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,602 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,603 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,603 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,603 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,603 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,604 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,604 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,604 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,605 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,605 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,605 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,605 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,606 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,606 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,606 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,606 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,607 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,607 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,607 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,608 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,608 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,608 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,608 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,609 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,609 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,609 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,609 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,610 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,610 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,611 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,613 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,614 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,614 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,614 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,615 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,616 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,617 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,617 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,617 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,618 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,618 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,618 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2026-04-20 07:05:32,619 ERROR wandb-AsyncioManager-main:1749141 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 45, in publish await self._send_server_request(request) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 78, in _send_server_request raise self._broken_exc.with_traceback(self._broken_tb) File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 87, in _send_server_request await self._drain_writer() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/site-packages/wandb/sdk/lib/service/service_client.py", line 96, in _drain_writer await self._writer.drain() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 392, in drain await self._protocol._drain_helper() File "/home/sunhc/miniconda3/envs/sad/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost