wandb_128/offline-run-20260227_130446-lxvmjzq8/logs/debug-internal.log

2026-02-27 13:04:46,338 INFO    StreamThr :1518997 [internal.py:wandb_internal():85] W&B internal server running at pid: 1518997, started at: 2026-02-27 13:04:46.335438
2026-02-27 13:04:46,340 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status
2026-02-27 13:04:46,365 INFO    WriterThread:1518997 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:04:46,476 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: run_start
2026-02-27 13:04:46,515 DEBUG   HandlerThread:1518997 [system_info.py:__init__():26] System info init
2026-02-27 13:04:46,515 DEBUG   HandlerThread:1518997 [system_info.py:__init__():41] System info init done
2026-02-27 13:04:46,516 INFO    HandlerThread:1518997 [system_monitor.py:start():194] Starting system monitor
2026-02-27 13:04:46,516 INFO    SystemMonitor:1518997 [system_monitor.py:_start():158] Starting system asset monitoring threads
2026-02-27 13:04:46,516 INFO    HandlerThread:1518997 [system_monitor.py:probe():214] Collecting system info
2026-02-27 13:04:46,517 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started cpu monitoring
2026-02-27 13:04:46,517 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started disk monitoring
2026-02-27 13:04:46,518 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started gpu monitoring
2026-02-27 13:04:46,518 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started memory monitoring
2026-02-27 13:04:46,519 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started network monitoring
2026-02-27 13:04:46,640 DEBUG   HandlerThread:1518997 [system_info.py:probe():152] Probing system
2026-02-27 13:04:46,642 DEBUG   HandlerThread:1518997 [system_info.py:_probe_git():137] Probing git
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_info.py:_probe_git():145] Probing git done
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_info.py:probe():200] Probing system done
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.94.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.10.19', 'heartbeatAt': '2026-02-27T12:04:46.640569', 'startedAt': '2026-02-27T12:04:46.285930', 'docker': None, 'cuda': None, 'args': ('+name=train_stage_a_mamba', 'algorithm=df_video_mamba3stage', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'dataset=video_minecraft_latent', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=200', '+dataset.n_frames_valid=700', '+dataset.angle_range=110', '+dataset.pos_range=8', '+dataset.customized_validation=true', '+dataset.add_timestamp_embedding=true', 'dataset.use_explicit_memory_frames=false', 'algorithm.training_stage=stage_a_memory_pretrain', 'algorithm.use_mamba_memory_pipeline=true', 'algorithm.use_oracle_pose_eval=false', 'algorithm.use_precomputed_features=true', 'algorithm.context_frames=1', '+algorithm.require_pose_prediction=false', '+algorithm.use_memory_attention=false', '+algorithm.relative_embedding=false', '+algorithm.memory_retrieval_topk=8', 'experiment.training.batch_size=64', 'experiment.training.checkpointing.every_n_train_steps=2500', 'experiment.training.max_steps=700000', '+output_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'), 'state': 'running', 'program': '-m main', 'codePathLocal': None, 'git': {'remote': 'https://huggingface.co/BonanDing/WorldMem_Repro', 'commit': 'f0d488aceb5425378cec21c5d35bd92f64346f51'}, 'email': 'fahad.khan@liu.se', 'root': '/proj/cvl/users/x_fahkh2/WorldMem_Repro', 'host': 'node020', 'username': 'x_fahkh', 'executable': '/proj/cvl/users/x_fahkh2/envs/worldmem/bin/python', 'cpu_count': 128, 'cpu_count_logical': 256, 'cpu_freq': {'current': 2440.1533984375005, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3380.836, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.286, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.06, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.554, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.695, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.677, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.853, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.748, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.632, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.626, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.484, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.038, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.444, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.89, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.093, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.439, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.443, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3387.401, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.58, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.459, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.238, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.572, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.965, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.185, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.956, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.13, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.513, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.297, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.601, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.412, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.741, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.294, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.686, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.043, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3381.512, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.025, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 29.166847229003906, 'used': 15.824077606201172}}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 2, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1007.6652145385742}}
2026-02-27 13:04:46,725 INFO    HandlerThread:1518997 [system_monitor.py:probe():224] Finished collecting system info
2026-02-27 13:04:46,725 INFO    HandlerThread:1518997 [system_monitor.py:probe():227] Publishing system info
2026-02-27 13:04:46,726 DEBUG   HandlerThread:1518997 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment
2026-02-27 13:04:53,157 DEBUG   HandlerThread:1518997 [system_info.py:_save_conda():224] Saving conda packages done
2026-02-27 13:04:53,164 INFO    HandlerThread:1518997 [system_monitor.py:probe():229] Finished publishing system info
2026-02-27 13:04:53,178 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:53,178 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: keepalive
2026-02-27 13:04:53,178 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:54,467 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: python_packages
2026-02-27 13:04:57,179 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:57,180 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:58,193 DEBUG   SenderThread:1518997 [sender.py:send():391] send: exit
2026-02-27 13:04:58,193 INFO    SenderThread:1518997 [sender.py:send_exit():598] handling exit code: 1
2026-02-27 13:04:58,194 INFO    SenderThread:1518997 [sender.py:send_exit():600] handling runtime: 11
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:send_exit():606] send defer
2026-02-27 13:04:58,197 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,197 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 0
2026-02-27 13:04:58,197 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 0
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 1
2026-02-27 13:04:58,197 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,197 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 1
2026-02-27 13:04:58,198 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,198 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 1
2026-02-27 13:04:58,198 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 2
2026-02-27 13:04:58,198 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,198 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 2
2026-02-27 13:04:58,198 INFO    HandlerThread:1518997 [system_monitor.py:finish():203] Stopping system monitor
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():183] Publishing last batch of metrics
2026-02-27 13:04:58,199 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined cpu monitor
2026-02-27 13:04:58,200 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined disk monitor
2026-02-27 13:04:58,983 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined gpu monitor
2026-02-27 13:04:58,983 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined memory monitor
2026-02-27 13:04:58,984 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined network monitor
2026-02-27 13:04:58,984 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 2
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 3
2026-02-27 13:04:58,984 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,984 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 3
2026-02-27 13:04:58,984 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 3
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 4
2026-02-27 13:04:58,984 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 4
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 4
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 5
2026-02-27 13:04:58,985 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 5
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 5
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 6
2026-02-27 13:04:58,985 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 6
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 6
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 7
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 7
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 7
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 8
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 8
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 8
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 9
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 9
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 9
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 10
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 10
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 10
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 11
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 11
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 11
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 12
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 12
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 12
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 13
2026-02-27 13:04:58,988 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,988 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 13
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 13
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 14
2026-02-27 13:04:58,988 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,988 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 14
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send():391] send: final
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 14
2026-02-27 13:04:58,989 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:04:58,989 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: server_info
2026-02-27 13:04:58,990 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: server_info
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: get_summary
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: sampled_history
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: internal_messages
2026-02-27 13:04:58,991 INFO    MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4088] rendering history
2026-02-27 13:04:58,991 INFO    MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4120] rendering summary
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: shutdown
2026-02-27 13:04:58,991 INFO    HandlerThread:1518997 [handler.py:finish():884] shutting down handler
2026-02-27 13:04:59,990 INFO    WriterThread:1518997 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:04:59,990 INFO    SenderThread:1518997 [sender.py:finish():1631] shutting down sender