2026-02-27 13:04:46,338 INFO StreamThr :1518997 [internal.py:wandb_internal():85] W&B internal server running at pid: 1518997, started at: 2026-02-27 13:04:46.335438 2026-02-27 13:04:46,340 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status 2026-02-27 13:04:46,365 INFO WriterThread:1518997 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb 2026-02-27 13:04:46,476 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: run_start 2026-02-27 13:04:46,515 DEBUG HandlerThread:1518997 [system_info.py:__init__():26] System info init 2026-02-27 13:04:46,515 DEBUG HandlerThread:1518997 [system_info.py:__init__():41] System info init done 2026-02-27 13:04:46,516 INFO HandlerThread:1518997 [system_monitor.py:start():194] Starting system monitor 2026-02-27 13:04:46,516 INFO SystemMonitor:1518997 [system_monitor.py:_start():158] Starting system asset monitoring threads 2026-02-27 13:04:46,516 INFO HandlerThread:1518997 [system_monitor.py:probe():214] Collecting system info 2026-02-27 13:04:46,517 INFO SystemMonitor:1518997 [interfaces.py:start():188] Started cpu monitoring 2026-02-27 13:04:46,517 INFO SystemMonitor:1518997 [interfaces.py:start():188] Started disk monitoring 2026-02-27 13:04:46,518 INFO SystemMonitor:1518997 [interfaces.py:start():188] Started gpu monitoring 2026-02-27 13:04:46,518 INFO SystemMonitor:1518997 [interfaces.py:start():188] Started memory monitoring 2026-02-27 13:04:46,519 INFO SystemMonitor:1518997 [interfaces.py:start():188] Started network monitoring 2026-02-27 13:04:46,640 DEBUG HandlerThread:1518997 [system_info.py:probe():152] Probing system 2026-02-27 13:04:46,642 DEBUG HandlerThread:1518997 [system_info.py:_probe_git():137] Probing git 2026-02-27 13:04:46,725 DEBUG HandlerThread:1518997 [system_info.py:_probe_git():145] Probing git done 2026-02-27 13:04:46,725 DEBUG HandlerThread:1518997 [system_info.py:probe():200] Probing system done 2026-02-27 13:04:46,725 DEBUG HandlerThread:1518997 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.94.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.10.19', 'heartbeatAt': '2026-02-27T12:04:46.640569', 'startedAt': '2026-02-27T12:04:46.285930', 'docker': None, 'cuda': None, 'args': ('+name=train_stage_a_mamba', 'algorithm=df_video_mamba3stage', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'dataset=video_minecraft_latent', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=200', '+dataset.n_frames_valid=700', '+dataset.angle_range=110', '+dataset.pos_range=8', '+dataset.customized_validation=true', '+dataset.add_timestamp_embedding=true', 'dataset.use_explicit_memory_frames=false', 'algorithm.training_stage=stage_a_memory_pretrain', 'algorithm.use_mamba_memory_pipeline=true', 'algorithm.use_oracle_pose_eval=false', 'algorithm.use_precomputed_features=true', 'algorithm.context_frames=1', '+algorithm.require_pose_prediction=false', '+algorithm.use_memory_attention=false', '+algorithm.relative_embedding=false', '+algorithm.memory_retrieval_topk=8', 'experiment.training.batch_size=64', 'experiment.training.checkpointing.every_n_train_steps=2500', 'experiment.training.max_steps=700000', '+output_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'), 'state': 'running', 'program': '-m main', 'codePathLocal': None, 'git': {'remote': 'https://huggingface.co/BonanDing/WorldMem_Repro', 'commit': 'f0d488aceb5425378cec21c5d35bd92f64346f51'}, 'email': 'fahad.khan@liu.se', 'root': '/proj/cvl/users/x_fahkh2/WorldMem_Repro', 'host': 'node020', 'username': 'x_fahkh', 'executable': '/proj/cvl/users/x_fahkh2/envs/worldmem/bin/python', 'cpu_count': 128, 'cpu_count_logical': 256, 'cpu_freq': {'current': 2440.1533984375005, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3380.836, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.286, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.06, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.554, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.695, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.677, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.853, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.748, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.632, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.626, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.484, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.038, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.444, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.89, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.093, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.439, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.443, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3387.401, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.58, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.459, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.238, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.572, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.965, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.185, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.956, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.13, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.513, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.297, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.601, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.412, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.741, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.294, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.686, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.043, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3381.512, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.025, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 29.166847229003906, 'used': 15.824077606201172}}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 2, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1007.6652145385742}} 2026-02-27 13:04:46,725 INFO HandlerThread:1518997 [system_monitor.py:probe():224] Finished collecting system info 2026-02-27 13:04:46,725 INFO HandlerThread:1518997 [system_monitor.py:probe():227] Publishing system info 2026-02-27 13:04:46,726 DEBUG HandlerThread:1518997 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2026-02-27 13:04:53,157 DEBUG HandlerThread:1518997 [system_info.py:_save_conda():224] Saving conda packages done 2026-02-27 13:04:53,164 INFO HandlerThread:1518997 [system_monitor.py:probe():229] Finished publishing system info 2026-02-27 13:04:53,178 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:04:53,178 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: keepalive 2026-02-27 13:04:53,178 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:04:54,467 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: python_packages 2026-02-27 13:04:57,179 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:04:57,180 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:04:58,193 DEBUG SenderThread:1518997 [sender.py:send():391] send: exit 2026-02-27 13:04:58,193 INFO SenderThread:1518997 [sender.py:send_exit():598] handling exit code: 1 2026-02-27 13:04:58,194 INFO SenderThread:1518997 [sender.py:send_exit():600] handling runtime: 11 2026-02-27 13:04:58,197 INFO SenderThread:1518997 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2026-02-27 13:04:58,197 INFO SenderThread:1518997 [sender.py:send_exit():606] send defer 2026-02-27 13:04:58,197 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,197 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 0 2026-02-27 13:04:58,197 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,197 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 0 2026-02-27 13:04:58,197 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 1 2026-02-27 13:04:58,197 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,197 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 1 2026-02-27 13:04:58,198 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,198 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 1 2026-02-27 13:04:58,198 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 2 2026-02-27 13:04:58,198 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,198 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 2 2026-02-27 13:04:58,198 INFO HandlerThread:1518997 [system_monitor.py:finish():203] Stopping system monitor 2026-02-27 13:04:58,198 DEBUG SystemMonitor:1518997 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2026-02-27 13:04:58,198 DEBUG SystemMonitor:1518997 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2026-02-27 13:04:58,198 DEBUG SystemMonitor:1518997 [system_monitor.py:_start():183] Publishing last batch of metrics 2026-02-27 13:04:58,199 INFO HandlerThread:1518997 [interfaces.py:finish():200] Joined cpu monitor 2026-02-27 13:04:58,200 INFO HandlerThread:1518997 [interfaces.py:finish():200] Joined disk monitor 2026-02-27 13:04:58,983 INFO HandlerThread:1518997 [interfaces.py:finish():200] Joined gpu monitor 2026-02-27 13:04:58,983 INFO HandlerThread:1518997 [interfaces.py:finish():200] Joined memory monitor 2026-02-27 13:04:58,984 INFO HandlerThread:1518997 [interfaces.py:finish():200] Joined network monitor 2026-02-27 13:04:58,984 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,984 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 2 2026-02-27 13:04:58,984 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 3 2026-02-27 13:04:58,984 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,984 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 3 2026-02-27 13:04:58,984 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,984 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 3 2026-02-27 13:04:58,984 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 4 2026-02-27 13:04:58,984 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,985 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 4 2026-02-27 13:04:58,985 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 4 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 5 2026-02-27 13:04:58,985 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,985 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 5 2026-02-27 13:04:58,985 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 5 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 6 2026-02-27 13:04:58,985 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,985 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 6 2026-02-27 13:04:58,985 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 6 2026-02-27 13:04:58,985 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 7 2026-02-27 13:04:58,986 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:04:58,986 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,986 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 7 2026-02-27 13:04:58,986 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:04:58,986 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 7 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 8 2026-02-27 13:04:58,986 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,986 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 8 2026-02-27 13:04:58,986 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 8 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 9 2026-02-27 13:04:58,986 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,986 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 9 2026-02-27 13:04:58,986 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 9 2026-02-27 13:04:58,986 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 10 2026-02-27 13:04:58,987 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,987 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 10 2026-02-27 13:04:58,987 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 10 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 11 2026-02-27 13:04:58,987 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,987 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 11 2026-02-27 13:04:58,987 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 11 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 12 2026-02-27 13:04:58,987 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,987 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 12 2026-02-27 13:04:58,987 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 12 2026-02-27 13:04:58,987 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 13 2026-02-27 13:04:58,988 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,988 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 13 2026-02-27 13:04:58,988 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,988 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 13 2026-02-27 13:04:58,988 INFO SenderThread:1518997 [sender.py:transition_state():626] send defer: 14 2026-02-27 13:04:58,988 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:04:58,988 INFO HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 14 2026-02-27 13:04:58,988 DEBUG SenderThread:1518997 [sender.py:send():391] send: final 2026-02-27 13:04:58,988 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: defer 2026-02-27 13:04:58,988 INFO SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 14 2026-02-27 13:04:58,989 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit 2026-02-27 13:04:58,989 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit 2026-02-27 13:04:58,990 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: server_info 2026-02-27 13:04:58,990 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: server_info 2026-02-27 13:04:58,990 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit 2026-02-27 13:04:58,990 DEBUG SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit 2026-02-27 13:04:58,990 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: get_summary 2026-02-27 13:04:58,991 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: sampled_history 2026-02-27 13:04:58,991 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: internal_messages 2026-02-27 13:04:58,991 INFO MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4088] rendering history 2026-02-27 13:04:58,991 INFO MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4120] rendering summary 2026-02-27 13:04:58,991 DEBUG HandlerThread:1518997 [handler.py:handle_request():158] handle_request: shutdown 2026-02-27 13:04:58,991 INFO HandlerThread:1518997 [handler.py:finish():884] shutting down handler 2026-02-27 13:04:59,990 INFO WriterThread:1518997 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb 2026-02-27 13:04:59,990 INFO SenderThread:1518997 [sender.py:finish():1631] shutting down sender