2026-02-27 13:48:42,881 INFO StreamThr :1845920 [internal.py:wandb_internal():85] W&B internal server running at pid: 1845920, started at: 2026-02-27 13:48:42.879498 2026-02-27 13:48:42,882 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status 2026-02-27 13:48:43,210 INFO WriterThread:1845920 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_134842-lxvmjzq8/run-lxvmjzq8.wandb 2026-02-27 13:48:43,304 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: run_start 2026-02-27 13:48:43,324 DEBUG HandlerThread:1845920 [system_info.py:__init__():26] System info init 2026-02-27 13:48:43,324 DEBUG HandlerThread:1845920 [system_info.py:__init__():41] System info init done 2026-02-27 13:48:43,324 INFO HandlerThread:1845920 [system_monitor.py:start():194] Starting system monitor 2026-02-27 13:48:43,324 INFO SystemMonitor:1845920 [system_monitor.py:_start():158] Starting system asset monitoring threads 2026-02-27 13:48:43,325 INFO HandlerThread:1845920 [system_monitor.py:probe():214] Collecting system info 2026-02-27 13:48:43,325 INFO SystemMonitor:1845920 [interfaces.py:start():188] Started cpu monitoring 2026-02-27 13:48:43,326 INFO SystemMonitor:1845920 [interfaces.py:start():188] Started disk monitoring 2026-02-27 13:48:43,326 INFO SystemMonitor:1845920 [interfaces.py:start():188] Started gpu monitoring 2026-02-27 13:48:43,326 INFO SystemMonitor:1845920 [interfaces.py:start():188] Started memory monitoring 2026-02-27 13:48:43,327 INFO SystemMonitor:1845920 [interfaces.py:start():188] Started network monitoring 2026-02-27 13:48:43,432 DEBUG HandlerThread:1845920 [system_info.py:probe():152] Probing system 2026-02-27 13:48:43,434 DEBUG HandlerThread:1845920 [system_info.py:_probe_git():137] Probing git 2026-02-27 13:48:43,479 DEBUG HandlerThread:1845920 [system_info.py:_probe_git():145] Probing git done 2026-02-27 13:48:43,479 DEBUG HandlerThread:1845920 [system_info.py:probe():200] Probing system done 2026-02-27 13:48:43,479 DEBUG HandlerThread:1845920 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.94.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.10.19', 'heartbeatAt': '2026-02-27T12:48:43.432625', 'startedAt': '2026-02-27T12:48:42.835440', 'docker': None, 'cuda': None, 'args': ('+name=train_stage_a_mamba', 'algorithm=df_video_mamba3stage', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'dataset=video_minecraft_latent', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.n_frames=200', '+dataset.n_frames_valid=700', '+dataset.angle_range=110', '+dataset.pos_range=8', '+dataset.customized_validation=true', '+dataset.add_timestamp_embedding=true', 'dataset.use_explicit_memory_frames=false', 'algorithm.training_stage=stage_a_memory_pretrain', 'algorithm.use_mamba_memory_pipeline=true', 'algorithm.use_oracle_pose_eval=false', 'algorithm.use_precomputed_features=true', 'algorithm.context_frames=1', '+algorithm.require_pose_prediction=false', '+algorithm.use_memory_attention=false', '+algorithm.relative_embedding=false', '+algorithm.memory_retrieval_topk=8', 'experiment.training.batch_size=64', 'experiment.training.checkpointing.every_n_train_steps=2500', 'experiment.training.max_steps=700000', '+output_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'), 'state': 'running', 'program': '-m main', 'codePathLocal': None, 'git': {'remote': 'https://huggingface.co/BonanDing/WorldMem_Repro', 'commit': 'f0d488aceb5425378cec21c5d35bd92f64346f51'}, 'email': 'fahad.khan@liu.se', 'root': '/proj/cvl/users/x_fahkh2/WorldMem_Repro', 'host': 'node043', 'username': 'x_fahkh', 'executable': '/proj/cvl/users/x_fahkh2/envs/worldmem/bin/python', 'cpu_count': 128, 'cpu_count_logical': 256, 'cpu_freq': {'current': 2285.5670195312505, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3391.832, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3389.842, 'min': 1500.0, 'max': 2250.0}, {'current': 3390.122, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.725, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.782, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.36, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.645, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 29.166847229003906, 'used': 15.821720123291016}}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 2, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1007.6507377624512}} 2026-02-27 13:48:43,480 INFO HandlerThread:1845920 [system_monitor.py:probe():224] Finished collecting system info 2026-02-27 13:48:43,480 INFO HandlerThread:1845920 [system_monitor.py:probe():227] Publishing system info 2026-02-27 13:48:43,480 DEBUG HandlerThread:1845920 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2026-02-27 13:48:50,058 DEBUG HandlerThread:1845920 [system_info.py:_save_conda():224] Saving conda packages done 2026-02-27 13:48:50,064 INFO HandlerThread:1845920 [system_monitor.py:probe():229] Finished publishing system info 2026-02-27 13:48:50,089 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:48:50,089 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: keepalive 2026-02-27 13:48:50,090 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:48:51,668 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: python_packages 2026-02-27 13:48:53,090 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:48:53,090 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:48:58,091 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:48:58,092 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:03,092 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:03,093 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:08,093 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:08,093 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:13,094 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:13,094 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:18,095 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:18,095 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:23,096 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:23,096 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:28,097 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:28,097 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:33,098 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:33,098 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:38,099 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:38,099 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:43,100 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:43,100 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:43,327 DEBUG SystemMonitor:1845920 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2026-02-27 13:49:48,101 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:48,101 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:53,102 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:53,102 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:49:58,103 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:49:58,103 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:50:03,104 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:50:03,104 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:50:03,595 DEBUG SenderThread:1845920 [sender.py:send():391] send: exit 2026-02-27 13:50:03,595 INFO SenderThread:1845920 [sender.py:send_exit():598] handling exit code: 0 2026-02-27 13:50:03,595 INFO SenderThread:1845920 [sender.py:send_exit():600] handling runtime: 80 2026-02-27 13:50:03,600 INFO SenderThread:1845920 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2026-02-27 13:50:03,601 INFO SenderThread:1845920 [sender.py:send_exit():606] send defer 2026-02-27 13:50:03,601 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:03,601 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 0 2026-02-27 13:50:03,601 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:03,601 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 0 2026-02-27 13:50:03,601 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 1 2026-02-27 13:50:03,601 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:03,601 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 1 2026-02-27 13:50:03,601 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:03,601 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 1 2026-02-27 13:50:03,601 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 2 2026-02-27 13:50:03,601 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:03,601 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 2 2026-02-27 13:50:03,601 INFO HandlerThread:1845920 [system_monitor.py:finish():203] Stopping system monitor 2026-02-27 13:50:03,601 DEBUG SystemMonitor:1845920 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2026-02-27 13:50:03,602 DEBUG SystemMonitor:1845920 [system_monitor.py:_start():183] Publishing last batch of metrics 2026-02-27 13:50:03,605 INFO HandlerThread:1845920 [interfaces.py:finish():200] Joined cpu monitor 2026-02-27 13:50:03,605 INFO HandlerThread:1845920 [interfaces.py:finish():200] Joined disk monitor 2026-02-27 13:50:04,450 INFO HandlerThread:1845920 [interfaces.py:finish():200] Joined gpu monitor 2026-02-27 13:50:04,450 INFO HandlerThread:1845920 [interfaces.py:finish():200] Joined memory monitor 2026-02-27 13:50:04,450 INFO HandlerThread:1845920 [interfaces.py:finish():200] Joined network monitor 2026-02-27 13:50:04,451 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,451 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 2 2026-02-27 13:50:04,451 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 3 2026-02-27 13:50:04,451 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,451 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 3 2026-02-27 13:50:04,452 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,452 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 3 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 4 2026-02-27 13:50:04,453 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,453 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 4 2026-02-27 13:50:04,453 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 4 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 5 2026-02-27 13:50:04,453 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,453 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 5 2026-02-27 13:50:04,453 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 5 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 6 2026-02-27 13:50:04,453 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,453 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 6 2026-02-27 13:50:04,453 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 6 2026-02-27 13:50:04,453 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 7 2026-02-27 13:50:04,453 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report 2026-02-27 13:50:04,453 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,453 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 7 2026-02-27 13:50:04,454 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: status_report 2026-02-27 13:50:04,454 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 7 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 8 2026-02-27 13:50:04,454 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,454 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 8 2026-02-27 13:50:04,454 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 8 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 9 2026-02-27 13:50:04,454 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,454 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 9 2026-02-27 13:50:04,454 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 9 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 10 2026-02-27 13:50:04,454 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,454 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 10 2026-02-27 13:50:04,454 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 10 2026-02-27 13:50:04,454 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 11 2026-02-27 13:50:04,454 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,454 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 11 2026-02-27 13:50:04,455 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 11 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 12 2026-02-27 13:50:04,455 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,455 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 12 2026-02-27 13:50:04,455 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 12 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 13 2026-02-27 13:50:04,455 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,455 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 13 2026-02-27 13:50:04,455 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 13 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:transition_state():626] send defer: 14 2026-02-27 13:50:04,455 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer 2026-02-27 13:50:04,455 INFO HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 14 2026-02-27 13:50:04,455 DEBUG SenderThread:1845920 [sender.py:send():391] send: final 2026-02-27 13:50:04,455 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: defer 2026-02-27 13:50:04,455 INFO SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 14 2026-02-27 13:50:04,456 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: poll_exit 2026-02-27 13:50:04,456 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: poll_exit 2026-02-27 13:50:04,456 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: server_info 2026-02-27 13:50:04,457 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: server_info 2026-02-27 13:50:04,457 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: poll_exit 2026-02-27 13:50:04,457 DEBUG SenderThread:1845920 [sender.py:send_request():418] send_request: poll_exit 2026-02-27 13:50:04,457 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: get_summary 2026-02-27 13:50:04,457 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: sampled_history 2026-02-27 13:50:04,458 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: internal_messages 2026-02-27 13:50:04,458 INFO MainThread:1845920 [wandb_run.py:_footer_history_summary_info():4088] rendering history 2026-02-27 13:50:04,458 INFO MainThread:1845920 [wandb_run.py:_footer_history_summary_info():4120] rendering summary 2026-02-27 13:50:04,459 DEBUG HandlerThread:1845920 [handler.py:handle_request():158] handle_request: shutdown 2026-02-27 13:50:04,459 INFO HandlerThread:1845920 [handler.py:finish():884] shutting down handler 2026-02-27 13:50:05,457 INFO WriterThread:1845920 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_134842-lxvmjzq8/run-lxvmjzq8.wandb 2026-02-27 13:50:05,457 INFO SenderThread:1845920 [sender.py:finish():1631] shutting down sender