Hugging Face's logo

Amshaker
/
WorldMem

Model card Files
xet
 Community
WorldMem
File size: 30,670 Bytes
f0bbb7a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
2026-02-27 13:04:46,338 INFO    StreamThr :1518997 [internal.py:wandb_internal():85] W&B internal server running at pid: 1518997, started at: 2026-02-27 13:04:46.335438
2026-02-27 13:04:46,340 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status
2026-02-27 13:04:46,365 INFO    WriterThread:1518997 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:04:46,476 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: run_start
2026-02-27 13:04:46,515 DEBUG   HandlerThread:1518997 [system_info.py:__init__():26] System info init
2026-02-27 13:04:46,515 DEBUG   HandlerThread:1518997 [system_info.py:__init__():41] System info init done
2026-02-27 13:04:46,516 INFO    HandlerThread:1518997 [system_monitor.py:start():194] Starting system monitor
2026-02-27 13:04:46,516 INFO    SystemMonitor:1518997 [system_monitor.py:_start():158] Starting system asset monitoring threads
2026-02-27 13:04:46,516 INFO    HandlerThread:1518997 [system_monitor.py:probe():214] Collecting system info
2026-02-27 13:04:46,517 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started cpu monitoring
2026-02-27 13:04:46,517 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started disk monitoring
2026-02-27 13:04:46,518 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started gpu monitoring
2026-02-27 13:04:46,518 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started memory monitoring
2026-02-27 13:04:46,519 INFO    SystemMonitor:1518997 [interfaces.py:start():188] Started network monitoring
2026-02-27 13:04:46,640 DEBUG   HandlerThread:1518997 [system_info.py:probe():152] Probing system
2026-02-27 13:04:46,642 DEBUG   HandlerThread:1518997 [system_info.py:_probe_git():137] Probing git
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_info.py:_probe_git():145] Probing git done
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_info.py:probe():200] Probing system done
2026-02-27 13:04:46,725 DEBUG   HandlerThread:1518997 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.94.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.10.19', 'heartbeatAt': '2026-02-27T12:04:46.640569', 'startedAt': '2026-02-27T12:04:46.285930', 'docker': None, 'cuda': None, 'args': ('+name=train_stage_a_mamba', 'algorithm=df_video_mamba3stage', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'dataset=video_minecraft_latent', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=200', '+dataset.n_frames_valid=700', '+dataset.angle_range=110', '+dataset.pos_range=8', '+dataset.customized_validation=true', '+dataset.add_timestamp_embedding=true', 'dataset.use_explicit_memory_frames=false', 'algorithm.training_stage=stage_a_memory_pretrain', 'algorithm.use_mamba_memory_pipeline=true', 'algorithm.use_oracle_pose_eval=false', 'algorithm.use_precomputed_features=true', 'algorithm.context_frames=1', '+algorithm.require_pose_prediction=false', '+algorithm.use_memory_attention=false', '+algorithm.relative_embedding=false', '+algorithm.memory_retrieval_topk=8', 'experiment.training.batch_size=64', 'experiment.training.checkpointing.every_n_train_steps=2500', 'experiment.training.max_steps=700000', '+output_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'), 'state': 'running', 'program': '-m main', 'codePathLocal': None, 'git': {'remote': 'https://huggingface.co/BonanDing/WorldMem_Repro', 'commit': 'f0d488aceb5425378cec21c5d35bd92f64346f51'}, 'email': 'fahad.khan@liu.se', 'root': '/proj/cvl/users/x_fahkh2/WorldMem_Repro', 'host': 'node020', 'username': 'x_fahkh', 'executable': '/proj/cvl/users/x_fahkh2/envs/worldmem/bin/python', 'cpu_count': 128, 'cpu_count_logical': 256, 'cpu_freq': {'current': 2440.1533984375005, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3380.836, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.286, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.06, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.554, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.695, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.677, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.853, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.748, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.632, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.626, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.484, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.494, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.038, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.444, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.89, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.093, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.439, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.443, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3387.401, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.58, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.459, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.238, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.572, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.965, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.185, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.956, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.13, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.513, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.297, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.601, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.412, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.741, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.294, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.686, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.043, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3381.512, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.025, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 29.166847229003906, 'used': 15.824077606201172}}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 2, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1007.6652145385742}}
2026-02-27 13:04:46,725 INFO    HandlerThread:1518997 [system_monitor.py:probe():224] Finished collecting system info
2026-02-27 13:04:46,725 INFO    HandlerThread:1518997 [system_monitor.py:probe():227] Publishing system info
2026-02-27 13:04:46,726 DEBUG   HandlerThread:1518997 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment
2026-02-27 13:04:53,157 DEBUG   HandlerThread:1518997 [system_info.py:_save_conda():224] Saving conda packages done
2026-02-27 13:04:53,164 INFO    HandlerThread:1518997 [system_monitor.py:probe():229] Finished publishing system info
2026-02-27 13:04:53,178 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:53,178 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: keepalive
2026-02-27 13:04:53,178 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:54,467 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: python_packages
2026-02-27 13:04:57,179 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:57,180 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:58,193 DEBUG   SenderThread:1518997 [sender.py:send():391] send: exit
2026-02-27 13:04:58,193 INFO    SenderThread:1518997 [sender.py:send_exit():598] handling exit code: 1
2026-02-27 13:04:58,194 INFO    SenderThread:1518997 [sender.py:send_exit():600] handling runtime: 11
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:send_exit():606] send defer
2026-02-27 13:04:58,197 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,197 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 0
2026-02-27 13:04:58,197 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 0
2026-02-27 13:04:58,197 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 1
2026-02-27 13:04:58,197 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,197 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 1
2026-02-27 13:04:58,198 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,198 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 1
2026-02-27 13:04:58,198 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 2
2026-02-27 13:04:58,198 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,198 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 2
2026-02-27 13:04:58,198 INFO    HandlerThread:1518997 [system_monitor.py:finish():203] Stopping system monitor
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2026-02-27 13:04:58,198 DEBUG   SystemMonitor:1518997 [system_monitor.py:_start():183] Publishing last batch of metrics
2026-02-27 13:04:58,199 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined cpu monitor
2026-02-27 13:04:58,200 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined disk monitor
2026-02-27 13:04:58,983 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined gpu monitor
2026-02-27 13:04:58,983 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined memory monitor
2026-02-27 13:04:58,984 INFO    HandlerThread:1518997 [interfaces.py:finish():200] Joined network monitor
2026-02-27 13:04:58,984 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 2
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 3
2026-02-27 13:04:58,984 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,984 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 3
2026-02-27 13:04:58,984 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 3
2026-02-27 13:04:58,984 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 4
2026-02-27 13:04:58,984 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 4
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 4
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 5
2026-02-27 13:04:58,985 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 5
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 5
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 6
2026-02-27 13:04:58,985 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,985 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 6
2026-02-27 13:04:58,985 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 6
2026-02-27 13:04:58,985 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 7
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 7
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 7
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 8
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 8
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 8
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 9
2026-02-27 13:04:58,986 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,986 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 9
2026-02-27 13:04:58,986 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 9
2026-02-27 13:04:58,986 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 10
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 10
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 10
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 11
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 11
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 11
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 12
2026-02-27 13:04:58,987 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,987 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 12
2026-02-27 13:04:58,987 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 12
2026-02-27 13:04:58,987 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 13
2026-02-27 13:04:58,988 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,988 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 13
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 13
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:transition_state():626] send defer: 14
2026-02-27 13:04:58,988 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:04:58,988 INFO    HandlerThread:1518997 [handler.py:handle_request_defer():184] handle defer: 14
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send():391] send: final
2026-02-27 13:04:58,988 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: defer
2026-02-27 13:04:58,988 INFO    SenderThread:1518997 [sender.py:send_request_defer():622] handle sender defer: 14
2026-02-27 13:04:58,989 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:04:58,989 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: server_info
2026-02-27 13:04:58,990 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: server_info
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   SenderThread:1518997 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:04:58,990 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: get_summary
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: sampled_history
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: internal_messages
2026-02-27 13:04:58,991 INFO    MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4088] rendering history
2026-02-27 13:04:58,991 INFO    MainThread:1518997 [wandb_run.py:_footer_history_summary_info():4120] rendering summary
2026-02-27 13:04:58,991 DEBUG   HandlerThread:1518997 [handler.py:handle_request():158] handle_request: shutdown
2026-02-27 13:04:58,991 INFO    HandlerThread:1518997 [handler.py:finish():884] shutting down handler
2026-02-27 13:04:59,990 INFO    WriterThread:1518997 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_130446-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:04:59,990 INFO    SenderThread:1518997 [sender.py:finish():1631] shutting down sender