Hugging Face's logo

Amshaker
/
WorldMem

Model card Files
xet
 Community
WorldMem
File size: 33,814 Bytes
f0bbb7a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
2026-02-27 13:48:42,881 INFO    StreamThr :1845920 [internal.py:wandb_internal():85] W&B internal server running at pid: 1845920, started at: 2026-02-27 13:48:42.879498
2026-02-27 13:48:42,882 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status
2026-02-27 13:48:43,210 INFO    WriterThread:1845920 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_134842-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:48:43,304 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: run_start
2026-02-27 13:48:43,324 DEBUG   HandlerThread:1845920 [system_info.py:__init__():26] System info init
2026-02-27 13:48:43,324 DEBUG   HandlerThread:1845920 [system_info.py:__init__():41] System info init done
2026-02-27 13:48:43,324 INFO    HandlerThread:1845920 [system_monitor.py:start():194] Starting system monitor
2026-02-27 13:48:43,324 INFO    SystemMonitor:1845920 [system_monitor.py:_start():158] Starting system asset monitoring threads
2026-02-27 13:48:43,325 INFO    HandlerThread:1845920 [system_monitor.py:probe():214] Collecting system info
2026-02-27 13:48:43,325 INFO    SystemMonitor:1845920 [interfaces.py:start():188] Started cpu monitoring
2026-02-27 13:48:43,326 INFO    SystemMonitor:1845920 [interfaces.py:start():188] Started disk monitoring
2026-02-27 13:48:43,326 INFO    SystemMonitor:1845920 [interfaces.py:start():188] Started gpu monitoring
2026-02-27 13:48:43,326 INFO    SystemMonitor:1845920 [interfaces.py:start():188] Started memory monitoring
2026-02-27 13:48:43,327 INFO    SystemMonitor:1845920 [interfaces.py:start():188] Started network monitoring
2026-02-27 13:48:43,432 DEBUG   HandlerThread:1845920 [system_info.py:probe():152] Probing system
2026-02-27 13:48:43,434 DEBUG   HandlerThread:1845920 [system_info.py:_probe_git():137] Probing git
2026-02-27 13:48:43,479 DEBUG   HandlerThread:1845920 [system_info.py:_probe_git():145] Probing git done
2026-02-27 13:48:43,479 DEBUG   HandlerThread:1845920 [system_info.py:probe():200] Probing system done
2026-02-27 13:48:43,479 DEBUG   HandlerThread:1845920 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.94.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.10.19', 'heartbeatAt': '2026-02-27T12:48:43.432625', 'startedAt': '2026-02-27T12:48:42.835440', 'docker': None, 'cuda': None, 'args': ('+name=train_stage_a_mamba', 'algorithm=df_video_mamba3stage', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'dataset=video_minecraft_latent', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.n_frames=200', '+dataset.n_frames_valid=700', '+dataset.angle_range=110', '+dataset.pos_range=8', '+dataset.customized_validation=true', '+dataset.add_timestamp_embedding=true', 'dataset.use_explicit_memory_frames=false', 'algorithm.training_stage=stage_a_memory_pretrain', 'algorithm.use_mamba_memory_pipeline=true', 'algorithm.use_oracle_pose_eval=false', 'algorithm.use_precomputed_features=true', 'algorithm.context_frames=1', '+algorithm.require_pose_prediction=false', '+algorithm.use_memory_attention=false', '+algorithm.relative_embedding=false', '+algorithm.memory_retrieval_topk=8', 'experiment.training.batch_size=64', 'experiment.training.checkpointing.every_n_train_steps=2500', 'experiment.training.max_steps=700000', '+output_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'), 'state': 'running', 'program': '-m main', 'codePathLocal': None, 'git': {'remote': 'https://huggingface.co/BonanDing/WorldMem_Repro', 'commit': 'f0d488aceb5425378cec21c5d35bd92f64346f51'}, 'email': 'fahad.khan@liu.se', 'root': '/proj/cvl/users/x_fahkh2/WorldMem_Repro', 'host': 'node043', 'username': 'x_fahkh', 'executable': '/proj/cvl/users/x_fahkh2/envs/worldmem/bin/python', 'cpu_count': 128, 'cpu_count_logical': 256, 'cpu_freq': {'current': 2285.5670195312505, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3391.832, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3389.842, 'min': 1500.0, 'max': 2250.0}, {'current': 3390.122, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.725, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3392.782, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3386.36, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.645, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 29.166847229003906, 'used': 15.821720123291016}}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 2, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1007.6507377624512}}
2026-02-27 13:48:43,480 INFO    HandlerThread:1845920 [system_monitor.py:probe():224] Finished collecting system info
2026-02-27 13:48:43,480 INFO    HandlerThread:1845920 [system_monitor.py:probe():227] Publishing system info
2026-02-27 13:48:43,480 DEBUG   HandlerThread:1845920 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment
2026-02-27 13:48:50,058 DEBUG   HandlerThread:1845920 [system_info.py:_save_conda():224] Saving conda packages done
2026-02-27 13:48:50,064 INFO    HandlerThread:1845920 [system_monitor.py:probe():229] Finished publishing system info
2026-02-27 13:48:50,089 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:48:50,089 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: keepalive
2026-02-27 13:48:50,090 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:48:51,668 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: python_packages
2026-02-27 13:48:53,090 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:48:53,090 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:48:58,091 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:48:58,092 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:03,092 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:03,093 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:08,093 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:08,093 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:13,094 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:13,094 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:18,095 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:18,095 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:23,096 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:23,096 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:28,097 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:28,097 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:33,098 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:33,098 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:38,099 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:38,099 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:43,100 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:43,100 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:43,327 DEBUG   SystemMonitor:1845920 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2026-02-27 13:49:48,101 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:48,101 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:53,102 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:53,102 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:49:58,103 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:49:58,103 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:50:03,104 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:50:03,104 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:50:03,595 DEBUG   SenderThread:1845920 [sender.py:send():391] send: exit
2026-02-27 13:50:03,595 INFO    SenderThread:1845920 [sender.py:send_exit():598] handling exit code: 0
2026-02-27 13:50:03,595 INFO    SenderThread:1845920 [sender.py:send_exit():600] handling runtime: 80
2026-02-27 13:50:03,600 INFO    SenderThread:1845920 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
2026-02-27 13:50:03,601 INFO    SenderThread:1845920 [sender.py:send_exit():606] send defer
2026-02-27 13:50:03,601 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:03,601 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 0
2026-02-27 13:50:03,601 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:03,601 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 0
2026-02-27 13:50:03,601 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 1
2026-02-27 13:50:03,601 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:03,601 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 1
2026-02-27 13:50:03,601 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:03,601 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 1
2026-02-27 13:50:03,601 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 2
2026-02-27 13:50:03,601 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:03,601 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 2
2026-02-27 13:50:03,601 INFO    HandlerThread:1845920 [system_monitor.py:finish():203] Stopping system monitor
2026-02-27 13:50:03,601 DEBUG   SystemMonitor:1845920 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2026-02-27 13:50:03,602 DEBUG   SystemMonitor:1845920 [system_monitor.py:_start():183] Publishing last batch of metrics
2026-02-27 13:50:03,605 INFO    HandlerThread:1845920 [interfaces.py:finish():200] Joined cpu monitor
2026-02-27 13:50:03,605 INFO    HandlerThread:1845920 [interfaces.py:finish():200] Joined disk monitor
2026-02-27 13:50:04,450 INFO    HandlerThread:1845920 [interfaces.py:finish():200] Joined gpu monitor
2026-02-27 13:50:04,450 INFO    HandlerThread:1845920 [interfaces.py:finish():200] Joined memory monitor
2026-02-27 13:50:04,450 INFO    HandlerThread:1845920 [interfaces.py:finish():200] Joined network monitor
2026-02-27 13:50:04,451 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,451 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 2
2026-02-27 13:50:04,451 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 3
2026-02-27 13:50:04,451 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,451 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 3
2026-02-27 13:50:04,452 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,452 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 3
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 4
2026-02-27 13:50:04,453 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,453 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 4
2026-02-27 13:50:04,453 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 4
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 5
2026-02-27 13:50:04,453 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,453 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 5
2026-02-27 13:50:04,453 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 5
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 6
2026-02-27 13:50:04,453 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,453 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 6
2026-02-27 13:50:04,453 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 6
2026-02-27 13:50:04,453 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 7
2026-02-27 13:50:04,453 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: status_report
2026-02-27 13:50:04,453 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,453 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 7
2026-02-27 13:50:04,454 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: status_report
2026-02-27 13:50:04,454 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 7
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 8
2026-02-27 13:50:04,454 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,454 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 8
2026-02-27 13:50:04,454 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 8
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 9
2026-02-27 13:50:04,454 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,454 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 9
2026-02-27 13:50:04,454 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 9
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 10
2026-02-27 13:50:04,454 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,454 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 10
2026-02-27 13:50:04,454 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 10
2026-02-27 13:50:04,454 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 11
2026-02-27 13:50:04,454 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,454 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 11
2026-02-27 13:50:04,455 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 11
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 12
2026-02-27 13:50:04,455 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,455 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 12
2026-02-27 13:50:04,455 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 12
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 13
2026-02-27 13:50:04,455 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,455 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 13
2026-02-27 13:50:04,455 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 13
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:transition_state():626] send defer: 14
2026-02-27 13:50:04,455 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: defer
2026-02-27 13:50:04,455 INFO    HandlerThread:1845920 [handler.py:handle_request_defer():184] handle defer: 14
2026-02-27 13:50:04,455 DEBUG   SenderThread:1845920 [sender.py:send():391] send: final
2026-02-27 13:50:04,455 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: defer
2026-02-27 13:50:04,455 INFO    SenderThread:1845920 [sender.py:send_request_defer():622] handle sender defer: 14
2026-02-27 13:50:04,456 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:50:04,456 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:50:04,456 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: server_info
2026-02-27 13:50:04,457 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: server_info
2026-02-27 13:50:04,457 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: poll_exit
2026-02-27 13:50:04,457 DEBUG   SenderThread:1845920 [sender.py:send_request():418] send_request: poll_exit
2026-02-27 13:50:04,457 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: get_summary
2026-02-27 13:50:04,457 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: sampled_history
2026-02-27 13:50:04,458 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: internal_messages
2026-02-27 13:50:04,458 INFO    MainThread:1845920 [wandb_run.py:_footer_history_summary_info():4088] rendering history
2026-02-27 13:50:04,458 INFO    MainThread:1845920 [wandb_run.py:_footer_history_summary_info():4120] rendering summary
2026-02-27 13:50:04,459 DEBUG   HandlerThread:1845920 [handler.py:handle_request():158] handle_request: shutdown
2026-02-27 13:50:04,459 INFO    HandlerThread:1845920 [handler.py:finish():884] shutting down handler
2026-02-27 13:50:05,457 INFO    WriterThread:1845920 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/offline-run-20260227_134842-lxvmjzq8/run-lxvmjzq8.wandb
2026-02-27 13:50:05,457 INFO    SenderThread:1845920 [sender.py:finish():1631] shutting down sender