| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9 | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Configure stats pid to 1284257 | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/wandb/settings | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from environment variables: {} | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False} | |
| 2026-02-11 07:40:09,452 WARNING MainThread:1284257 [wandb_setup.py:_flush():77] Could not find program at -m main | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'} | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_setup.py:_flush():77] Applying login settings: {} | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074009-ku1rnk9p/logs/debug.log | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074009-ku1rnk9p/logs/debug-internal.log | |
| 2026-02-11 07:40:09,452 INFO MainThread:1284257 [wandb_init.py:init():608] calling init triggers | |
| 2026-02-11 07:40:09,453 INFO MainThread:1284257 [wandb_init.py:init():615] wandb.init called with sweep_config: {} | |
| config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 2e-05, 'batch_size': 4, 'max_epochs': -1, 'max_steps': 120000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/worldmem_minecraft_dataset', 'n_frames': 8, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft', 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 2, 'memory_condition_length': 8, 'customized_validation': True, 'add_timestamp_embedding': True, 'wo_updown': True}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 600, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, '_name': 'df_video_worldmemminecraft', 'n_tokens': 8, 'memory_condition_length': 8, 'relative_embedding': True, 'log_video': True, 'add_timestamp_embedding': True, 'metrics': ['lpips', 'psnr']}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': None, 'load': None, 'name': 'train', 'diffusion_model_path': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/checkpoints/diffusion_only.ckpt', 'vae_path': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/checkpoints/vae_only.ckpt', 'customized_load': True, 'seperate_load': True, 'zero_init_gate': True, 'output_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set'} | |
| 2026-02-11 07:40:09,453 INFO MainThread:1284257 [wandb_init.py:init():658] starting backend | |
| 2026-02-11 07:40:09,453 INFO MainThread:1284257 [wandb_init.py:init():662] setting up manager | |
| 2026-02-11 07:40:09,468 INFO MainThread:1284257 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2026-02-11 07:40:09,475 INFO MainThread:1284257 [wandb_init.py:init():670] backend started and connected | |
| 2026-02-11 07:40:09,505 INFO MainThread:1284257 [wandb_init.py:init():768] updated telemetry | |
| 2026-02-11 07:40:09,565 INFO MainThread:1284257 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout | |
| 2026-02-11 07:40:10,115 ERROR MainThread:1284257 [wandb_init.py:init():827] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) | |
| 2026-02-11 07:40:10,115 ERROR MainThread:1284257 [wandb_init.py:init():1239] error in wandb.init() | |
| Traceback (most recent call last): | |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1229, in init | |
| return wi.init() | |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 833, in init | |
| raise error | |
| wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) | |
| 2026-02-11 07:40:12,653 WARNING MsgRouterThr:1284257 [router.py:message_loop():77] message_loop has been closed | |