2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Configure stats pid to 1291330
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/wandb/settings
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False}
2026-02-11 07:41:02,250 WARNING MainThread:1291330 [wandb_setup.py:_flush():77] Could not find program at -m main
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'}
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_setup.py:_flush():77] Applying login settings: {}
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/logs/debug.log
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/logs/debug-internal.log
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:init():608] calling init triggers
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 2e-05, 'batch_size': 4, 'max_epochs': -1, 'max_steps': 240000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/worldmem_minecraft_dataset', 'n_frames': 8, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft', 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 8, 'memory_condition_length': 8, 'customized_validation': True, 'add_timestamp_embedding': True, 'wo_updown': True}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 600, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, '_name': 'df_video_worldmemminecraft', 'n_tokens': 8, 'memory_condition_length': 8, 'relative_embedding': True, 'log_video': True, 'add_timestamp_embedding': True, 'metrics': ['lpips', 'psnr']}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': 'ot7jqmgn', 'load': None, 'name': 'train', 'output_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set'}
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:init():658] starting backend
2026-02-11 07:41:02,250 INFO    MainThread:1291330 [wandb_init.py:init():662] setting up manager
2026-02-11 07:41:02,290 INFO    MainThread:1291330 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-02-11 07:41:02,300 INFO    MainThread:1291330 [wandb_init.py:init():670] backend started and connected
2026-02-11 07:41:02,388 INFO    MainThread:1291330 [wandb_init.py:init():768] updated telemetry
2026-02-11 07:41:02,447 INFO    MainThread:1291330 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
2026-02-11 07:41:03,016 ERROR   MainThread:1291330 [wandb_init.py:init():827] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-11 07:41:03,017 ERROR   MainThread:1291330 [wandb_init.py:init():1239] error in wandb.init()
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1229, in init
    return wi.init()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 833, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-11 07:41:04,359 WARNING MsgRouterThr:1291330 [router.py:message_loop():77] message_loop has been closed