wandb/run-20260211_074009-ku1rnk9p/logs/debug.log

2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Configure stats pid to 1284257
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/wandb/settings
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False}
2026-02-11 07:40:09,452 WARNING MainThread:1284257 [wandb_setup.py:_flush():77] Could not find program at -m main
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'}
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_setup.py:_flush():77] Applying login settings: {}
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074009-ku1rnk9p/logs/debug.log
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074009-ku1rnk9p/logs/debug-internal.log
2026-02-11 07:40:09,452 INFO    MainThread:1284257 [wandb_init.py:init():608] calling init triggers
2026-02-11 07:40:09,453 INFO    MainThread:1284257 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 2e-05, 'batch_size': 4, 'max_epochs': -1, 'max_steps': 120000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/worldmem_minecraft_dataset', 'n_frames': 8, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft', 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 2, 'memory_condition_length': 8, 'customized_validation': True, 'add_timestamp_embedding': True, 'wo_updown': True}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 600, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, '_name': 'df_video_worldmemminecraft', 'n_tokens': 8, 'memory_condition_length': 8, 'relative_embedding': True, 'log_video': True, 'add_timestamp_embedding': True, 'metrics': ['lpips', 'psnr']}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': None, 'load': None, 'name': 'train', 'diffusion_model_path': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/checkpoints/diffusion_only.ckpt', 'vae_path': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/checkpoints/vae_only.ckpt', 'customized_load': True, 'seperate_load': True, 'zero_init_gate': True, 'output_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set'}
2026-02-11 07:40:09,453 INFO    MainThread:1284257 [wandb_init.py:init():658] starting backend
2026-02-11 07:40:09,453 INFO    MainThread:1284257 [wandb_init.py:init():662] setting up manager
2026-02-11 07:40:09,468 INFO    MainThread:1284257 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-02-11 07:40:09,475 INFO    MainThread:1284257 [wandb_init.py:init():670] backend started and connected
2026-02-11 07:40:09,505 INFO    MainThread:1284257 [wandb_init.py:init():768] updated telemetry
2026-02-11 07:40:09,565 INFO    MainThread:1284257 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
2026-02-11 07:40:10,115 ERROR   MainThread:1284257 [wandb_init.py:init():827] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-11 07:40:10,115 ERROR   MainThread:1284257 [wandb_init.py:init():1239] error in wandb.init()
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1229, in init
    return wi.init()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 833, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-11 07:40:12,653 WARNING MsgRouterThr:1284257 [router.py:message_loop():77] message_loop has been closed