2026-02-27 13:46:36,573 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Configure stats pid to 1829233
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldMem_Repro/wandb/settings
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False}
2026-02-27 13:46:36,574 WARNING MainThread:1829233 [wandb_setup.py:_flush():77] Could not find program at -m main
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'}
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_setup.py:_flush():77] Applying login settings: {}
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_134636-lxvmjzq8/logs/debug.log
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_134636-lxvmjzq8/logs/debug-internal.log
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:init():608] calling init triggers
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 2e-05, 'batch_size': 64, 'max_epochs': -1, 'max_steps': 700000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'n_frames': 200, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft_latent', 'precomputed_feature_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'use_explicit_memory_frames': False, 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 8, 'customized_validation': True, 'add_timestamp_embedding': True}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 1, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, 'memory_condition_length': 0, 'log_video': True, 'use_compressed_causal_memory': False, 'compressed_memory_dim': 256, 'compressed_memory_slots': 64, 'compressed_memory_topk': 4, 'use_mamba_memory_pipeline': True, 'training_stage': 'stage_a_memory_pretrain', 'stage_c_memory_aux_weight': 0.1, 'diff_window_size': 8, 'memory_gap_aux_weight': 0.1, 'memory_gap_aux_weight_init': 0.5, 'memory_gap_aux_weight_final': 0.1, 'memory_gap_aux_decay_fraction': 0.3, 'memory_gap_aux_decay_steps': 0, 'memory_gap_aux_exp_k': 5.0, 'use_precomputed_features': True, 'mamba_latent_channels': 16, 'mamba_model_dim': 256, 'mamba_depth': 4, 'mamba_cond_dim': 256, 'mamba_d_state': 16, 'mamba_d_conv': 4, 'mamba_expand': 2, 'allow_mamba_fallback': False, 'strict_causal_training': True, 'strict_causal_evaluation': True, 'use_oracle_pose_eval': False, 'enable_memory_noise_curriculum': True, 'curriculum_phase_boundaries': [0.2, 0.7], 'curriculum_noise_ranges': [[600, 1000], [200, 900], [0, 400]], 'curriculum_horizons': [8, 64, 256], '_name': 'df_video_mamba3stage', 'require_pose_prediction': False, 'use_memory_attention': False, 'relative_embedding': False, 'memory_retrieval_topk': 8}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': None, 'load': None, 'name': 'train_stage_a_mamba', 'customized_load': True, 'seperate_load': True, 'diffusion_model_path': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', 'vae_path': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'output_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'}
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:init():658] starting backend
2026-02-27 13:46:36,574 INFO    MainThread:1829233 [wandb_init.py:init():662] setting up manager
2026-02-27 13:46:36,585 INFO    MainThread:1829233 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-02-27 13:46:36,592 INFO    MainThread:1829233 [wandb_init.py:init():670] backend started and connected
2026-02-27 13:46:36,612 INFO    MainThread:1829233 [wandb_init.py:init():768] updated telemetry
2026-02-27 13:46:36,662 INFO    MainThread:1829233 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
2026-02-27 13:46:37,245 ERROR   MainThread:1829233 [wandb_init.py:init():827] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-27 13:46:37,246 ERROR   MainThread:1829233 [wandb_init.py:init():1239] error in wandb.init()
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1229, in init
    return wi.init()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 833, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-27 13:46:38,381 WARNING MsgRouterThr:1829233 [router.py:message_loop():77] message_loop has been closed