2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Configure stats pid to 1505055
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldMem_Repro/wandb/settings
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False}
2026-02-27 13:02:22,434 WARNING MainThread:1505055 [wandb_setup.py:_flush():77] Could not find program at -m main
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'}
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_setup.py:_flush():77] Applying login settings: {}
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_130222-lxvmjzq8/logs/debug.log
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_130222-lxvmjzq8/logs/debug-internal.log
2026-02-27 13:02:22,434 INFO    MainThread:1505055 [wandb_init.py:init():608] calling init triggers
2026-02-27 13:02:22,435 INFO    MainThread:1505055 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 2e-05, 'batch_size': 64, 'max_epochs': -1, 'max_steps': 700000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'n_frames': 200, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft_latent', 'precomputed_feature_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'use_explicit_memory_frames': False, 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 8, 'customized_validation': True, 'add_timestamp_embedding': True}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 1, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, 'memory_condition_length': 0, 'log_video': True, 'use_compressed_causal_memory': False, 'compressed_memory_dim': 256, 'compressed_memory_slots': 64, 'compressed_memory_topk': 4, 'use_mamba_memory_pipeline': True, 'training_stage': 'stage_a_memory_pretrain', 'stage_c_memory_aux_weight': 0.1, 'diff_window_size': 8, 'memory_gap_aux_weight': 0.1, 'memory_gap_aux_weight_init': 0.5, 'memory_gap_aux_weight_final': 0.1, 'memory_gap_aux_decay_fraction': 0.3, 'memory_gap_aux_decay_steps': 0, 'memory_gap_aux_exp_k': 5.0, 'use_precomputed_features': True, 'mamba_latent_channels': 16, 'mamba_model_dim': 256, 'mamba_depth': 4, 'mamba_cond_dim': 256, 'mamba_d_state': 16, 'mamba_d_conv': 4, 'mamba_expand': 2, 'allow_mamba_fallback': False, 'strict_causal_training': True, 'strict_causal_evaluation': True, 'use_oracle_pose_eval': False, 'enable_memory_noise_curriculum': True, 'curriculum_phase_boundaries': [0.2, 0.7], 'curriculum_noise_ranges': [[600, 1000], [200, 900], [0, 400]], 'curriculum_horizons': [8, 64, 256], '_name': 'df_video_mamba3stage', 'require_pose_prediction': False, 'use_memory_attention': False, 'relative_embedding': False, 'memory_retrieval_topk': 8}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': None, 'load': None, 'name': 'train_stage_a_mamba', 'customized_load': True, 'seperate_load': True, 'diffusion_model_path': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/diffusion_only.ckpt', 'vae_path': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vae_only.ckpt', 'output_dir': '/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/'}
2026-02-27 13:02:22,435 INFO    MainThread:1505055 [wandb_init.py:init():658] starting backend
2026-02-27 13:02:22,435 INFO    MainThread:1505055 [wandb_init.py:init():662] setting up manager
2026-02-27 13:02:22,459 INFO    MainThread:1505055 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-02-27 13:02:22,468 INFO    MainThread:1505055 [wandb_init.py:init():670] backend started and connected
2026-02-27 13:02:23,404 INFO    MainThread:1505055 [wandb_init.py:init():768] updated telemetry
2026-02-27 13:02:23,488 INFO    MainThread:1505055 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
2026-02-27 13:02:24,058 ERROR   MainThread:1505055 [wandb_init.py:init():827] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-27 13:02:24,058 ERROR   MainThread:1505055 [wandb_init.py:init():1239] error in wandb.init()
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1229, in init
    return wi.init()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 833, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-27 13:02:25,191 WARNING MsgRouterThr:1505055 [router.py:message_loop():77] message_loop has been closed