dememwm-separate-memory-16762273.out

*************************************************** 
 You have loaded a buildenv module 
*************************************************** 
The buildenv-gcccuda/12.1.1-gcc12.3.0 module makes available: 
 - GCC: 12.3.0 
 - CUDA: 12.1.1 
 - OpenMPI: 4.1.5 
 - FFTW: 3.3.10 
 - OpenBLAS: 0.3.23 
 - ScaLAPACK: 2.2.0 
 - MAGMA: 2.7.1 
 - Eigen: 3.4.0 
 - OpenCV: 4.8.0 
 - LLVM: 17.0.1 
 
These libraries are accessible via the standard environment 
variables CPATH (for headers) and LIBRARY_PATH (for libraries), 
which are picked up automatically by the compiler toolchain. 
 
Already on 'dememwm-separate-mem'
Your branch is up to date with 'origin/dememwm-separate-mem'.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
Error executing job with overrides: ['+name=train_dememwm_separate_memory_berzelius_8a100_bs16_global128_350k', '+output_dir=/proj/cvl/users/x_fahkh2/DeMemWM/results/dememwm_separate_memory_berzelius_8a100_bs16_global128_350k/', 'auto_resume=true', 'experiment.tasks=[training]', 'algorithm=dememwm_memory_dit', '+customized_load=true', '+seperate_load=true', '+diffusion_model_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/oasis500m.safetensors', '+vae_path=/proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/vit-l-20.safetensors', '+only_tune_memory=false', 'dataset=video_minecraft_latent', 'dataset.save_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft', 'dataset.precomputed_feature_dir=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft/vae_features', 'dataset.n_frames=1000', '+dataset.n_frames_valid=500', '+dataset.customized_validation=true', '+dataset.wo_updown=false', '+dataset.angle_range=180', '+dataset.pos_range=8', '++algorithm.n_tokens=4', 'algorithm.x_shape=[16,18,32]', '++algorithm.context_frames=100', '++algorithm.log_video=true', '++algorithm.diffusion.sampling_timesteps=20', '++algorithm.dememwm.generated_history_proxy.enabled=true', '++algorithm.dememwm.generated_history_proxy.start_step=40000', '++algorithm.dememwm.generated_history_proxy.ramp_steps=40000', '++algorithm.dememwm.generated_history_proxy.max_prob=0.25', '++algorithm.dememwm.generated_history_proxy.noise_std=0.25', '++algorithm.dememwm.generated_history_proxy.dropout_prob=0.0', '++algorithm.dememwm.anchor.enabled=true', '++algorithm.dememwm.anchor.anchor_indices=[0,1,2,3]', '++algorithm.dememwm.anchor.diverse_selection=true', '++algorithm.dememwm.anchor.compress.downsample_ratio=2', '++algorithm.dememwm.anchor.allow_generated_as_anchor=false', '++algorithm.dememwm.dynamic.enabled=true', '++algorithm.dememwm.dynamic.recent_frames=4', '++algorithm.dememwm.dynamic.conv_stride_t=2', '++algorithm.dememwm.dynamic.compress.downsample_ratio=2', '++algorithm.dememwm.revisit.enabled=true', '++algorithm.dememwm.revisit.deterministic_pose_retrieval=true', '++algorithm.dememwm.revisit.fov_overlap_threshold=0.60', '++algorithm.dememwm.revisit.pose_preselect_topk=64', '++algorithm.dememwm.revisit.fov_yaw_samples=25', '++algorithm.dememwm.revisit.fov_pitch_samples=20', '++algorithm.dememwm.revisit.fov_depth_samples=20', '++algorithm.dememwm.revisit.plucker_weight=0.10', '++algorithm.dememwm.revisit.max_frames=2', '++algorithm.dememwm.revisit.compress.downsample_ratio=2', '++algorithm.dememwm.cache.enabled=true', '++algorithm.dememwm.cache.device=cpu', '++algorithm.dememwm.cache.keep_raw_latents=all', '++algorithm.dememwm.cache.keep_compressed_records=true', '++algorithm.dememwm.cache.eviction_policy=none', '++algorithm.dememwm.cache.no_evict=true', '++algorithm.dememwm.cache.clear_between_videos=true', '++algorithm.dememwm.cache.max_records=null', '++algorithm.dememwm.cache.on_capacity_exceeded=warn', '++algorithm.dememwm.curriculum.enabled=true', '++algorithm.dememwm.curriculum.full_stage_start_step=20000', '++algorithm.dememwm.curriculum.freeze_vae=true', '++algorithm.dememwm.curriculum.dit_freeze.enabled=true', '++algorithm.dememwm.curriculum.lr.dememwm_modules=4.0e-5', '++algorithm.dememwm.curriculum.lr.memory_adapters=4.0e-5', '++algorithm.dememwm.curriculum.lr.full_dit=1.0e-5', 'experiment.training.batch_size=16', 'experiment.training.optim.accumulate_grad_batches=1', 'experiment.validation.batch_size=1', 'experiment.validation.limit_batch=1', 'experiment.training.checkpointing.every_n_train_steps=5000', 'experiment.validation.val_every_n_step=5000', 'experiment.training.max_steps=350000']
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 318, in <module>
    run()  # pylint: disable=no-value-for-parameter
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 314, in run
    run_local(cfg)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/main.py", line 238, in run_local
    experiment.exec_task(task)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 350, in exec_task
    getattr(self, task)()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 447, in training
    self.algo = self._build_algo()
  File "/proj/cvl/users/x_fahkh2/DeMemWM/experiments/exp_base.py", line 336, in _build_algo
    return self.compatible_algorithms[algo_name](self.root_cfg.algorithm)
  File "/proj/cvl/users/x_fahkh2/DeMemWM/algorithms/worldmem/df_video.py", line 341, in __init__
    self.memory_condition_length = cfg.memory_condition_length
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'memory_condition_length' is not in struct
    full_key: algorithm.memory_condition_length
    object_type=dict
srun: error: node092: tasks 0-7: Exited with exit code 1
srun: Terminating StepId=16762273.0